phase: 28_gateway_agent_proxy
**Phase 28 final pass — complete, 100 % green.** All five tasks were already `complete/`; this pass verified the criteria, re-ran the live half, and did the docs pass. - Gate `bash .agents/validate.sh` **rc=0**: `1650 passed, 15 skipped`, coverage **98.97 %** (floor 94), `ruff` clean, `pyright` **0 errors**, `node --check` clean, hygiene clean (`/tmp/validate-done.log`). - Live: `pytest -m live --live` **11 passed**; `./scripts/dev-server.sh status` nine sources `ok`; mTLS re-read of `gateway` → agent **1.2.0**, all three sections `available: true`, 6/6 frontends routed, 11 nginx servers + `$targetBackend`, **19** listeners, no core errors (`data/logs/28-finalpass-live.out`); installed agent file byte-identical to this tree (`b0e9ebf1b718…`). - Criteria: 1 ✅ pytest/coverage · 2 ✅ three sections over a captured tree, `false`/`null` + one `errors` line each elsewhere with `cpu`/`mem`/`disk_io`/`system` intact · 3 ✅ `AGENT_VERSION` 1.2.0 + SCHEMA names all three · 4 ✅ six fixtures verbatim, capture refuses a bad read, 116 socket address columns hold no globally routable endpoint · 5 ✅ gateway runs 1.2.0 and answers over mTLS (transcript `data/logs/28-task05-rollout.md`, `28-finalpass-live.out`) · 6 ✅ validate · 7 ✅ additive: HEAD's 1.1.0 and this 1.2.0 `collect()`ed over one `--root` are key-for-key equal plus the three new sections, `--no-proxy` equal exactly (`data/logs/verify-28-final.py` → 41 PASS, `28-final-verify.out`). - Docs fixed (README/AGENTS/Containerfile, all re-green): README's opening said "Four dashboards" above a five-row table and said the new sections are "stored in the snapshot" (they are in the **agent's** document — no `/api` payload, no `sample` row, nothing charts them until phase 29); `Containerfile`'s header still said "three dashboards"; AGENTS's coverage figure (98.94 → **98.97 %**), invariant 6 gained the captured-socket-table identifier rule, the agent bullet gained "phase 28 stops at the wire", and the fleet record names the stash-and-compare control plus the volatile listener count. Full record: `data/logs/NOTES-28-final.md`. - Deviations: none functional; no git index/history touched, phase files untouched (`PLAN.md` and `complete/` unchanged). `uv pip list --outdated` still lists 4 packages (fastapi 0.153→0.161 etc.) — advisory only, dependencies untouched by this phase, left for a deliberate pass. - Next pending phase: `.agents/phases/todo/29_gateway_dashboard_page/` (the page that consumes these three sections).
This commit is contained in:
@@ -0,0 +1,8 @@
|
||||
**Phase 28 final pass — complete, 100 % green.** All five tasks were already `complete/`; this pass verified the criteria, re-ran the live half, and did the docs pass.
|
||||
|
||||
- Gate `bash .agents/validate.sh` **rc=0**: `1650 passed, 15 skipped`, coverage **98.97 %** (floor 94), `ruff` clean, `pyright` **0 errors**, `node --check` clean, hygiene clean (`/tmp/validate-done.log`).
|
||||
- Live: `pytest -m live --live` **11 passed**; `./scripts/dev-server.sh status` nine sources `ok`; mTLS re-read of `gateway` → agent **1.2.0**, all three sections `available: true`, 6/6 frontends routed, 11 nginx servers + `$targetBackend`, **19** listeners, no core errors (`data/logs/28-finalpass-live.out`); installed agent file byte-identical to this tree (`b0e9ebf1b718…`).
|
||||
- Criteria: 1 ✅ pytest/coverage · 2 ✅ three sections over a captured tree, `false`/`null` + one `errors` line each elsewhere with `cpu`/`mem`/`disk_io`/`system` intact · 3 ✅ `AGENT_VERSION` 1.2.0 + SCHEMA names all three · 4 ✅ six fixtures verbatim, capture refuses a bad read, 116 socket address columns hold no globally routable endpoint · 5 ✅ gateway runs 1.2.0 and answers over mTLS (transcript `data/logs/28-task05-rollout.md`, `28-finalpass-live.out`) · 6 ✅ validate · 7 ✅ additive: HEAD's 1.1.0 and this 1.2.0 `collect()`ed over one `--root` are key-for-key equal plus the three new sections, `--no-proxy` equal exactly (`data/logs/verify-28-final.py` → 41 PASS, `28-final-verify.out`).
|
||||
- Docs fixed (README/AGENTS/Containerfile, all re-green): README's opening said "Four dashboards" above a five-row table and said the new sections are "stored in the snapshot" (they are in the **agent's** document — no `/api` payload, no `sample` row, nothing charts them until phase 29); `Containerfile`'s header still said "three dashboards"; AGENTS's coverage figure (98.94 → **98.97 %**), invariant 6 gained the captured-socket-table identifier rule, the agent bullet gained "phase 28 stops at the wire", and the fleet record names the stash-and-compare control plus the volatile listener count. Full record: `data/logs/NOTES-28-final.md`.
|
||||
- Deviations: none functional; no git index/history touched, phase files untouched (`PLAN.md` and `complete/` unchanged). `uv pip list --outdated` still lists 4 packages (fastapi 0.153→0.161 etc.) — advisory only, dependencies untouched by this phase, left for a deliberate pass.
|
||||
- Next pending phase: `.agents/phases/todo/29_gateway_dashboard_page/` (the page that consumes these three sections).
|
||||
@@ -0,0 +1 @@
|
||||
ok
|
||||
@@ -0,0 +1,71 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 8%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 30%]
|
||||
.......s................................................................ [ 34%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 69%]
|
||||
....................................sssssssssss......................... [ 73%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 95%]
|
||||
.......................sss.............................................. [ 99%]
|
||||
......... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 440 0 100%
|
||||
src/dashboards/app.py 186 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 206 4 98% 304, 308-309, 326
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3496 36 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.97%
|
||||
1650 passed, 15 skipped, 1 warning in 157.81s (0:02:37)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
coverage 7.16.2 7.17.0 wheel
|
||||
fastapi 0.153.0 0.161.1 wheel
|
||||
pycparser 3.11 3.12 wheel
|
||||
ruff 0.17.1 0.17.2 wheel
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
**Task 28/01 complete — green.**
|
||||
|
||||
- `agent/host_agent.py`: new `haproxy` snapshot section — `haproxy_config` argument (+`--haproxy-config`, `--no-proxy`), module-level `parse_haproxy_config()` / `_scan_haproxy_config()`, `Collector.haproxy()`, wired into `snapshot()` before `errors`. Shape matches the task's assumptions; `:::22`→`addr "::"`, port `22`; a portless target keeps `port: null`; `use_backend %[…]` contributes no name; `listen` becomes the pair haproxy defines; design-rules + SCHEMA docstring updated (`AGENT_VERSION` left at 1.1.0 — task 04 owns the bump).
|
||||
- Three-way `available` pinned: `true` / `false` (missing file, one `haproxy: … not found …` line) / `null` (mode-000, a directory, oversized, non-haproxy text) — each exactly **one** line, snapshot and `/health` intact. `--no-proxy` removes the key and the line, Collector-level and through `main()`.
|
||||
- `src/dashboards/collectors.py`: `ADDON_ERROR_PREFIXES = ("smart", "haproxy")` — defect found while implementing. The task mandates the absence line, but `degraded` (that list) is `alerts.py`'s input, so shipping it unfiltered would have put a red banner + overview WARN on every host with no haproxy after the upgrade. Filtered there, still verbatim in `agent.errors` for phase 29.
|
||||
- Tests: 19 new (17 `tests/test_agent.py`, 2 `tests/test_collectors.py`); three existing `errors == []` assertions now filter add-on lines via `core_errors()`.
|
||||
- `tests/fixtures/gateway_haproxy.cfg`: **read-only** ssh capture, body byte-identical to `gateway:/etc/haproxy/haproxy.cfg` (mode 0644 — so the agent user can read it; no credential in it). Parsed end-to-end: 6 frontends / 6 backends, all `default_backend`s resolve.
|
||||
- Docs: README agent section, AGENTS.md agent + `collectors.py` bullets, gateway's fleet row, pytest tally.
|
||||
- Gate: `bash .agents/validate.sh` **rc=0** — 1568 passed / 15 skipped, coverage 98.94 % (floor 94), ruff clean, pyright 0 errors, `node --check` clean. Only the pre-existing advisory (fastapi 0.153.0→0.159.1; no dependency touched).
|
||||
- Dev server restarted: 9 sources `ok`; `/api/system/snapshot` unchanged on the live fleet (agents are 1.1.0, so no `haproxy` key at all). Evidence: `data/logs/NOTES-28-haproxy.md`.
|
||||
- No live change made to any host; no git index/history touched; phase files unmodified. Cosmetic note for task 05/29: after a fleet upgrade, a non-haproxy host's collapsed "More detail" banner will print the absence line — same as a smart line today; alerts are unaffected.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 13%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 27%]
|
||||
.........s.............................................................. [ 31%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 68%]
|
||||
..........................sssssssssss................................... [ 72%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 95%]
|
||||
.............sss....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 440 0 100%
|
||||
src/dashboards/app.py 186 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 206 5 98% 296-297, 301-302, 319
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3496 37 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.94%
|
||||
1568 passed, 15 skipped, 1 warning in 165.97s (0:02:45)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
fastapi 0.153.0 0.159.1 wheel
|
||||
@@ -0,0 +1,17 @@
|
||||
**Implemented (phase 28 task 02 — agent `nginx` section)**
|
||||
- `agent/host_agent.py`: additive `nginx` section — `Collector.nginx(errors)` + `nginx_include_files()` + pure `nginx_include_paths()` / `parse_nginx_config()`; brace-block scanner (`_nginx_tokens`/`_nginx_statements`/`_collect_nginx`) so quoted `;`/`{`, comments, nested `server`/`location`/`upstream`/`map` all parse; `--nginx-config` flag; shared `_config_path`/`read_config`/`_config_label` now used by both proxy sections; SCHEMA docstring + design-rules bullet updated.
|
||||
- Three-way `available` with exactly one `nginx:` line per arm ("not found" pinned); includes followed **one level**, bounded 8 files / 1 MB (excess dropped with one line naming the count, section stays `available: true`); `files` = main + merged docs in the config's own spelling; `proxy_pass $var` served with `variable` + `target: null` (never merged); `--no-proxy` now removes `haproxy` **and** `nginx`.
|
||||
- `collectors.ADDON_ERROR_PREFIXES` grew `nginx` in the same commit (AGENTS.md rule) + new pin that the agent-side and dashboard-side lists are equal (that pin was claimed but missing — defect fixed).
|
||||
- Verified the parser read-only against the **real** gateway tree (`cat` over ssh): 11 servers, both maps, 17 `$targetBackend` entries, the `127.0.0.1:8081` literal, stream `proxy_pass $targetBackend` unmerged. No live change made (install is task 05).
|
||||
|
||||
**Tests / lint** (`uv run …`)
|
||||
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1603 passed, 15 skipped, 98.94%** (was 1568; 34 new nginx cases, one 6-way parametrise over malformed files).
|
||||
- `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → 0 errors.
|
||||
- `uv run pytest -m live --live` → 11 passed; `./scripts/dev-server.sh status` → 9 sources `ok`.
|
||||
- Agent-file coverage via `coverage run --branch --source=agent`: 94%, new code fully covered except the unreachable defensive `except` in `_config_label` (>90% criterion met).
|
||||
|
||||
**Docs**: `README.md` (nginx add-on paragraph), `AGENTS.md` (agent bullet for the `nginx` section, `ADDON_ERROR_PREFIXES` list, gateway fleet row, test tally). Transcript: `data/logs/NOTES-28-nginx.md` (gitignored).
|
||||
|
||||
**Deviations**: map entry keys carry `variable`/`key` with the `$` dropped on both sides of the proxy_pass↔map join (one canonical spelling instead of the task's mixed `$targetBackend`/`targetBackend`); map values that are not addresses report `target: null` rather than a host-shaped guess. `uv pip list --outdated` still reports a pre-existing `fastapi 0.153.0 → 0.159.1` (validate.sh prints it as an advisory only; deps untouched by this task).
|
||||
|
||||
**Next pending task**: `28_gateway_agent_proxy/03_socket_section.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 8%]
|
||||
........................................................................ [ 13%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 26%]
|
||||
............................................s........................... [ 31%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 40%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 66%]
|
||||
.............................................................sssssssssss [ 71%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 93%]
|
||||
................................................sss..................... [ 97%]
|
||||
.................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 440 0 100%
|
||||
src/dashboards/app.py 186 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 206 5 98% 302-303, 307-308, 325
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3496 37 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.94%
|
||||
1603 passed, 15 skipped, 1 warning in 162.84s (0:02:42)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
fastapi 0.153.0 0.159.1 wheel
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
Task 03 (`sockets` section) is complete.
|
||||
|
||||
**Implemented** — `agent/host_agent.py`:
|
||||
- `parse_proc_net_tcp(text, family)` keeps only `0A` LISTEN → `{family, addr, port}` and `01` ESTABLISHED → `{family, local, remote}`; every other state is skipped silently. `_proc_net_addr` unrolls `/proc`'s per-32-bit-word little-endian spelling through `socket.inet_ntop`, so wildcards stay literal `0.0.0.0` / `::` and a bad word yields `addr: null` (never `0`); a bad port is `null` too.
|
||||
- `Collector.read_proc_table()` bounds each table to `PROC_NET_MAX_LINES` (4096, `/proc`'s header among them) and reports the excess as **one** line naming the path; `Collector.sockets(errors)` reads v4/v6 independently — either answering gives `available: true`, a missing `tcp6` is silent (kernel without IPv6), a denied table always costs one line, neither answering gives `available: null` plus one combined line. There is deliberately **no `false` arm** (`false` beside empty lists would read "no listeners").
|
||||
- `document["sockets"]` is attached inside `Collector.snapshot()` only when `proxy_enabled`, wrapped in `_safe`, so an unexpected exception costs one `sockets:` line and nothing else — `/snapshot` and `/health` never fail. `--no-proxy` removes the key with no error line.
|
||||
- `src/dashboards/collectors.py`: `ADDON_ERROR_PREFIXES = ("smart", "haproxy", "nginx", "sockets")` so absence/errors do not turn `/system` red.
|
||||
|
||||
**Tests** — `tests/test_agent.py`: v4+v6 parsing incl. wildcard `0.0.0.0`/`::`, every non-`0A`/`01` state skipped, malformed endpoint words, header line ignored, `--root` fake-tree reads, additivity (only `ts`/`duration_ms`/`sockets`/`errors` move), both-missing → one line naming both paths, missing `tcp6` silent, denied table costs one line, truncation reported once, `--no-proxy` (payload, `--once`, `/schema`), `_safe` fallback, `inet_ntop` raising, and a real-`/proc` shape test. `tests/test_collectors.py`: `sockets:` lines excluded from `degraded`, prefix tuple pinned.
|
||||
|
||||
**Verification** — `bash .agents/validate.sh`: **1630 passed, 15 skipped**; coverage **98.94%** (floor 94% met); `ruff` clean; `pyright` 0 errors. Real read-only `--once` on this box: 14 listeners / 8 established, matching `ss -ltnH` exactly, no error lines. No live-infrastructure changes; no git index/history commands.
|
||||
|
||||
**Docs** — README agent section + AGENTS.md `sockets` rules and the add-on-prefix sentence.
|
||||
|
||||
**Next pending task, not started**: `04_version_and_capture.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+67
@@ -0,0 +1,67 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 8%]
|
||||
........................................................................ [ 13%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 26%]
|
||||
.......................................................................s [ 30%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 70%]
|
||||
................sssssssssss............................................. [ 74%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 96%]
|
||||
...sss....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 440 0 100%
|
||||
src/dashboards/app.py 186 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 206 5 98% 303-304, 308-309, 326
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3496 37 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.94%
|
||||
1630 passed, 15 skipped, 1 warning in 449.51s (0:07:29)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
fastapi 0.153.0 0.159.1 wheel
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
**Task 04 complete — `AGENT_VERSION` 1.2.0, the SCHEMA, and the gateway captures.**
|
||||
|
||||
**Agent** — `agent/host_agent.py`: `AGENT_VERSION` `1.1.0` → `1.2.0`; the `GET /schema` block already names `haproxy`/`nginx`/`sockets` (each with its `available`), `--no-proxy`, `--haproxy-config`, `--nginx-config`. `tests/test_agent.py` pins the constant plus all of that; the phase-15 TLS pin keeps its schema assertions and no longer competes over the number.
|
||||
|
||||
**Capture** — new `scripts/capture-gateway-config.sh` (`GATEWAY_SSH`, default `gateway-ops`): one `ssh … cat` per document (`scp` would copy `/proc/net/tcp` as 0 bytes), no sudo, haproxy.cfg + `nginx.conf` **plus every document its own glob `include` names** + `/proc/net/tcp{,6}`, provenance header naming the remote path and script, and a refusal (`exit 1`, nothing written, temp removed) for a failed/empty read, an unnameable include dir or a missing glob. Ran it: 7 fixtures now in `tests/fixtures/` (haproxy, nginx main + `conf_d_http` + `stream_d_stream`, both socket tables).
|
||||
|
||||
**Deviation from the task's "scrubs nothing" assumption** — configs stay verbatim (domains, `proxy_pass` targets, key *paths*); the two socket tables now replace **only globally routable endpoints** with documentation space (RFC 5737/3849), idempotent and column-width-preserving, because a global IPv6 here is EUI-64 (MAC-derived) and an inbound visitor's address is not ours to publish — the smartctl serial rule (invariant 6). Documented in README + AGENTS.md and pinned in both test files.
|
||||
|
||||
**Tests** — `test_agent.py`: 7 tests over the captures (header/lineage, the merged include tree, SNI map + `proxy_pass $var` unmerged, every table row decodes, config↔socket join on 22/443/4443/8443/9120, no globally routable endpoint, one collector answering all three sections). `test_deploy_scripts.py`: 10 pins (alias dialed and nothing else, no `sudo`/IP in code, full tree recorded, domains verbatim, dead gateway + empty read leave fixtures alone, the two refusal messages, substitution behaviour + idempotence).
|
||||
|
||||
**Live probe (read-only)** — `data/logs/verify-28-task04.py` → `28-task04-probe.out`: gateway `active`, installed **1.1.0**, CA-only `/health` 200, `/snapshot` 403 `client certificate required`, mTLS snapshot has every pre-1.2.0 section with live numbers and **none** of the three new ones → **PASS: no behaviour change**.
|
||||
|
||||
**Gate** — `bash .agents/validate.sh`: **1650 passed, 15 skipped**, coverage **98.94%**, ruff clean, pyright 0 errors, `node --check` clean, hygiene clean (only the pre-existing `fastapi` advisory); `pytest -m live --live` with `.env.dev`: **11 passed**. Docs updated in README + AGENTS.md (capture script, the three capture rules, the fleet read). Nothing staged or committed; new script/fixtures are untracked for task 05.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+68
@@ -0,0 +1,68 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 8%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 30%]
|
||||
.......s................................................................ [ 34%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 69%]
|
||||
....................................sssssssssss......................... [ 73%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 95%]
|
||||
.......................sss.............................................. [ 99%]
|
||||
......... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 440 0 100%
|
||||
src/dashboards/app.py 186 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 206 5 98% 303-304, 308-309, 326
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3496 37 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.94%
|
||||
1650 passed, 15 skipped, 1 warning in 457.28s (0:07:37)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
fastapi 0.153.0 0.159.1 wheel
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
**Task 05 complete — gateway host now runs agent 1.2.0, verified over mTLS.**
|
||||
|
||||
- Re-staged `/home/ops/agent-install/` on `gateway-ops` (gap-10 rule): `host_agent.py` 1.2.0 (`b0e9ebf1…`, md5 `9e062e4f…`) + installer (`ffbcd91f…`), old files snapshotted as `*-20261022-backup` (names the sudo grant doesn't name).
|
||||
- Installed via the granted path with the unit's own args (`--bind 10.1.1.27 --port 9120`); **re-run guard printed nothing** — `systemctl cat` byte-identical before/after (`26bd812…`); bundle + token reused, not rewritten.
|
||||
- `sudo systemctl restart dashboards-agent` → `active`, pid 20708, `https … client cert required, rev 2`.
|
||||
- Verified: CA-only `--status` answers `{"ok":true,…,"version":"1.2.0"}` and still 403s `/snapshot`; mTLS probe (`data/logs/verify-28-task05.py` → `28-task05-verify.out`, exit 0) asserts version 1.2.0, all three new keys, `haproxy.available=true` (6 routed frontends/6 backends), `nginx.available=true` (11 servers, `$targetBackend` map, 19 entries), `sockets.available=true` (17 listens incl. **443**), and every pre-1.2.0 section intact with live numbers, no core error.
|
||||
- Dashboard side untouched: `./scripts/dev-server.sh status` → **nine sources `ok`**; `/api/system/snapshot` gateway row `agent.version 1.2.0`, no `degraded`; `/api/disks/snapshot` still 2 hosts.
|
||||
- Transcript: `data/logs/28-task05-rollout.md` (+ `28-task05-raw.log`) — hashes, ports and counts only, no config body, no address, no credential.
|
||||
|
||||
**Gates:** `bash .agents/validate.sh` → **exit 0**: `uv run pytest --cov --cov-fail-under=94` **1650 passed / 15 skipped**, coverage **98.94 %**; `ruff check src tests agent` clean; `pyright` clean; `node --check` clean; hygiene clean. Re-run after doc edits `uv run pytest -q` → exit 0; doc pins `tests/test_auth_docs.py tests/test_deploy_scripts.py` → **141 passed**; `uv run pytest -m live --live` → **11 passed** (read-only).
|
||||
|
||||
**Docs:** `AGENTS.md` fleet record (gateway on 1.2.0, other two still 1.1.0 and why that costs nothing; the rollout procedure, byte-identical unit, verification pointers) + gap 11's end-state line; `README.md` new "Upgrading an already-registered host is one file, not a rollout" paragraph (bundle-on-disk vs `--cert-bundle` flag, silent re-run guard, restart, both probes).
|
||||
|
||||
**Deviations:** no code change (as the task states); `uv pip list --outdated` shows pre-existing `fastapi 0.153.0 → 0.159.1` drift left alone — this task moved no dependencies and a bump belongs in its own change.
|
||||
|
||||
**Next pending:** phase 28 has no further tasks; `29_gateway_dashboard_page/01_…` is next in `todo/`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+68
@@ -0,0 +1,68 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 8%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 21%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 30%]
|
||||
.......s................................................................ [ 34%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 69%]
|
||||
....................................sssssssssss......................... [ 73%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 95%]
|
||||
.......................sss.............................................. [ 99%]
|
||||
......... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 440 0 100%
|
||||
src/dashboards/app.py 186 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 206 5 98% 303-304, 308-309, 326
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 3496 37 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.94%
|
||||
1650 passed, 15 skipped, 1 warning in 455.76s (0:07:35)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
fastapi 0.153.0 0.159.1 wheel
|
||||
@@ -116,11 +116,20 @@ page still renders.
|
||||
looser), dials `GET /api/certs/<name>/bundle` over HTTP instead of importing `ensure_certs` — so
|
||||
it cannot mint even by accident — and prints one line per host with no PEM, no token and no
|
||||
password in it (`tests/test_deploy_scripts.py` greps the output for all three).
|
||||
**A captured socket table is the same case** (phase 28): `scripts/capture-gateway-config.sh`
|
||||
keeps every config byte (a `server_name` and its `proxy_pass` are the ingress map the page draws,
|
||||
and a key *path* is a filename, not a key) and rewrites **only** the two address columns of
|
||||
`/proc/net/tcp{,6}`, and only when `ipaddress` calls the address globally routable — a Linux
|
||||
global IPv6 is EUI-64, so it encodes this box's NIC, and an inbound visitor's address was never
|
||||
ours to publish. Documentation space (RFC 5737 / RFC 3849), first-seen order, same width, same
|
||||
`::ffff:` spelling, idempotent; ports, states and every other column stay verbatim.
|
||||
`tests/test_agent.py` fails the day a fixture grows a routable endpoint back.
|
||||
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
|
||||
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
|
||||
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
|
||||
`scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing
|
||||
exporters) rather than trusting a README or a memory of an upstream.
|
||||
exporters, `scripts/capture-gateway-config.sh` for the agent's `haproxy`/`nginx`/`sockets`
|
||||
ground truth) rather than trusting a README or a memory of an upstream.
|
||||
8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale.
|
||||
Check after touching dependencies: `uv pip list --outdated` must come back empty.
|
||||
Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile
|
||||
@@ -194,7 +203,8 @@ store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ─
|
||||
Responsibilities, one line each:
|
||||
|
||||
- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`,
|
||||
`statvfs`; serves `GET /snapshot` + `GET /info` + `GET /schema` + `GET /health`, plus an
|
||||
`statvfs` — plus, for the proxy add-on, the haproxy and nginx configs and `/proc/net/tcp{,6}`
|
||||
on the host that has them; serves `GET /snapshot` + `GET /info` + `GET /schema` + `GET /health`, plus an
|
||||
HTML status page at `GET /`. Refuses to bind off-loopback without a token **or** a certificate
|
||||
bundle (exit 2). Counters stay counters: the dashboard computes
|
||||
rates so the agent never remembers who polled it last.
|
||||
@@ -238,6 +248,105 @@ Responsibilities, one line each:
|
||||
*system* source red every four minutes over an add-on that page never draws. The refresher
|
||||
is single-flight and keeps the old cache on failure, so a bad pass retries on the next
|
||||
expiry instead of hammering the disk.
|
||||
**The proxy-config add-on (phase 28 task 01)** is the `haproxy` section: `Collector.haproxy()`
|
||||
reads `--haproxy-config` (default `/etc/haproxy/haproxy.cfg`, resolved under `--root` so the
|
||||
fake-tree tests read the fake tree, one size-bounded read per snapshot) and
|
||||
`parse_haproxy_config()` turns its text into `{frontends: [{name, binds: [{addr, port}],
|
||||
default_backend, backends}], backends: [{name, servers: [{name, target, port}]}]}`. Three rules
|
||||
keep it honest. (a) `available` is `smartctl_available`'s three-way answer — `true` read and
|
||||
parsed, `false` no file at that path, `null` a file that could not be read or holds no haproxy
|
||||
section — and each non-`true` arm costs exactly **one** `haproxy:` line for the pass, the missing
|
||||
one saying **not found** (pinned wording, the same cross-module contract as smartctl's, because
|
||||
the dashboard reads that word to answer "is haproxy installed?"). (b) The section reports the
|
||||
config's own text and infers nothing about a request: a bind keeps the address as written
|
||||
(`:::22` splits at the **last** colon into `"::"` and `22`, so an IPv6 bind survives, and
|
||||
`bind :8080` keeps `addr: null` because `*` is a spelling the config did not use), a `server`
|
||||
with no port keeps `port: null` and never `0`, a `use_backend` whose target is a fetch
|
||||
expression (`%[req.hdr(host),map_dom(...)]`) contributes **no name** rather than an expression
|
||||
where a backend belongs, and everything outside `bind`/`default_backend`/`use_backend`/`server`
|
||||
is ignored without a word. The frontend→backend routing is therefore the names, which the
|
||||
dashboard joins. (c) A `listen` section is reported as the frontend+backend pair haproxy itself
|
||||
defines it as, so a config written in that shorthand cannot read as "this gateway routes
|
||||
nothing". The section is absent from the document with `--no-proxy` (silence, the `--no-smart`
|
||||
precedent — "not collected" stays distinguishable from "collected and empty"), it is additive in
|
||||
the payload, and nothing in it can fail `/snapshot` or `/health`.
|
||||
`tests/fixtures/gateway_haproxy.cfg` is the verbatim capture the parser is pinned to.
|
||||
**The same add-on's `nginx` section (phase 28 task 02)** reads `--nginx-config` (default
|
||||
`/etc/nginx/nginx.conf`) **plus the files its own `include` lines name, one level deep** — the
|
||||
gateway keeps its vhosts in `conf.d/*.conf` and its SNI routing in `stream.d/*.conf`, so the
|
||||
main config alone is not the picture. `parse_nginx_config(main_text, included)` is the pure
|
||||
half over text, and it needs a real scanner rather than haproxy's indent loop because nginx
|
||||
nests by `{}`: `_nginx_tokens` keeps a `;` inside a quoted value as data (the real config writes
|
||||
`add_header … "max-age=31536000; includeSubDomains" always;`), `_nginx_statements` builds the
|
||||
block tree and raises only for a file nginx itself would refuse to start on, and the walk fills
|
||||
`server` blocks (`listen`, `server_name`, `proxy_pass`), `upstream` blocks and `map` blocks.
|
||||
Same three rules as `haproxy`: `available` is three-way with exactly one `nginx:` line per arm
|
||||
(the missing one saying **not found**, the pinned wording the dashboard reads), `--no-proxy`
|
||||
removes the key in silence, and nothing in the section can fail `/snapshot` or `/health` —
|
||||
including a config that will not tokenize, which is `available: null` plus its one line with
|
||||
every other section intact. What it refuses to do is interpret: `listen` keeps the address as
|
||||
written (`[::]:8443` → `"::"`, a bare `80` → `addr: null`) and ignores the `ssl`/`proxy_protocol`
|
||||
flags, `server_name _` stays the literal `_`, a map's value becomes a `target` only when it is
|
||||
an address (the `$connection_upgrade` map's `upgrade`/`close` are header words, and calling one
|
||||
a host is inventing a fact to fill a field), and a `proxy_pass $var` travels with `variable`
|
||||
(the `$` dropped once here, so both sides of the join spell it identically) and a **null
|
||||
`target`** — the map decides per connection, so a merged target would be rule 2 with a page's
|
||||
clothes on. Include-following is one level and bounded (8 files / 1 MB of included text, the
|
||||
smart pass-budget shape): the excess is dropped with **one** line naming the count and the
|
||||
section stays `available: true`; `files` is the main config then the documents actually merged,
|
||||
spelled the way the config spells them (`/etc/nginx/conf.d/http.conf`, never the path under
|
||||
`--root`, which is also what lets the parser match its own `include` lines); a literal include
|
||||
naming a file this host does not have costs nothing and no line; and an `include` inside an
|
||||
included file is not followed at all — `root=False` makes that structural, not a convention.
|
||||
**The same add-on's `sockets` section (phase 28 task 03)** is the live half: `read_proc_table`
|
||||
reads `/proc/net/tcp` and `/proc/net/tcp6` (resolved under `--root`, the config rule) through at
|
||||
most `PROC_NET_MAX_LINES` = **4096 lines per table** — `/proc`'s own header among them, so 4095
|
||||
sockets — because a gateway's file is one line per socket and the read must not cost the poll
|
||||
somebody else's connection count; the excess is reported in **one** line naming the bound and the
|
||||
section stays `available: true`. `parse_proc_net_tcp(text, family)` keeps two kernel states only:
|
||||
`0A` LISTEN → `{family, addr, port}` and `01` ESTABLISHED →
|
||||
`{family, local: {addr, port}, remote: {addr, port}}`; every other state is a socket in transit
|
||||
(`TIME_WAIT` above all) and is skipped in silence, since drawing it would put two pipes on one
|
||||
connection and bury the live pairs. `_proc_net_addr` unrolls `/proc`'s per-32-bit-word
|
||||
little-endian spelling through `socket.inet_ntop`, so a wildcard keeps its literal `0.0.0.0` /
|
||||
`::` — the value the inbound join compares against — and a word that is not an address is `addr:
|
||||
null` with the port kept, never a guess and never `0`. The two tables are read **independently**:
|
||||
`available` is `true` when either answered (a kernel built without IPv6 has no `tcp6`, which is
|
||||
absence, so no line — the nginx include rule), `null` with one combined `sockets:` line when
|
||||
neither did, and a table that is there and denied costs one line even beside a good one. This
|
||||
section has **no `false` arm on purpose** — `/proc/net/tcp` is not something a host leaves
|
||||
uninstalled, and `false` beside an empty list would read as "no listeners", which is rule 2 — and
|
||||
**no join with the config**: a socket does not know its domain, so phase 29's page joins these
|
||||
ports to the `haproxy`/`nginx` sections and an unclaimed socket stays visible as one.
|
||||
**Task 04 made the three sections a versioned contract with captured ground truth:**
|
||||
`AGENT_VERSION` is **1.2.0** (`tests/test_agent.py` pins the constant and asserts the
|
||||
`GET /schema` block names `haproxy`/`nginx`/`sockets` — each with its `available` flag — plus
|
||||
`--no-proxy`, `--haproxy-config` and `--nginx-config`), and `scripts/capture-gateway-config.sh`
|
||||
re-records the six fixtures the parsers are pinned to over `GATEWAY_SSH` (`gateway-ops`): the
|
||||
haproxy config, the nginx main config **plus every document its own glob `include` names** (so
|
||||
the fixture tree cannot drift from the config's tree while the parser still follows one level),
|
||||
and `/proc/net/tcp{,6}`. Three rules it exists to keep. (a) **No privilege and no copy tool**: one
|
||||
`ssh <alias> cat <path>` per document, because `/proc/net/tcp` reports size 0 and `scp` would copy
|
||||
an empty file, and every path it reads is world-readable — the same read the agent performs as its
|
||||
own user, so a capture that needed sudo would be recording something the agent cannot see.
|
||||
(b) **A bad read is never a fixture**: a nonzero `cat`/`ls`, an empty body or a missing glob include
|
||||
exits non-zero having written nothing, and a half-written file is removed — the pin is that the
|
||||
recorded bytes survive a dead alias and an empty read both. (c) **Configs verbatim, socket tables
|
||||
identifier-free**: the configs keep every `server_name`, `proxy_pass` and key path (a domain is the
|
||||
map the page draws and a key path is a filename, not a key, while an agent-only bundle carries no
|
||||
key at all), but a globally routable address in a socket table becomes a documentation one (RFC
|
||||
5737 `203.0.113.0/24`, RFC 3849 `2001:db8::/32`) through an idempotent column substitution — ports,
|
||||
states, every other column and every space are verbatim, loopback/private/link-local/wildcard
|
||||
addresses are kept, and the IPv4-mapped `::ffff:` spelling survives — because a global IPv6 on
|
||||
Linux is EUI-64 (derived from the NIC MAC) and an inbound visitor's address is not this repo to
|
||||
publish: the same rule that scrubs a drive's serial (invariant 6). Those two tables are volatile by
|
||||
nature, so `tests/test_agent.py` asserts their **shape**, the ports the recorded configs claim, and
|
||||
the absence of a globally routable endpoint — never a particular row, which a re-run may lawfully
|
||||
change. What phase 28 stops at is the wire: `sources/host.py` passes **none** of the three into
|
||||
`/api/system/snapshot` and `store.py` records no series for a proxy or a socket, so the sections
|
||||
live in the agent's own document and reach no page until phase 29's. What the dashboard does read
|
||||
today is their **error lines**, through `collectors.ADDON_ERROR_PREFIXES`, so a host with no proxy
|
||||
config stays out of `/system`'s `degraded` (invariant 3).
|
||||
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
|
||||
env var always wins. Defaults are safe except the sources, which default to unset. The auth
|
||||
block (phase 11) is `DASH_ADMIN_PASSWORD` (the **seed**, and `_env()` strips, so `VAR=` means
|
||||
@@ -301,11 +410,17 @@ Responsibilities, one line each:
|
||||
same sentence on both pages and only the `timeout` differs (the disks poll has to outlast
|
||||
the agent's smartctl pass, and `None` is translated into httpx's sentinel because a
|
||||
literal `None` means *no timeout at all*). Agent `errors`
|
||||
become the payload's `degraded` **except** the `smart:` add-on lines — a host without
|
||||
smartmontools or device read access is unconfigured, not broken (invariant 3); they stay
|
||||
in `agent.errors` for the `/disks` page to explain — which since phase 22 is only the
|
||||
device-read case, because a host whose agent said `smartctl` is *missing* is not listed on
|
||||
that page at all (`api._disk_visible`).
|
||||
become the payload's `degraded` **except** the add-on lines named by
|
||||
`ADDON_ERROR_PREFIXES` (`smart`, `haproxy`, `nginx`, `sockets`) — a host without smartmontools,
|
||||
without device read access, without a proxy config, or whose kernel socket table did not answer
|
||||
is unconfigured, not broken (invariant 3), and `degraded` is `alerts.py`'s input, so a fleet-wide
|
||||
agent upgrade would otherwise put an overview WARN on every box that never ran haproxy or nginx.
|
||||
Those lines stay verbatim in `agent.errors` for the page that owns the add-on — `/disks` explains
|
||||
the `smart:` ones (and since phase 22 a host whose agent said `smartctl` is *missing* is not
|
||||
listed on that page at all, `api._disk_visible`), phase 29's gateway page reads the
|
||||
`haproxy:`/`nginx:`/`sockets:` ones the same way.
|
||||
**A new agent add-on section adds its prefix here in the same commit**, or its absence-report
|
||||
becomes a red banner on a page that has nothing to say about it.
|
||||
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
|
||||
(`gpu.busy_pct@<index>`). `agent_tls_fields()` is the phase-15 passthrough: a TLS agent's
|
||||
`tls`/`bundle_rev` ride into the payload's existing `agent` block, and a token-mode snapshot
|
||||
@@ -979,7 +1094,7 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
| ----------------- | ------------------ | -------------------------------------------------------------------- |
|
||||
| `driveripper` | `driveripper-ops` | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards |
|
||||
| `deskwork` | `deskwork-ops` | Strix Halo (Radeon 8060S iGPU), gufo `:8731` |
|
||||
| `gateway` | `gateway-ops` | router, no GPU |
|
||||
| `gateway` | `gateway-ops` | router, no GPU, the fleet's ingress: haproxy (`/etc/haproxy/haproxy.cfg`, mode `0644`: six `frontend`s on `:::<port> v4v6` → six `backend`s, one `server <name> <domain>:<port> check inter 5s` each — the file the agent's `haproxy` section parses and `tests/fixtures/gateway_haproxy.cfg` captures) and nginx (`/etc/nginx/nginx.conf` plus `conf.d/http.conf` vhosts on `4443`/`8443` and `stream.d/stream.conf`'s `map $ssl_preread_server_name $targetBackend` → `proxy_pass $targetBackend` on `443` — the tree the agent's `nginx` section parses) |
|
||||
|
||||
- **Transport is direct dial: dev is the prod topology, repo-wide (phase 10 task 05).** The
|
||||
designed shape is one `dashboards-agent` **systemd service per host**, installed by `ops`
|
||||
@@ -1023,14 +1138,41 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
`host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`,
|
||||
`llm:vllm`.
|
||||
**The fleet is mTLS since 2026-10-21 (phase 24 task 03), so the bearer-token transport is now
|
||||
history rather than the designed shape.** All three hosts run `agent/host_agent.py` **1.1.0**
|
||||
from an installed bundle, every unit carries `--cert-bundle /etc/dashboards-agent.certs` (plus
|
||||
history rather than the designed shape.** `gateway` runs `agent/host_agent.py` **1.2.0** (phase 28
|
||||
task 05, 2026-10-22) and `deskwork`/`driveripper` still run **1.1.0**; the spread costs nothing,
|
||||
because 1.2.0 is purely additive — the three proxy-config sections are new keys and a 1.1.0
|
||||
snapshot simply has none — so those two move at their next re-stage rather than on a flag day.
|
||||
The control that proves "purely additive" is `data/logs/verify-28-final.py`: it stashes the whole
|
||||
working tree, imports both `host_agent.py` files side by side, calls both `collect()`s against
|
||||
the same `--root`, and demands the 1.2.0 document equal the 1.1.0 document key-for-key with
|
||||
`haproxy`/`nginx`/`sockets` added — and 1.2.0 with `--no-proxy` equal it exactly.
|
||||
All three are installed from a bundle, every unit carries `--cert-bundle /etc/dashboards-agent.certs` (plus
|
||||
`RuntimeDirectory=dashboards-agent`, and still its `--token-file` line), and `HOST_TARGETS` is
|
||||
`"driveripper=https://10.1.0.126:9120 deskwork=https://10.1.0.51:9120
|
||||
gateway=https://10.1.1.27:9120"` — **no `http://` entry and no userinfo anywhere**: the
|
||||
certificate is the credential (the pre-flip `.env.dev` is backed up in `data/env-backup/`). `agent.tls: true` + `agent.bundle_rev: 2` in
|
||||
every payload, `deployed_bundle_rev == bundle_rev` for all three on `/certs`, and
|
||||
`./scripts/dev-server.sh status` reads the nine sources `ok`. The per-host order was the safety:
|
||||
`./scripts/dev-server.sh status` reads the nine sources `ok`. **Phase 28 moved the gateway to
|
||||
1.2.0 on 2026-10-22 and only the agent file moved.** Task 04 read it read-only first
|
||||
(`data/logs/verify-28-task04.py` → `28-task04-probe.out`): 1.1.0, `agent.tls: true` + `bundle_rev 2`,
|
||||
every pre-1.2.0 section answering with live numbers and **no** `haproxy`/`nginx`/`sockets` key —
|
||||
the control that made the install provably additive. Task 05 then ran the phase-24 procedure on
|
||||
that one host (re-stage `/home/ops/agent-install/` first with `*-20261022-backup` snapshots, gap 10;
|
||||
`sudo /home/ops/agent-install/deploy/install-agent.sh --bind 10.1.1.27 --port 9120` with the
|
||||
arguments read back off `systemctl cat`; `systemctl restart dashboards-agent`) and verified it two
|
||||
ways: the CA-only `--status` probe answers `version 1.2.0` on `/health` and still refuses
|
||||
`/snapshot` with `client certificate required`, and the dashboard's own mTLS dial
|
||||
(`data/logs/verify-28-task05.py` → `28-task05-verify.out`) reads `agent_version 1.2.0` with all
|
||||
three sections `available: true` (6 frontends all routed / 6 backends, 11 nginx servers and the
|
||||
`$targetBackend` map, 17 listeners including 443) beside every pre-1.2.0 section intact and
|
||||
`host:gateway`/`disks:gateway` `ok`. That listener count is a reading, not a fact: the same probe
|
||||
re-run on 2026-10-22 for this phase's completion pass (`data/logs/verify-28-final.py` →
|
||||
`28-finalpass-live.out`) answered **19** listeners with the config halves unchanged (6/6, 11
|
||||
servers) — `/proc/net/tcp` is what a busy gateway writes every second, which is exactly why no
|
||||
test asserts a row of it. The one fact from that transcript worth carrying to the next
|
||||
host: the bundle **on disk** — not the `--cert-bundle` flag — is what puts the TLS lines into the
|
||||
unit render, so an upgrade re-run that omits the flag is the silent path (no guard refusal) and
|
||||
`systemctl cat` came back byte-identical (`26bd812…`); only `host_agent.py` changed. The per-host order was the safety:
|
||||
install agent 1.1.0 + bundle → `systemctl restart` → the CA-only `--status` probe answers →
|
||||
*then* that host's URL flips, so no source was red longer than its own host's restart.
|
||||
`data/logs/NOTES-24-rollout.md` is the transcript (sha256 of what was staged where, the unit
|
||||
@@ -1112,8 +1254,9 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
filter removal, nor the re-run guard — running it would have replaced driveripper's `User=root`
|
||||
+ `CapabilityBoundingSet` unit with the snapshot's capability-less one, **silently**. That trap
|
||||
is now closed on all three hosts: `/home/ops/agent-install/` is `ops`-owned, so the repo's
|
||||
`deploy/install-agent.sh` (sha256 `ffbcd91f…`) and `agent/host_agent.py` (**1.1.0**) were staged
|
||||
over it without root, and each snapshot was left beside the grant as
|
||||
`deploy/install-agent.sh` (sha256 `ffbcd91f…`) and `agent/host_agent.py` (**1.1.0**, re-staged as
|
||||
**1.2.0** on `gateway` in phase 28) were staged over it without root, and each snapshot was left
|
||||
beside the grant as
|
||||
`install-agent.sh.oct18-backup`, which sudo will not run. **Re-staging is therefore the first
|
||||
step of any install, not an optional one** — the grant is only as current as the file it points
|
||||
at. The `systemctl restart *` grant on those two hosts is the other half of an unattended
|
||||
@@ -1233,9 +1376,9 @@ uv run python scripts/dev-certs-export.py --url http://127.0.0.1:8090 \
|
||||
# GET /api/certs/<name>/bundle and writes
|
||||
# data/certs/<name>.bundle at 0600 — mints nothing
|
||||
|
||||
uv run pytest # hermetic: 1549 passed + 15 skipped (11 live,
|
||||
uv run pytest # hermetic: 1650 passed + 15 skipped (11 live,
|
||||
# 4 store_perf), no network
|
||||
uv run pytest --cov # coverage gate (98.94%; floor 94%)
|
||||
uv run pytest --cov # coverage gate (98.97%; floor 94%)
|
||||
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
|
||||
uv run pytest -m store_perf --store-perf # opt-in: the store hot-path timings over the seeded
|
||||
# store, priced against `store_perf.BUDGETS` (the cached
|
||||
@@ -1270,6 +1413,11 @@ uv run ruff check src tests agent && uv run pyright
|
||||
SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
|
||||
# ... plus smartctl per physical disk (needs device read
|
||||
# access: plain smartctl, else `sudo -n`; scrubs serials)
|
||||
./scripts/capture-gateway-config.sh # ... plus the gateway's proxy tree and socket tables:
|
||||
# haproxy.cfg, nginx.conf + its glob includes, and
|
||||
# /proc/net/tcp{,6} over $GATEWAY_SSH (gateway-ops);
|
||||
# world-readable paths only, and a failed or empty read
|
||||
# never replaces a good fixture
|
||||
|
||||
bash .agents/validate.sh # the phase gate: the four checks above (pytest
|
||||
# --cov --cov-fail-under=94, ruff, pyright, plus a
|
||||
@@ -2342,7 +2490,9 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
||||
(phase 15 tasks 01-05 landed 2026-10-20; phase 24 task 03 rolled the fleet out 2026-10-21).
|
||||
The rollout is done and verified: gateway, deskwork and driveripper each run `agent/host_agent.py`
|
||||
**1.1.0** with `/etc/dashboards-agent.certs` installed `0640 root:dashboards-agent`, their units
|
||||
carry `--cert-bundle` + `RuntimeDirectory=dashboards-agent` (and keep `--token-file`),
|
||||
carry `--cert-bundle` + `RuntimeDirectory=dashboards-agent` (and keep `--token-file`) — the
|
||||
gateway has since moved to **1.2.0** for phase 28's proxy-config sections on the same unit and
|
||||
the same bundle, see the fleet record above —
|
||||
`HOST_TARGETS` is `https://` for all three with no userinfo, `agent.bundle_rev` reads `2` on
|
||||
every host and equals what `/api/certs` issued, and the nine sources are `ok`. The transcript is
|
||||
`data/logs/NOTES-24-rollout.md`, and task 04 re-read the whole end state **read-only** afterwards
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
# Homelab dashboards — one image, three dashboards, one port.
|
||||
# Homelab dashboards — one image, four dashboards and their overview, one port.
|
||||
#
|
||||
# podman build -t dashboards .
|
||||
# podman run --rm -p 8090:8090 --env-file dashboards.env \
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# Homelab Dashboards
|
||||
|
||||
Four dashboards, one domain, one process, one SQLite file:
|
||||
Four dashboards and the overview page that points at them — one domain, one process, one SQLite
|
||||
file:
|
||||
|
||||
| Page | What it answers | Source |
|
||||
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
|
||||
@@ -90,7 +91,8 @@ that exact line. upsd is dialled at
|
||||
## The host agent
|
||||
|
||||
`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
|
||||
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one
|
||||
`/sys/block`, `/sys/class/drm` and `statvfs` — and, on the host that fronts the fleet, the
|
||||
haproxy/nginx configs and `/proc/net/tcp{,6}` — and answers `GET /snapshot` (everything, one
|
||||
JSON document; `?top=N` trims the process lists), `GET /info` (slow-changing identity: OS,
|
||||
kernel, CPU model, GPU list), `GET /schema` (the schema version plus this document) and
|
||||
`GET /health` (the one route that needs no credential) as JSON, with a plain HTML status
|
||||
@@ -147,6 +149,81 @@ one installed before phase 09 has no SMART pass to run. Opt out silently with `-
|
||||
The drive truth this is all developed against lives in `tests/fixtures/smartctl_driveripper_*.txt`:
|
||||
16 real drives, recorded with serials scrubbed, each one parsed by a test.
|
||||
|
||||
The `haproxy` section is the same kind of best-effort add-on, for the host that fronts the fleet.
|
||||
The agent reads `/etc/haproxy/haproxy.cfg` (`--haproxy-config` names another path) and reports
|
||||
what the config says: every `frontend` block with its name, each `bind` address:port exactly as
|
||||
written (`:::22` is the address `::` on port `22`) and the backend names it routes to
|
||||
(`default_backend`, plus every named `use_backend`), and every `backend` block with each `server`'s
|
||||
name, target and port — `port: null` when the line names none, never `0`. It interprets nothing:
|
||||
the routing between the two lists is the **names**, so joining them is the page's job (the gateway
|
||||
page is phase 29) and the agent never evaluates a rule. `available` answers three questions the way `smartctl_available` does —
|
||||
`true` read and parsed, `false` no config at that path (this host runs no haproxy), `null` a file
|
||||
is there but could not be read or understood — and each non-`true` answer costs exactly **one**
|
||||
`errors` line for the pass. Like the SMART lines, those lines never make the *system* page amber:
|
||||
a host that never installed haproxy is unconfigured, not broken, and the page that owns the add-on
|
||||
reads the sentence out of `agent.errors`. A failing config read cannot fail `/snapshot` or
|
||||
`/health`, and `--no-proxy` drops the section and the line both. The parser is pinned to
|
||||
`tests/fixtures/gateway_haproxy.cfg`, the verbatim capture from gateway. Neither path is an env
|
||||
var and `deploy/install-agent.sh` passes neither flag — the unit it writes runs on the defaults —
|
||||
so a box whose configs live elsewhere means adding the flag to `$UNIT` after installing.
|
||||
|
||||
The `nginx` section is that add-on's other half, and it reads more than one file: the main config
|
||||
(`/etc/nginx/nginx.conf`, `--nginx-config` names another) **plus the files its own `include` lines
|
||||
name**, one level deep, because that is where nginx keeps its vhosts
|
||||
(`include /etc/nginx/conf.d/*.conf`) and its SNI routing (`/etc/nginx/stream.d/*.conf`). It
|
||||
reports every `server` block with its `listen` address:port pairs (`[::]:8443` is the address `::`
|
||||
on port `8443`; a bare `listen 80` has no address, and the `ssl`/`proxy_protocol` flags are not
|
||||
this section's business), its `server_name` list (the catch-all `_` included verbatim — "any
|
||||
domain" is what the page says about it), and every `proxy_pass`; every `upstream` block with its
|
||||
`server` targets; and every `map` block with its keys. A `proxy_pass $var` is served with the
|
||||
variable's name and `target: null`: nginx resolves that per connection from the `map` that defines
|
||||
it, so the agent reports both facts **unmerged** and leaves the joining to a page — a merged target
|
||||
would be a number the wire never published. `files` names the main config and the included
|
||||
documents actually merged, and the include pass is bounded (8 files / 1 MB): what it could not
|
||||
take is said in one `errors` line naming the count, and the section still reads `true`. The same
|
||||
three-way `available`, the same one line per arm, the same rule that a missing, unreadable or
|
||||
unparseable config degrades only this section — and `--no-proxy` drops the whole family with it,
|
||||
in silence.
|
||||
|
||||
The `sockets` section is that add-on's live half: what is actually open and carrying traffic, as
|
||||
opposed to what a config claims. The agent reads `/proc/net/tcp` and `/proc/net/tcp6` and keeps
|
||||
only the two states a gateway page can draw pipes from — kernel state `0A` LISTEN into `listens`
|
||||
(`{family, addr, port}`) and `01` ESTABLISHED into `established` (`{family, local, remote}`, each
|
||||
end an `{addr, port}`). Every other state is a socket in transit: a `TIME_WAIT` row is a
|
||||
connection that stopped being true before the page painted, and a proxy accumulates thousands of
|
||||
them, so they are skipped without a word. Each table is read through **at most 4096 lines**
|
||||
(`/proc`'s own header among them, so 4095 sockets per table); a longer one reports the rows it read
|
||||
and costs one `errors` line naming the bound, with `available` still `true`. `addr` is the address
|
||||
itself, unrolled from `/proc`'s per-32-bit-word little-endian spelling — `0.0.0.0` and `::` keep
|
||||
the literal wildcard form, because "open on every address" is the fact the inbound join keys on —
|
||||
and a word that is not an address is `null`, never a guess. The tables are read independently: a
|
||||
kernel without IPv6 has no `tcp6` file, which costs nothing and no line, while a table that is
|
||||
there and denied costs one. `available` is `true` when either table answered and `null` when
|
||||
neither did, with one line saying why; there is no `false`, because a table that did not answer is
|
||||
"no answer" and not "this host has no sockets". Nothing here is joined to a config: a socket does
|
||||
not know its domain or its backend, so phase 29's gateway page will be what joins these ports to
|
||||
the `haproxy` and `nginx` sections — where a socket no frontend claims stays visible as one. Until
|
||||
that page exists the three sections live only in the agent's own `/snapshot` document: no `/api`
|
||||
payload grows them, no `sample` row is stored for them, and nothing charts them.
|
||||
|
||||
All three parsers are pinned to captures rather than to descriptions, and
|
||||
`./scripts/capture-gateway-config.sh` re-records them over the `GATEWAY_SSH` alias (default
|
||||
`gateway-ops`): `/etc/haproxy/haproxy.cfg`, `/etc/nginx/nginx.conf` plus every document its own
|
||||
glob `include` lines name (`gateway_nginx_conf_d_*.conf`, `gateway_nginx_stream_d_*.conf`), and
|
||||
`/proc/net/tcp{,6}`. No sudo — those paths are world-readable, which is exactly what the agent
|
||||
reads as itself — and a read that fails, or answers empty, leaves the existing fixture alone and
|
||||
exits 1, so a laptop with no route to the gateway cannot turn six captures into six empty files.
|
||||
The configs are not rewritten (their domains are the ingress map, and key material appears only
|
||||
as the paths nginx is told to read); the socket tables lose only their identifiers, in line with
|
||||
the rule that scrubs a drive's serial: a globally routable address becomes a documentation one
|
||||
(RFC 5737 / RFC 3849), because a global IPv6 on Linux is EUI-64 and an inbound visitor's address
|
||||
is not ours to publish. Every port, state, column and width in those tables is verbatim, and
|
||||
those files are volatile by nature — a proxy in service has a different connection set every
|
||||
second — so the tests assert shape and the ports the configs claim, never a particular row.
|
||||
`AGENT_VERSION` is **1.2.0**, the version that ships all three sections; a host still running
|
||||
1.1.0 answers `GET /schema` with a block that names none of them, and its snapshot simply has no
|
||||
such keys.
|
||||
|
||||
`deploy/install-agent.sh --fetch-bundle <url> --cert-name <name> --session <token>` is the same
|
||||
TLS install with the download done inside the registration run: it asks
|
||||
`GET <url>/api/certs/<name>/bundle`, writes the answer to a temp file, and hands it to the
|
||||
@@ -338,6 +415,20 @@ for a bundle install, so step 3 is a copy rather than a guess. A host left on `h
|
||||
working over its bearer token — the rollout is per host, and one revoked or expired certificate
|
||||
never blinks the others.
|
||||
|
||||
**Upgrading an already-registered host to a newer agent is one file, not a rollout.** Re-stage
|
||||
`agent/host_agent.py` into `/home/ops/agent-install/agent/` (the snapshot-first rule applies: a path
|
||||
grant is only as current as the file it names, so copy what you are replacing to
|
||||
`host_agent.py-<date>-backup` first — a name the sudo grant does not name, so sudo will not run it),
|
||||
run the granted path with **the unit's own arguments and nothing else**
|
||||
(`sudo /home/ops/agent-install/deploy/install-agent.sh --bind <addr> --port 9120`), and restart.
|
||||
No `--cert-bundle`, no `--fetch-bundle`, no `--force`: the bundle already on disk is what the unit
|
||||
render reads — the flag only replaces the file — so the render equals the installed unit, the re-run
|
||||
guard prints nothing, and `/etc/dashboards-agent.certs` and the token file keep their bytes. Then
|
||||
the same two checks the registration uses: the CA-only `--status` probe (with `--bind`) answers
|
||||
`{"ok": true, …, "version": "1.2.0"}` on `/health`, and the dashboard's own `https://` dial answers
|
||||
whatever the new version added — `agent_version` plus the `haproxy` / `nginx` / `sockets` sections
|
||||
for a 1.2.0 agent — beside the sections it always had.
|
||||
|
||||
Two things the first full rollout learned, both cheap to hit and expensive to guess at:
|
||||
|
||||
* **A CA row older than the identifier fix must be rotated before any bundle is installed.** A
|
||||
@@ -953,6 +1044,11 @@ uv run ruff check src tests agent && uv run pyright # lint + types
|
||||
SMART_HOSTS="host-a-ops host-b-ops" ./scripts/capture-fixtures.sh
|
||||
# ... and smartctl per physical disk: plain smartctl,
|
||||
# `sudo -n` if that is denied; scrubs serials/WWNs
|
||||
./scripts/capture-gateway-config.sh # ... and the gateway's proxy tree + socket tables:
|
||||
# haproxy.cfg, nginx.conf + its glob includes,
|
||||
# /proc/net/tcp{,6} over $GATEWAY_SSH (gateway-ops).
|
||||
# World-readable paths only; a failed read never
|
||||
# replaces a good fixture
|
||||
bash .agents/validate.sh # all of the above + `node --check` on every page
|
||||
# script + repo hygiene, with the 94% coverage floor
|
||||
```
|
||||
|
||||
+1070
-6
File diff suppressed because it is too large
Load Diff
Executable
+352
@@ -0,0 +1,352 @@
|
||||
#!/usr/bin/env bash
|
||||
# Re-record the gateway's proxy configs and kernel socket tables into tests/fixtures/.
|
||||
#
|
||||
# ./scripts/capture-gateway-config.sh # dials the `gateway-ops` ssh alias
|
||||
# GATEWAY_SSH=lab-gw ./scripts/capture-gateway-config.sh
|
||||
#
|
||||
# What lands where — one fixture per remote document, in `tests/fixtures/`:
|
||||
#
|
||||
# gateway_haproxy.cfg cat /etc/haproxy/haproxy.cfg
|
||||
# gateway_nginx_main.conf cat /etc/nginx/nginx.conf
|
||||
# gateway_nginx_conf_d_<n>.conf cat /etc/nginx/conf.d/*.conf
|
||||
# gateway_nginx_stream_d_<n>.conf cat /etc/nginx/stream.d/*.conf
|
||||
# gateway_proc_net_tcp.txt cat /proc/net/tcp
|
||||
# gateway_proc_net_tcp6.txt cat /proc/net/tcp6
|
||||
#
|
||||
# These documents are the ground truth behind `agent/host_agent.py`'s `haproxy`, `nginx` and
|
||||
# `sockets` sections (phase 28) the way `tests/fixtures/nut_<ups>.vars` pins the UPS wire and
|
||||
# `smartctl_driveripper_<dev>.txt` pins the drives. The include globs are read **out of the
|
||||
# captured main config**, so a gateway that moves its vhosts to a third directory grows a third
|
||||
# fixture on the next run instead of keeping a fixture set that quietly stopped being the tree
|
||||
# the agent merges.
|
||||
#
|
||||
# Four rules, three of them `scripts/capture-fixtures.sh`'s:
|
||||
#
|
||||
# * **Verbatim, never hand-written** (AGENTS.md invariant 7). A fixture you typed is a test of
|
||||
# your memory. Each file carries a `#` provenance block naming the remote path it came from.
|
||||
# The header is comment-shaped because `#` is a comment in haproxy's config and in nginx's,
|
||||
# and a `/proc` line whose 4th field is not a kernel state is skipped by the socket parser —
|
||||
# provenance that costs no parser anything, which `tests/test_agent.py` pins by parsing the
|
||||
# fixtures exactly as they sit on disk.
|
||||
# * **The configs are not rewritten at all.** A domain is not an identifier to hide here: a
|
||||
# `server_name` and the `proxy_pass` target beside it *are* the ingress map the gateway page
|
||||
# exists to draw, and a scrubbed fixture is a fixture that tests a config this gateway does
|
||||
# not run. Key material appears in these files only as the paths the proxy is told to read
|
||||
# (`ssl_certificate_key /etc/letsencrypt/…`), which is a filename and not a key. No text
|
||||
# filter of any kind sits between the read and a config fixture —
|
||||
# `tests/test_deploy_scripts.py` runs this script against a fake remote holding a domain and
|
||||
# demands the bytes back.
|
||||
# * **A socket table keeps its shape and loses its identifiers** (AGENTS.md invariant 6, and
|
||||
# the rule `capture-fixtures.sh` already applies to a drive: "Identifiers never go in the
|
||||
# repo" — Serial, LU WWN, and the NVMe EUI-64 *which encodes the serial*). A Linux global
|
||||
# IPv6 is EUI-64: `2603:…:5054:ff:fe84:b2e0` is this box's NIC address, and the other thing
|
||||
# `/proc/net/tcp` carries is the public address of whoever was connected at the moment of
|
||||
# the capture, which is not ours to publish. So any address `ipaddress` calls globally
|
||||
# routable becomes a documentation address (RFC 5737 / RFC 3849) in first-seen order,
|
||||
# re-encoded through the kernel's own per-32-bit-word little-endian spelling. Loopback,
|
||||
# private, link-local, wildcard and already-documentation addresses are left exactly as they
|
||||
# were, so the run is idempotent and the fixture still meets every shape the parser has to
|
||||
# survive — `0.0.0.0`, `::`, `::ffff:a.b.c.d`, 32-hex-digit v6 words. Every port, state,
|
||||
# queue, uid, inode and column width is byte-for-byte what the kernel answered, and only the
|
||||
# two address *columns* are even looked at (a tx/rx queue pair is spelled like an address
|
||||
# and is not one). The tables are volatile anyway — a proxy in service has a different
|
||||
# connection set every second — so the tests assert shape, the two states kept, and the
|
||||
# ports the captured configs claim to bind. Never a particular row.
|
||||
# * **A bad read never replaces a good fixture.** Empty output, an ssh failure or a permission
|
||||
# denial prints one line naming the file and the reason, leaves whatever is on disk alone,
|
||||
# and exits 1 — the `slots_gufo.json` rule, so a laptop with no route to the gateway cannot
|
||||
# turn six verbatim captures into six empty files.
|
||||
#
|
||||
# No sudo, deliberately: haproxy's config is 0644, nginx's tree is 0644 and `/proc/net/tcp{,6}`
|
||||
# is world-readable, which is exactly what `dashboards-agent` reads as itself. A capture that
|
||||
# needed root would record more than the agent can see — the mistake the smartctl pass in
|
||||
# `capture-fixtures.sh` documents in the other direction.
|
||||
#
|
||||
# A file the host stops having is left in place rather than deleted: when the config moves,
|
||||
# remove the orphan fixture in the same commit that updates the parser tests naming it.
|
||||
set -uo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
FIX="$ROOT/tests/fixtures"
|
||||
GATEWAY_SSH="${GATEWAY_SSH:-gateway-ops}"
|
||||
mkdir -p "$FIX"
|
||||
|
||||
# `-n` matters: `capture_included` walks the remote listing on stdin, and an ssh that reads
|
||||
# stdin would eat the rest of that list and quietly capture one vhost file instead of all of them.
|
||||
SSH_OPTS=(-n -o BatchMode=yes -o ConnectTimeout=8)
|
||||
failed=0
|
||||
|
||||
usage() {
|
||||
cat <<'TXT'
|
||||
Usage: ./scripts/capture-gateway-config.sh [-h]
|
||||
|
||||
Re-records the gateway's proxy configs and kernel socket tables into tests/fixtures/:
|
||||
/etc/haproxy/haproxy.cfg, /etc/nginx/nginx.conf plus the documents its own glob includes
|
||||
name, and /proc/net/tcp{,6}. Every read goes over the `$GATEWAY_SSH` ssh alias (default
|
||||
`gateway-ops`) of a world-readable path, and a read that fails leaves the existing fixture
|
||||
alone and exits 1. The configs land unedited; in the socket tables a globally routable
|
||||
address becomes a documentation one and nothing else moves. See the block comment.
|
||||
TXT
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
-h|--help) usage; exit 0 ;;
|
||||
esac
|
||||
|
||||
# The socket tables' one edit, and the only place in this script that touches a captured byte.
|
||||
# Kept as a here-doc in the house style (capture-fixtures.sh embeds two); it is pinned by
|
||||
# running this script against a fake remote in tests/test_deploy_scripts.py.
|
||||
SOCKET_IDENTIFIERS_OUT=$(cat <<'PY'
|
||||
"""Move the globally-routable addresses in a /proc/net/tcp{,6} table to documentation space.
|
||||
|
||||
Every column, every port and every byte of spacing is the kernel's. What changes is an address
|
||||
`ipaddress` calls global, which becomes a documentation address in first-seen order (RFC 5737
|
||||
for v4, RFC 3849 for v6) re-encoded through the kernel's own per-32-bit-word little-endian
|
||||
spelling — so the replacement is exactly as wide as the original and an IPv4-mapped v6 row stays
|
||||
an IPv4-mapped v6 row. AGENTS.md invariant 6 is the reason: this repo's capture rule is that
|
||||
identifiers never go in tracked files, and a global IPv6 on Linux is EUI-64 (it encodes the
|
||||
NIC's MAC) while a visitor's public address belongs to the visitor. Loopback, private,
|
||||
link-local, multicast, wildcard and already-documentation addresses are returned untouched,
|
||||
which is what makes re-running this over its own output change nothing.
|
||||
"""
|
||||
|
||||
import ipaddress
|
||||
import re
|
||||
import sys
|
||||
|
||||
DOC_V4 = int(ipaddress.IPv4Address("203.0.113.0"))
|
||||
DOC_V6 = int(ipaddress.IPv6Address("2001:db8::"))
|
||||
|
||||
|
||||
def unroll(word: str) -> bytes:
|
||||
"""`/proc`'s per-word little-endian spelling -> the address's own bytes."""
|
||||
raw = bytes.fromhex(word)
|
||||
return b"".join(raw[i:i + 4][::-1] for i in range(0, len(raw), 4))
|
||||
|
||||
|
||||
def roll(addr: ipaddress.IPv4Address | ipaddress.IPv6Address) -> str:
|
||||
"""The inverse, in the kernel's uppercase."""
|
||||
packed = addr.packed
|
||||
return "".join(packed[i:i + 4][::-1].hex().upper() for i in range(0, len(packed), 4))
|
||||
|
||||
|
||||
class Replacer:
|
||||
def __init__(self) -> None:
|
||||
self.counters: dict[int, int] = {4: 0, 6: 0}
|
||||
self.mapping: dict[str, str] = {}
|
||||
self.replacements = 0
|
||||
|
||||
def documentation(self, version: int) -> str:
|
||||
self.counters[version] += 1
|
||||
if version == 4:
|
||||
return str(ipaddress.IPv4Address(DOC_V4 + self.counters[version]))
|
||||
return str(ipaddress.IPv6Address(DOC_V6 + self.counters[version]))
|
||||
|
||||
def column(self, token: str) -> str:
|
||||
"""One `<word>:<port>` column: rewrite the word, keep the port and its spelling."""
|
||||
word, _, port = token.rpartition(":")
|
||||
if len(word) not in (8, 32) or any(c not in "0123456789abcdefABCDEF" for c in word):
|
||||
return token
|
||||
try:
|
||||
addr = ipaddress.ip_address(unroll(word))
|
||||
except ValueError:
|
||||
return token
|
||||
# Only IPv6Address has `ipv4_mapped`; a v4 word is its own subject.
|
||||
mapped = getattr(addr, "ipv4_mapped", None)
|
||||
subject = mapped if mapped is not None else addr
|
||||
if not subject.is_global:
|
||||
return token
|
||||
key = str(subject)
|
||||
if key not in self.mapping:
|
||||
self.mapping[key] = self.documentation(subject.version)
|
||||
replacement = self.mapping[key]
|
||||
self.replacements += 1
|
||||
# A v4-mapped row keeps its mapped spelling, so the row that exercises that branch of
|
||||
# the agent's decoder still exercises it after the capture.
|
||||
if mapped is not None:
|
||||
new: ipaddress.IPv4Address | ipaddress.IPv6Address = ipaddress.IPv6Address(
|
||||
f"::ffff:{replacement}")
|
||||
else:
|
||||
new = ipaddress.ip_address(replacement)
|
||||
return f"{roll(new)}:{port}"
|
||||
|
||||
def line(self, text: str) -> str:
|
||||
"""Rewrite columns 1 and 2 only — the two address columns of a socket table."""
|
||||
spans = [(m.start(), m.end(), m.group(0)) for m in re.finditer(r"\S+", text)]
|
||||
if len(spans) < 4:
|
||||
return text
|
||||
out = []
|
||||
last = 0
|
||||
for index, (start, end, token) in enumerate(spans):
|
||||
out.append(text[last:start])
|
||||
out.append(self.column(token) if index in (1, 2) else token)
|
||||
last = end
|
||||
out.append(text[last:])
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
replacer = Replacer()
|
||||
with open(argv[1], encoding="utf-8", errors="replace") as handle:
|
||||
for line in handle:
|
||||
sys.stdout.write(replacer.line(line))
|
||||
if replacer.replacements:
|
||||
print(f" ({replacer.replacements} globally-routable address column(s) moved to "
|
||||
f"documentation space; {len(replacer.mapping)} distinct endpoint(s))",
|
||||
file=sys.stderr)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv))
|
||||
PY
|
||||
)
|
||||
|
||||
# header <kind> <remote path> — the provenance block every fixture carries.
|
||||
header() {
|
||||
local kind="$1" path="$2"
|
||||
echo "# Verbatim $path from $GATEWAY_SSH, read read-only over the ssh key by"
|
||||
echo "# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the"
|
||||
echo "# deployed upstream, never guess its shape)."
|
||||
case "$kind" in
|
||||
config)
|
||||
echo "# Nothing here is rewritten: it is this fleet's ingress map, and the domains in it are"
|
||||
echo "# the data — an inbound name and the target it routes to are what these config sections"
|
||||
echo "# exist to report. Key material appears only as the file paths the proxy reads."
|
||||
;;
|
||||
sockets)
|
||||
echo "# One line per TCP socket, in the kernel's own spelling. The only edit is the repo's"
|
||||
echo "# identifier rule (invariant 6, the drive-serial scrub in capture-fixtures.sh): a"
|
||||
echo "# globally routable address became a documentation one (RFC 5737 / RFC 3849), because a"
|
||||
echo "# global IPv6 here is EUI-64 and a visitor's address is not ours to publish. Every port,"
|
||||
echo "# state, queue, uid, inode and column width is verbatim, as are the loopback, private"
|
||||
echo "# and wildcard addresses the gateway's own frontends use."
|
||||
echo "# Volatile by nature: a proxy in service has a different connection set every second, so"
|
||||
echo "# a diff here on re-capture is expected. The tests assert shape, the two states the agent"
|
||||
echo "# keeps, and the ports the captured configs claim to bind — never a particular row."
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# capture <fixture> <kind> <remote path> — one ssh call, one document, written only if the read
|
||||
# answered something. The remote command is a single `cat`: `scp` cannot read /proc at all (a
|
||||
# procfs file reports size 0, so scp copies an empty file), and one mechanism for six fixtures
|
||||
# is one mechanism to pin.
|
||||
capture() {
|
||||
local fixture="$1" kind="$2" path="$3"
|
||||
local tmp="$FIX/.$fixture.tmp" err="$FIX/.$fixture.err"
|
||||
local rc=0
|
||||
ssh "${SSH_OPTS[@]}" "$GATEWAY_SSH" "cat $path" >"$tmp" 2>"$err" || rc=$?
|
||||
if [ "$rc" -ne 0 ] || [ ! -s "$tmp" ]; then
|
||||
local why
|
||||
why="$(head -n 1 "$err" 2>/dev/null)"
|
||||
[ -n "$why" ] || { [ "$rc" -eq 0 ] && why="answered an empty file" || why="ssh exited $rc"; }
|
||||
rm -f "$tmp" "$err"
|
||||
echo " ✗ $fixture — $GATEWAY_SSH:$path unreadable ($why); the fixture on disk is left alone"
|
||||
failed=1
|
||||
return 1
|
||||
fi
|
||||
rm -f "$err"
|
||||
{
|
||||
header "$kind" "$path"
|
||||
case "$kind" in
|
||||
sockets) python3 -c "$SOCKET_IDENTIFIERS_OUT" "$tmp" ;;
|
||||
*) cat "$tmp" ;;
|
||||
esac
|
||||
} >"$FIX/$fixture"
|
||||
rc=0
|
||||
rm -f "$tmp"
|
||||
echo " ✓ $fixture ($(wc -l <"$FIX/$fixture") lines from $GATEWAY_SSH:$path)"
|
||||
}
|
||||
|
||||
# capture_included <fixture prefix> <remote dir> — every `*.conf` the main config's include line
|
||||
# names, one fixture each. The prefix names the directory the way the parser spells it
|
||||
# (`gateway_nginx_conf_d_` for /etc/nginx/conf.d), the rest is the file's own name:
|
||||
# /etc/nginx/conf.d/http.conf → gateway_nginx_conf_d_http.conf.
|
||||
capture_included() {
|
||||
local prefix="$1" dir="$2"
|
||||
local listing="$FIX/.$prefix.ls" err="$FIX/.$prefix.ls.err"
|
||||
local rc=0
|
||||
ssh "${SSH_OPTS[@]}" "$GATEWAY_SSH" "ls $dir/*.conf" >"$listing" 2>"$err" || rc=$?
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
local why
|
||||
why="$(head -n 1 "$err" 2>/dev/null)"
|
||||
[ -n "$why" ] || { [ "$rc" -eq 0 ] && why="answered an empty list" || why="ssh exited $rc"; }
|
||||
rm -f "$listing" "$err"
|
||||
echo " ✗ ${prefix}* — $GATEWAY_SSH:$dir could not be listed ($why); the fixtures on disk are left alone"
|
||||
failed=1
|
||||
return 1
|
||||
fi
|
||||
rm -f "$err"
|
||||
local path base
|
||||
while IFS= read -r path; do
|
||||
[ -n "$path" ] || continue
|
||||
base="${path##*/}"
|
||||
# Parameter expansion and nothing else builds the name, so there is no text filter of any
|
||||
# kind between what the host said and what the fixture holds.
|
||||
capture "${prefix}${base%.conf}.conf" config "$path"
|
||||
done <"$listing"
|
||||
rm -f "$listing"
|
||||
}
|
||||
|
||||
# --- haproxy -----------------------------------------------------------------
|
||||
# The file `parse_haproxy_config` is pinned to: frontends on `:::<port> v4v6`, one
|
||||
# `server <name> <domain>:<port> check inter 5s` per backend, and a `use_backend` whose target
|
||||
# is a fetch expression — which is why a name is reported only where the config names one.
|
||||
capture gateway_haproxy.cfg config /etc/haproxy/haproxy.cfg
|
||||
|
||||
# --- nginx -------------------------------------------------------------------
|
||||
# The main config, then the documents its own include lines name one level deep: the same set
|
||||
# `Collector.nginx` merges. A literal include naming a file this host does not have (this
|
||||
# gateway's `include /etc/nginx/mime.types;`) is neither captured nor an error — absence of a
|
||||
# document the parser needs nothing from is the shape the agent already reports as "costs
|
||||
# nothing and no line".
|
||||
if capture gateway_nginx_main.conf config /etc/nginx/nginx.conf; then
|
||||
globs=()
|
||||
while IFS= read -r line; do
|
||||
probe="${line#"${line%%[![:space:]]*}"}" # leading whitespace off, for the two tests below
|
||||
case "$probe" in ''|'#'*) continue ;; esac
|
||||
case "$probe" in include*) ;; *) continue ;; esac
|
||||
spec="${probe#include}"
|
||||
spec="${spec%%;*}"
|
||||
spec="${spec//[[:space:]]/}"
|
||||
# Anything with a `*` is a glob include, and the agent merges it — so it gets a fixture, or
|
||||
# the report below says the script has no name for it. Silent skipping is what would let the
|
||||
# fixture set stop being the tree the agent reads.
|
||||
case "$spec" in *"*)"*) continue ;; esac # not a directive then: a fetch expression
|
||||
case "$spec" in *"*"*) globs+=("$spec") ;; esac
|
||||
done <"$FIX/gateway_nginx_main.conf"
|
||||
if [ ${#globs[@]} -eq 0 ]; then
|
||||
echo " ✗ gateway_nginx_*.conf — the captured main config names no glob include, so the"
|
||||
echo " vhost fixtures were not refreshed: has the tree the agent merges moved?"
|
||||
failed=1
|
||||
fi
|
||||
for spec in "${globs[@]}"; do
|
||||
case "$spec" in
|
||||
/etc/nginx/conf.d/*.conf) capture_included gateway_nginx_conf_d_ "${spec%/*}" ;;
|
||||
/etc/nginx/stream.d/*.conf) capture_included gateway_nginx_stream_d_ "${spec%/*}" ;;
|
||||
*)
|
||||
echo " ✗ $spec — an include directory this script has no fixture name for"
|
||||
echo " (add a case above and a fixture-pinned test in tests/test_agent.py)"
|
||||
failed=1
|
||||
;;
|
||||
esac
|
||||
done
|
||||
fi
|
||||
|
||||
# --- the live sockets --------------------------------------------------------
|
||||
# Both tables, always: `available` is decided across them, because a kernel built without IPv6
|
||||
# answers tcp and not tcp6 and that is absence rather than a fault (no line, the include rule).
|
||||
# Keeping only LISTEN and ESTABLISHED is what makes a capture of a busy gateway usable — the
|
||||
# TIME_WAIT rows in here are the ones the parser must skip, which is exactly what
|
||||
# tests/test_agent.py asserts against this file.
|
||||
capture gateway_proc_net_tcp.txt sockets /proc/net/tcp
|
||||
capture gateway_proc_net_tcp6.txt sockets /proc/net/tcp6
|
||||
|
||||
if [ "$failed" -ne 0 ]; then
|
||||
echo
|
||||
echo "capture FAILED — every fixture named above is unchanged on disk. Is $GATEWAY_SSH an ssh"
|
||||
echo "alias that answers (ssh $GATEWAY_SSH true), and are the paths still where they were?"
|
||||
exit 1
|
||||
fi
|
||||
echo "done — read the diff before committing the fixtures."
|
||||
@@ -36,6 +36,22 @@ class UpstreamError(Exception):
|
||||
self.status = status
|
||||
|
||||
|
||||
#: Agent add-on sections that report their own absence on purpose. Their `errors` lines are
|
||||
#: facts about a source this page does not draw — `smart:` belongs to `/disks`, `haproxy:` and
|
||||
#: `nginx:` to the gateway page (phase 29) — so they never make the **system** payload
|
||||
#: `degraded`, which is the overview's alert input. Without this a fleet-wide agent upgrade
|
||||
#: would put a red "agent degraded" banner and an overview WARN on every host that has no
|
||||
#: haproxy or nginx installed, which is invariant 3's failure mode with an upgrade's clothes on.
|
||||
#: The page that owns the add-on still reads the lines: they stay verbatim in `agent.errors`.
|
||||
#:
|
||||
#: **A new agent add-on section adds its prefix here in the same commit.** The list the agent's
|
||||
#: own tests filter by is pinned equal to this one
|
||||
#: (`tests/test_agent.py::test_the_addon_error_prefixes_are_the_same_list_the_dashboard_filters`),
|
||||
#: so a section that ships its absence-report without a prefix here fails instead of turning
|
||||
#: every host amber.
|
||||
ADDON_ERROR_PREFIXES: tuple[str, ...] = ("smart", "haproxy", "nginx", "sockets")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# System hosts
|
||||
# --------------------------------------------------------------------------- #
|
||||
@@ -221,12 +237,15 @@ async def fetch_host_snapshot(client: httpx.AsyncClient, target: Target,
|
||||
if body.get("errors"):
|
||||
# The agent degraded a section (a /proc read failed) but still answered. Say so
|
||||
# in the payload rather than presenting a partial snapshot as a complete one —
|
||||
# except the SMART add-on: a host without smartmontools or without device read
|
||||
# access is *unconfigured, not broken* (invariant 3), and its `smart:` lines must
|
||||
# not hold the system page amber forever. They stay in the snapshot payload
|
||||
# (`agent.errors`) where the /disks page explains them in its own panel.
|
||||
# except the add-on sections: a host without smartmontools or without device read
|
||||
# access, without a proxy config on it, or whose kernel published no socket table, is
|
||||
# *unconfigured, not broken* (invariant 3), and those lines must not hold the system page
|
||||
# amber forever. They stay in the snapshot payload (`agent.errors`) where the page that
|
||||
# owns the add-on explains them — `/disks` reads the `smart:` ones, phase 29's gateway
|
||||
# page reads the `haproxy:`/`nginx:`/`sockets:` ones.
|
||||
errors = body["errors"]
|
||||
core = ([line for line in errors if not str(line).startswith("smart")]
|
||||
core = ([line for line in errors
|
||||
if not str(line).lstrip().lower().startswith(ADDON_ERROR_PREFIXES)]
|
||||
if isinstance(errors, list) else errors)
|
||||
if core:
|
||||
result.payload["degraded"] = core
|
||||
|
||||
Vendored
+57
@@ -0,0 +1,57 @@
|
||||
# Verbatim /etc/haproxy/haproxy.cfg from gateway-ops, read read-only over the ssh key by
|
||||
# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the
|
||||
# deployed upstream, never guess its shape).
|
||||
# Nothing here is rewritten: it is this fleet's ingress map, and the domains in it are
|
||||
# the data — an inbound name and the target it routes to are what these config sections
|
||||
# exist to report. Key material appears only as the file paths the proxy reads.
|
||||
defaults
|
||||
log stdout local0 info
|
||||
mode tcp
|
||||
option tcplog
|
||||
timeout connect 10s
|
||||
timeout client 30s
|
||||
timeout server 30s
|
||||
timeout tunnel 3600s
|
||||
default-server init-addr last,libc,none
|
||||
|
||||
frontend haproxy-22
|
||||
bind :::22 v4v6
|
||||
default_backend gitea-2222
|
||||
|
||||
frontend haproxy-2000
|
||||
bind :::2000 v4v6
|
||||
default_backend sec-gitea-2000
|
||||
|
||||
frontend haproxy-5201
|
||||
bind :::5201 v4v6
|
||||
default_backend iperf3-5201
|
||||
|
||||
frontend haproxy-8080
|
||||
bind :::8080 v4v6
|
||||
default_backend unifi-external-8080
|
||||
|
||||
frontend haproxy-25565
|
||||
bind :::25565 v4v6
|
||||
default_backend testcraft-25565
|
||||
|
||||
frontend haproxy-25566
|
||||
bind :::25566 v4v6
|
||||
default_backend nimcraft-25566
|
||||
|
||||
backend gitea-2222
|
||||
server gitea reeseapps-gitea.reeselink.com:2222 check inter 5s
|
||||
|
||||
backend sec-gitea-2000
|
||||
server sec-gitea stackexpected-gitea.reeselink.com:2222 check inter 5s
|
||||
|
||||
backend iperf3-5201
|
||||
server iperf3 web.reeselink.com:5201 check inter 5s
|
||||
|
||||
backend unifi-external-8080
|
||||
server unifi unifi-external.reeselink.com:8080 check inter 5s
|
||||
|
||||
backend testcraft-25565
|
||||
server mc1 minecraft.reeselink.com:25565 check inter 5s
|
||||
|
||||
backend nimcraft-25566
|
||||
server mc2 minecraft.reeselink.com:25566 check inter 5s
|
||||
+214
@@ -0,0 +1,214 @@
|
||||
# Verbatim /etc/nginx/conf.d/http.conf from gateway-ops, read read-only over the ssh key by
|
||||
# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the
|
||||
# deployed upstream, never guess its shape).
|
||||
# Nothing here is rewritten: it is this fleet's ingress map, and the domains in it are
|
||||
# the data — an inbound name and the target it routes to are what these config sections
|
||||
# exist to report. Key material appears only as the file paths the proxy reads.
|
||||
server {
|
||||
listen 4443 ssl proxy_protocol;
|
||||
listen [::]:4443 ssl proxy_protocol;
|
||||
server_name nextcloud.reeseapps.com;
|
||||
client_max_body_size 0;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/nextcloud.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/nextcloud.reeseapps.com/privkey.pem;
|
||||
|
||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||
add_header X-Content-Type-Options "nosniff" always;
|
||||
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||
|
||||
location / {
|
||||
proxy_pass http://nextcloud.reeselink.com:11000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8443 ssl;
|
||||
listen [::]:8443 ssl;
|
||||
server_name nextcloud.reeseapps.com;
|
||||
client_max_body_size 0;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/nextcloud.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/nextcloud.reeseapps.com/privkey.pem;
|
||||
|
||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||
add_header X-Content-Type-Options "nosniff" always;
|
||||
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||
|
||||
location / {
|
||||
proxy_pass http://nextcloud.reeselink.com:11001;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 4443 ssl proxy_protocol;
|
||||
listen [::]:4443 ssl proxy_protocol;
|
||||
server_name gitea.reeseapps.com;
|
||||
client_max_body_size 0;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/gitea.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/gitea.reeseapps.com/privkey.pem;
|
||||
|
||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||
add_header X-Content-Type-Options "nosniff" always;
|
||||
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||
|
||||
location / {
|
||||
proxy_pass http://gitea.reeselink.com:3000;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 4443 ssl proxy_protocol;
|
||||
listen [::]:4443 ssl proxy_protocol;
|
||||
server_name jellyfin.reeseapps.com;
|
||||
client_max_body_size 0;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/jellyfin.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/jellyfin.reeseapps.com/privkey.pem;
|
||||
|
||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||
add_header X-Content-Type-Options "nosniff" always;
|
||||
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||
|
||||
location / {
|
||||
proxy_pass http://jellyfin.reeselink.com:8096;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8443 ssl;
|
||||
listen [::]:8443 ssl;
|
||||
server_name alicia-ai-terminology.reeseapps.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/alicia-ai-terminology.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/alicia-ai-terminology.reeseapps.com/privkey.pem;
|
||||
|
||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||
add_header X-Content-Type-Options "nosniff" always;
|
||||
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:8081;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 4443 ssl proxy_protocol;
|
||||
listen [::]:4443 ssl proxy_protocol;
|
||||
server_name connors-legos.reeseapps.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/connors-legos.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/connors-legos.reeseapps.com/privkey.pem;
|
||||
|
||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||
add_header X-Content-Type-Options "nosniff" always;
|
||||
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||
|
||||
location / {
|
||||
proxy_pass http://bricktracker.reeselink.com:3333;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 4443 ssl proxy_protocol;
|
||||
listen [::]:4443 ssl proxy_protocol;
|
||||
server_name keycloak.reeseapps.com;
|
||||
client_max_body_size 0;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/keycloak.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/keycloak.reeseapps.com/privkey.pem;
|
||||
|
||||
location / {
|
||||
proxy_pass http://keycloak.reeselink.com:8080;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8443 ssl;
|
||||
listen [::]:8443 ssl;
|
||||
server_name admin.keycloak.reeseapps.com;
|
||||
client_max_body_size 0;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/admin.keycloak.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/admin.keycloak.reeseapps.com/privkey.pem;
|
||||
|
||||
location / {
|
||||
proxy_pass http://keycloak.reeselink.com:8080;
|
||||
proxy_set_header Host $host:$server_port;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header X-Forwarded-Port $server_port;
|
||||
proxy_set_header X-Forwarded-Host $host:$server_port;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 4443 ssl proxy_protocol;
|
||||
listen [::]:4443 ssl proxy_protocol;
|
||||
server_name emmaleaf.reeseapps.com;
|
||||
client_max_body_size 10m;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/emmaleaf.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/emmaleaf.reeseapps.com/privkey.pem;
|
||||
|
||||
location / {
|
||||
proxy_pass http://emmaleaf.reeselink.com:80;
|
||||
proxy_ssl_verify off;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
}
|
||||
}
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
# Verbatim /etc/nginx/nginx.conf from gateway-ops, read read-only over the ssh key by
|
||||
# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the
|
||||
# deployed upstream, never guess its shape).
|
||||
# Nothing here is rewritten: it is this fleet's ingress map, and the domains in it are
|
||||
# the data — an inbound name and the target it routes to are what these config sections
|
||||
# exist to report. Key material appears only as the file paths the proxy reads.
|
||||
user nginx;
|
||||
worker_processes auto;
|
||||
|
||||
error_log /var/log/nginx/error.log notice;
|
||||
pid /run/nginx.pid;
|
||||
|
||||
|
||||
events {
|
||||
worker_connections 1024;
|
||||
}
|
||||
|
||||
stream {
|
||||
include /etc/nginx/stream.d/*.conf;
|
||||
}
|
||||
|
||||
http {
|
||||
real_ip_header proxy_protocol;
|
||||
set_real_ip_from 127.0.0.1;
|
||||
set_real_ip_from 10.0.0.0/8;
|
||||
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
|
||||
log_format detailed '$remote_addr:$remote_port [$time_local] "$server_name" "$request" '
|
||||
'$status "$body_bytes_sent" "$http_referer" "$http_user_agent" '
|
||||
'$request_time "$upstream_addr" "$upstream_response_time" '
|
||||
'$ssl_protocol/$ssl_cipher';
|
||||
|
||||
access_log /var/log/nginx/access.log detailed;
|
||||
|
||||
sendfile on;
|
||||
tcp_nopush on;
|
||||
tcp_nodelay on;
|
||||
|
||||
keepalive_timeout 65;
|
||||
|
||||
gzip on;
|
||||
|
||||
server_tokens off;
|
||||
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
ssl_session_cache shared:SSL:10m;
|
||||
ssl_session_timeout 1d;
|
||||
|
||||
map $http_upgrade $connection_upgrade {
|
||||
default upgrade;
|
||||
'' close;
|
||||
}
|
||||
server {
|
||||
listen 80;
|
||||
listen [::]:80;
|
||||
server_name _;
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
|
||||
include /etc/nginx/conf.d/*.conf;
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
# Verbatim /etc/nginx/stream.d/stream.conf from gateway-ops, read read-only over the ssh key by
|
||||
# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the
|
||||
# deployed upstream, never guess its shape).
|
||||
# Nothing here is rewritten: it is this fleet's ingress map, and the domains in it are
|
||||
# the data — an inbound name and the target it routes to are what these config sections
|
||||
# exist to report. Key material appears only as the file paths the proxy reads.
|
||||
map $ssl_preread_server_name $targetBackend {
|
||||
default 127.0.0.1:4443;
|
||||
homepage.reeseapps.com reeseapps-homepage.reeselink.com:443;
|
||||
ntfy.reeseapps.com reeseapps-ntfy.reeselink.com:443;
|
||||
gitea.reeseapps.com reeseapps-gitea.reeselink.com:443;
|
||||
aipi.reeseapps.com reeseapps-litellm.reeselink.com:443;
|
||||
chatreesept.reeseapps.com reeseapps-openwebui.reeselink.com:443;
|
||||
keycloak.reeseapps.com reeseapps-keycloak.reeselink.com:443;
|
||||
forgejo.reeseapps.com reeseapps-forgejo.reeselink.com:443;
|
||||
brain.reeseapps.com reeseapps-brain-of-reese.reeselink.com:443;
|
||||
brain-of-code.reeseapps.com reeseapps-brain-of-code.reeselink.com:443;
|
||||
immich.reeseapps.com reeseapps-immich.reeselink.com:443;
|
||||
|
||||
stackexpected.com stackexpected.reeselink.com:443;
|
||||
gitea.stackexpected.com stackexpected-gitea.reeselink.com:443;
|
||||
keycloak.stackexpected.com stackexpected-keycloak.reeselink.com:443;
|
||||
nextcloud.stackexpected.com stackexpected-nextcloud.reeselink.com:443;
|
||||
hedgedoc.stackexpected.com stackexpected-hedgedoc.reeselink.com:443;
|
||||
wellsfamilycounseling.reeseapps.com wfc.reeselink.com:443;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443;
|
||||
listen [::]:443;
|
||||
|
||||
proxy_connect_timeout 1800s;
|
||||
proxy_timeout 1800s;
|
||||
proxy_protocol on;
|
||||
|
||||
proxy_pass $targetBackend;
|
||||
ssl_preread on;
|
||||
|
||||
resolver 10.1.0.1;
|
||||
}
|
||||
+58
@@ -0,0 +1,58 @@
|
||||
# Verbatim /proc/net/tcp from gateway-ops, read read-only over the ssh key by
|
||||
# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the
|
||||
# deployed upstream, never guess its shape).
|
||||
# One line per TCP socket, in the kernel's own spelling. The only edit is the repo's
|
||||
# identifier rule (invariant 6, the drive-serial scrub in capture-fixtures.sh): a
|
||||
# globally routable address became a documentation one (RFC 5737 / RFC 3849), because a
|
||||
# global IPv6 here is EUI-64 and a visitor's address is not ours to publish. Every port,
|
||||
# state, queue, uid, inode and column width is verbatim, as are the loopback, private
|
||||
# and wildcard addresses the gateway's own frontends use.
|
||||
# Volatile by nature: a proxy in service has a different connection set every second, so
|
||||
# a diff here on re-capture is expected. The tests assert shape, the two states the agent
|
||||
# keeps, and the ports the captured configs claim to bind — never a particular row.
|
||||
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
|
||||
0: 00000000:20FB 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9978 1 0000000000000000 100 0 0 10 0
|
||||
1: 00000000:08AE 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7797 1 0000000000000000 100 0 0 10 0
|
||||
2: 00000000:0050 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9974 1 0000000000000000 100 0 0 10 0
|
||||
3: 00000000:01BB 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9972 1 0000000000000000 100 0 0 10 0
|
||||
4: 00000000:115B 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9976 1 0000000000000000 100 0 0 10 0
|
||||
5: 1B01010A:23A0 00000000:0000 0A 00000000:00000000 00:00000000 00000000 994 0 9479 1 0000000000000000 100 0 0 10 0
|
||||
6: 1B01010A:BE2A B001010A:01BB 06 00000000:00000000 03:000013F5 00000000 0 0 0 3 0000000000000000
|
||||
7: 0100007F:115B 0100007F:818C 01 00000000:00000000 00:00000000 00000000 101 0 62860 1 0000000000000000 20 4 30 10 -1
|
||||
8: 1B01010A:BE2E B001010A:01BB 06 00000000:00000000 03:00001407 00000000 0 0 0 3 0000000000000000
|
||||
9: 1B01010A:23A0 0204A8C0:C626 06 00000000:00000000 03:00000820 00000000 0 0 0 3 0000000000000000
|
||||
10: 0100007F:115B 0100007F:C096 01 00000000:00000000 00:00000000 00000000 101 0 49558 1 0000000000000000 20 4 26 10 -1
|
||||
11: 0100007F:115B 0100007F:B502 06 00000000:00000000 03:00000841 00000000 0 0 0 3 0000000000000000
|
||||
12: 1B01010A:8290 A401010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 11850 1 0000000000000000 20 4 28 10 -1
|
||||
13: 1B01010A:01BB 017100CB:90FA 01 00000000:00000000 00:00000000 00000000 101 0 21476 1 0000000000000000 35 4 31 10 -1
|
||||
14: 1B01010A:23A0 0204A8C0:992E 06 00000000:00000000 03:00000C0A 00000000 0 0 0 3 0000000000000000
|
||||
15: 1B01010A:CB4C B001010A:01BB 06 00000000:00000000 03:00000ECB 00000000 0 0 0 3 0000000000000000
|
||||
16: 1B01010A:9C42 2700010A:01BB 06 00000000:00000000 03:00001259 00000000 0 0 0 3 0000000000000000
|
||||
17: 1B01010A:01BB 0100010A:B2AC 01 00000000:00000000 00:00000000 00000000 101 0 77631 1 0000000000000000 20 4 31 10 -1
|
||||
18: 1B01010A:01BB E600020A:BBE8 06 00000000:00000000 03:00000992 00000000 0 0 0 3 0000000000000000
|
||||
19: 0100007F:C99E 0100007F:115B 06 00000000:00000000 03:000016BE 00000000 0 0 0 3 0000000000000000
|
||||
20: 1B01010A:C1FE 2700010A:01BB 06 00000000:00000000 03:00000662 00000000 0 0 0 3 0000000000000000
|
||||
21: 1B01010A:01BB E600020A:B5C4 06 00000000:00000000 03:000005EE 00000000 0 0 0 3 0000000000000000
|
||||
22: 1B01010A:840E B001010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 21477 1 0000000000000000 20 4 30 10 -1
|
||||
23: 1B01010A:D22A 4E01010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 77632 1 0000000000000000 21 4 30 10 -1
|
||||
24: 1B01010A:01BB 017100CB:956A 01 00000000:00000000 00:00000000 00000000 101 0 62858 1 0000000000000000 29 4 30 10 -1
|
||||
25: 0100007F:115B 0100007F:B4F6 06 00000000:00000000 03:000005ED 00000000 0 0 0 3 0000000000000000
|
||||
26: 1B01010A:23A0 0204A8C0:8EE6 06 00000000:00000000 03:00001408 00000000 0 0 0 3 0000000000000000
|
||||
27: 1B01010A:01BB EC01010A:94DE 01 00000000:00000000 00:00000000 00000000 101 0 11849 1 0000000000000000 20 4 31 10 -1
|
||||
28: 0100007F:818C 0100007F:115B 01 00000000:00000000 00:00000000 00000000 101 0 62859 1 0000000000000000 21 4 24 10 -1
|
||||
29: 1B01010A:23A0 0204A8C0:D5DE 06 00000000:00000000 03:0000020F 00000000 0 0 0 3 0000000000000000
|
||||
30: 0100007F:115B 0100007F:B4F8 06 00000000:00000000 03:000005F6 00000000 0 0 0 3 0000000000000000
|
||||
31: 1B01010A:A828 7600010A:63DD 06 00000000:00000000 03:00000F7F 00000000 0 0 0 3 0000000000000000
|
||||
32: 0100007F:C096 0100007F:115B 01 00000000:00000000 00:00000000 00000000 101 0 49557 1 0000000000000000 20 4 24 10 -1
|
||||
33: 1B01010A:D444 FE00010A:2AF8 01 00000000:00000000 00:00000000 00000000 101 0 49559 1 0000000000000000 20 4 30 10 -1
|
||||
34: 1B01010A:9032 B001010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 11954 1 0000000000000000 20 4 30 10 -1
|
||||
35: 1B01010A:977E 2700010A:01BB 06 00000000:00000000 03:00000C72 00000000 0 0 0 3 0000000000000000
|
||||
36: 0100007F:115B 0100007F:B4EA 06 00000000:00000000 03:000005F0 00000000 0 0 0 3 0000000000000000
|
||||
37: 1B01010A:08AE 0204A8C0:8290 01 00000048:00000000 01:00000017 00000000 0 0 86478 4 0000000000000000 24 8 15 10 -1
|
||||
38: 1B01010A:9ED0 2700010A:01BB 06 00000000:00000000 03:0000170F 00000000 0 0 0 3 0000000000000000
|
||||
39: 1B01010A:0050 027100CB:843E 01 0000018C:00000000 01:00000E57 00000007 101 0 84306 2 0000000000000000 4864 4 31 1 7
|
||||
40: 1B01010A:CB5C B001010A:01BB 06 00000000:00000000 03:00000ED6 00000000 0 0 0 3 0000000000000000
|
||||
41: 1B01010A:9776 2700010A:01BB 06 00000000:00000000 03:00000875 00000000 0 0 0 3 0000000000000000
|
||||
42: 1B01010A:C59C FE00010A:2AF8 01 00000000:00000000 00:00000000 00000000 101 0 62861 1 0000000000000000 21 4 30 10 -1
|
||||
43: 1B01010A:9154 2700010A:01BB 06 00000000:00000000 03:00001572 00000000 0 0 0 3 0000000000000000
|
||||
44: 1B01010A:23A0 0204A8C0:9C42 06 00000000:00000000 03:00000FF3 00000000 0 0 0 3 0000000000000000
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
# Verbatim /proc/net/tcp6 from gateway-ops, read read-only over the ssh key by
|
||||
# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the
|
||||
# deployed upstream, never guess its shape).
|
||||
# One line per TCP socket, in the kernel's own spelling. The only edit is the repo's
|
||||
# identifier rule (invariant 6, the drive-serial scrub in capture-fixtures.sh): a
|
||||
# globally routable address became a documentation one (RFC 5737 / RFC 3849), because a
|
||||
# global IPv6 here is EUI-64 and a visitor's address is not ours to publish. Every port,
|
||||
# state, queue, uid, inode and column width is verbatim, as are the loopback, private
|
||||
# and wildcard addresses the gateway's own frontends use.
|
||||
# Volatile by nature: a proxy in service has a different connection set every second, so
|
||||
# a diff here on re-capture is expected. The tests assert shape, the two states the agent
|
||||
# keeps, and the ports the captured configs claim to bind — never a particular row.
|
||||
sl local_address remote_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
|
||||
0: 00000000000000000000000000000000:20FB 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9979 1 0000000000000000 100 0 0 10 0
|
||||
1: 00000000000000000000000000000000:08AE 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7799 1 0000000000000000 100 0 0 10 0
|
||||
2: 00000000000000000000000000000000:1451 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9181 1 0000000000000000 100 0 0 10 0
|
||||
3: 00000000000000000000000000000000:0050 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9975 1 0000000000000000 100 0 0 10 0
|
||||
4: 00000000000000000000000000000000:0016 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9179 1 0000000000000000 100 0 0 10 0
|
||||
5: 00000000000000000000000000000000:01BB 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9973 1 0000000000000000 100 0 0 10 0
|
||||
6: 00000000000000000000000000000000:115B 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9977 1 0000000000000000 100 0 0 10 0
|
||||
7: 00000000000000000000000000000000:63DE 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9184 1 0000000000000000 100 0 0 10 0
|
||||
8: 00000000000000000000000000000000:63DD 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9183 1 0000000000000000 100 0 0 10 0
|
||||
9: 00000000000000000000000000000000:07D0 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9180 1 0000000000000000 100 0 0 10 0
|
||||
10: 00000000000000000000000000000000:1F90 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9182 1 0000000000000000 100 0 0 10 0
|
||||
11: B80D0120000000000000000001000000:01BB 2377D3FDF349E0970000000003000000:C0AC 01 00000000:00000000 00:00000000 00000000 101 0 49556 1 0000000000000000 80 4 26 10 -1
|
||||
12: B80D0120000000000000000001000000:01BB 2377D3FDF349E0970000000003000000:A120 01 00000000:00000000 00:00000000 00000000 101 0 11953 1 0000000000000000 59 4 30 7 7
|
||||
+1662
-11
File diff suppressed because it is too large
Load Diff
@@ -249,6 +249,55 @@ async def test_real_degradations_still_show_when_smart_noise_is_alongside() -> N
|
||||
assert result.payload["degraded"] == ["temperatures: no hwmon sensors"]
|
||||
|
||||
|
||||
async def test_a_host_without_a_proxy_config_does_not_degrade_the_system_source() -> None:
|
||||
"""The proxy add-on's absence lines, same rule and same reason as `smart:` above: two of this
|
||||
fleet's three hosts run no haproxy and no nginx, so an agent upgrade must not put a
|
||||
`degraded` payload (and therefore an overview alert, `alerts.py` reads this key) on a box that
|
||||
never had either. The lines survive in `agent.errors` for the page that owns them — phase 29's
|
||||
gateway source reads them the way `/disks` reads `smart_errors`.
|
||||
|
||||
`collectors.ADDON_ERROR_PREFIXES` is the one list of these prefixes; this asserts every
|
||||
name by behaviour, so dropping any of them fails here, and `tests/test_agent.py` pins both
|
||||
the wording of the lines the agent actually writes and the equality of the two lists.
|
||||
"""
|
||||
missing_haproxy = ("haproxy: /etc/haproxy/haproxy.cfg not found — haproxy is not installed "
|
||||
"here, or its config lives elsewhere (--haproxy-config); --no-proxy "
|
||||
"opts out")
|
||||
missing_nginx = ("nginx: /etc/nginx/nginx.conf not found — nginx is not installed here, or "
|
||||
"its config lives elsewhere (--nginx-config); --no-proxy opts out")
|
||||
no_sockets = ("sockets: /proc/net/tcp not found, /proc/net/tcp6 not found — this host "
|
||||
"publishes no kernel socket table; --no-proxy opts out")
|
||||
body = agent_body()
|
||||
body["errors"] = [missing_haproxy, missing_nginx, no_sockets,
|
||||
"smart: smartctl 7.5 not found"]
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(200, json=body)
|
||||
|
||||
async with client_for(handler) as client:
|
||||
result = await fetch_host_snapshot(client, Target("gw", "http://agent.test"),
|
||||
HostCollector(RateEstimator()))
|
||||
assert "degraded" not in result.payload, "unconfigured is not broken (invariant 3)"
|
||||
for prefix in ("haproxy:", "nginx:", "sockets:"):
|
||||
assert any(prefix in str(e) for e in result.payload["agent"]["errors"]), \
|
||||
"the sentence survives for the page that explains it"
|
||||
|
||||
|
||||
async def test_a_real_degradation_shows_alongside_the_proxy_addon_noise() -> None:
|
||||
"""Filtering the add-ons must not swallow a genuine half-snapshot beside them."""
|
||||
body = agent_body()
|
||||
body["errors"] = ["nginx: /etc/nginx/nginx.conf is there but could not be read: boom",
|
||||
"temperatures: no hwmon sensors"]
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(200, json=body)
|
||||
|
||||
async with client_for(handler) as client:
|
||||
result = await fetch_host_snapshot(client, Target("gw", "http://agent.test"),
|
||||
HostCollector(RateEstimator()))
|
||||
assert result.payload["degraded"] == ["temperatures: no hwmon sensors"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# physical disks (the /disks source's fetch half)
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
@@ -1,4 +1,12 @@
|
||||
"""Contract tests for the scripts an operator runs: `deploy/` and `scripts/dev-*.sh`/`.py`.
|
||||
"""Contract tests for the scripts an operator runs: `deploy/`, `scripts/dev-*.sh`/`.py`, and
|
||||
`scripts/capture-gateway-config.sh`.
|
||||
|
||||
The capture script gets the same treatment as the installer because it can do the same kind of
|
||||
damage quietly: pointed at a gateway it cannot read, it is six empty files where six verbatim
|
||||
captures were, and every parser test that reads them then passes against nothing. So it is run
|
||||
here, against a stub `ssh` and a fake remote tree, and the assertions are about what lands on
|
||||
disk — including the one edit it makes (a globally routable socket address becomes documentation
|
||||
space) and the thing it never does (rewrite a config).
|
||||
|
||||
There is no shell test framework here — and adding one would buy nothing a subprocess does
|
||||
not — but phase 10 rests on a sentence that lives in **two** files: the sudoers grant in
|
||||
@@ -46,6 +54,7 @@ from dashboards.store import SampleStore
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
DEPLOY = ROOT / "deploy"
|
||||
SCRIPTS = ROOT / "scripts"
|
||||
INSTALLER = DEPLOY / "install-agent.sh"
|
||||
SUDOERS = DEPLOY / "agent-sudoers"
|
||||
BASH = shutil.which("bash") or "/bin/bash"
|
||||
@@ -2334,3 +2343,379 @@ def test_a_local_bundle_still_validates_with_its_own_flag_name(tmp_path: Path) -
|
||||
assert result.returncode == 2
|
||||
assert result.stderr.startswith("--cert-bundle"), result.stderr
|
||||
assert "--fetch-bundle" not in result.stderr
|
||||
|
||||
|
||||
# --- the gateway capture (phase 28 task 04) ------------------------------------------------
|
||||
|
||||
#: The script that records `tests/fixtures/gateway_*` — the ground truth the agent's `haproxy`,
|
||||
#: `nginx` and `sockets` parsers are pinned to. It has no Python-side coverage to speak of (the
|
||||
#: project's coverage source is `src/dashboards`), so the pins here **run it**, against a fake
|
||||
#: remote reached through a stubbed `ssh`, the way the installer sandbox stubs `install`.
|
||||
CAPTURE = SCRIPTS / "capture-gateway-config.sh"
|
||||
|
||||
#: The remote documents, and the fixture each one becomes. A new include directory on the gateway
|
||||
#: means a new entry here *and* a new `case` in the script — the script refuses silently-unknown
|
||||
#: globs for exactly this reason.
|
||||
CAPTURE_TARGETS = (
|
||||
"/etc/haproxy/haproxy.cfg",
|
||||
"/etc/nginx/nginx.conf",
|
||||
"/etc/nginx/conf.d/*.conf",
|
||||
"/etc/nginx/stream.d/*.conf",
|
||||
"/proc/net/tcp",
|
||||
"/proc/net/tcp6",
|
||||
)
|
||||
|
||||
#: The stub `ssh`. It drops the options the script passes, records every call, and answers `cat`
|
||||
#: and `ls` from `$FAKE_REMOTE` as if that directory were the remote filesystem's `/`. Nothing
|
||||
#: here can reach a network: the point is to exercise the script's own branches (the include
|
||||
#: globs, the refusal rule, the socket-table edit) with no gateway in sight.
|
||||
CAPTURE_SSH_STUB = """#!/bin/sh
|
||||
printf '%s\\n' "$*" >> "$STUB_LOG"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-o) shift 2 ;;
|
||||
-n) shift ;;
|
||||
*) break ;;
|
||||
esac
|
||||
done
|
||||
host="$1"; shift
|
||||
command="$*"
|
||||
if [ "${FAIL_SSH:-0}" = "1" ]; then
|
||||
echo "ssh: connect to host $host port 22: Connection refused" >&2
|
||||
exit 255
|
||||
fi
|
||||
case "$command" in
|
||||
"cat "*)
|
||||
path="${command#cat }"
|
||||
file="$FAKE_REMOTE$path"
|
||||
[ -f "$file" ] || { echo "cat: $path: No such file or directory" >&2; exit 1; }
|
||||
cat "$file"
|
||||
;;
|
||||
"ls "*)
|
||||
spec="${command#ls }"
|
||||
dir="${spec%/*}"
|
||||
glob="${spec##*/}"
|
||||
found=0
|
||||
for file in "$FAKE_REMOTE$dir"/$glob; do
|
||||
[ -e "$file" ] || continue
|
||||
found=1
|
||||
printf '%s\\n' "${file#"$FAKE_REMOTE"}"
|
||||
done
|
||||
[ "$found" = 1 ] || { echo "ls: cannot access '$spec': No such file or directory" >&2; exit 2; }
|
||||
;;
|
||||
*)
|
||||
echo "stub ssh: unsupported remote command: $command" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
"""
|
||||
|
||||
#: A gateway tree small enough to assert on whole, with the two shapes the parsers exist for: a
|
||||
#: `use_backend` whose target is a fetch expression, and a stream `proxy_pass` of a map variable.
|
||||
CAPTURE_FAKE_GATEWAY: dict[str, str] = {
|
||||
"/etc/haproxy/haproxy.cfg": (
|
||||
"global\n maxconn 4096\n"
|
||||
"defaults\n mode tcp\n"
|
||||
"frontend haproxy-22\n bind :::22 v4v6\n default_backend gitea-2222\n"
|
||||
"frontend haproxy-8443\n bind :::8443 v4v6\n"
|
||||
" use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/maps,b0)\n"
|
||||
"backend gitea-2222\n server gitea gitea.reeselink.com:2222 check inter 5s\n"
|
||||
),
|
||||
"/etc/nginx/nginx.conf": (
|
||||
"user nginx;\n"
|
||||
"stream {\n include /etc/nginx/stream.d/*.conf;\n}\n"
|
||||
"http {\n"
|
||||
" include /etc/nginx/mime.types;\n"
|
||||
" server { listen 80; listen [::]:80; server_name _; return 301 https://$host; }\n"
|
||||
" include /etc/nginx/conf.d/*.conf;\n}\n"
|
||||
),
|
||||
"/etc/nginx/conf.d/http.conf": (
|
||||
"server {\n"
|
||||
" listen 4443 ssl proxy_protocol;\n"
|
||||
" server_name nextcloud.reeseapps.com;\n"
|
||||
" ssl_certificate_key /etc/letsencrypt/live/nextcloud.reeseapps.com/privkey.pem;\n"
|
||||
" location / { proxy_pass http://nextcloud.reeselink.com:11000; }\n}\n"
|
||||
),
|
||||
"/etc/nginx/conf.d/tls.conf": (
|
||||
"server {\n listen 8443 ssl;\n server_name brain.reeseapps.com;\n"
|
||||
" location / { proxy_pass http://reeseapps-brain-of-reese.reeselink.com:443; }\n}\n"
|
||||
),
|
||||
"/etc/nginx/stream.d/stream.conf": (
|
||||
"map $ssl_preread_server_name $targetBackend {\n"
|
||||
" default 127.0.0.1:4443;\n"
|
||||
" brain.example reeseapps-brain-of-reese.reeselink.com:443;\n}\n"
|
||||
"server {\n listen 443;\n proxy_pass $targetBackend;\n}\n"
|
||||
),
|
||||
# 8.8.8.8 and 2603:6013::1 are globally routable, `::ffff:8.8.8.8` is a visitor behind a v6
|
||||
# socket, and 203.0.113.7 is already documentation space. The tx/rx queue pair on the last row
|
||||
# is spelled like an address column and is not one.
|
||||
"/proc/net/tcp": (
|
||||
" sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid"
|
||||
" timeout inode\n"
|
||||
" 0: 00000000:1EFF 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0"
|
||||
" 0 1 1 0000000000000000 100 0 0 10 0\n"
|
||||
" 1: 1B01010A:01BB 08080808:911A 01 00000000:00000000 00:00000000 00000000 101"
|
||||
" 0 2 1 00000000d22dedba 20 4 28 10 -1\n"
|
||||
" 2: 0100007F:23A0 017100CB:C0A8 06 08080808:08080808 00:00000000 00000000 0"
|
||||
" 0 3 3 0000000000000000 20 0 0 10 0\n"
|
||||
),
|
||||
"/proc/net/tcp6": (
|
||||
" sl local_address remote_address"
|
||||
" st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode\n"
|
||||
" 0: 00000000000000000000000000000000:01BB 00000000000000000000000000000000:0000 0A"
|
||||
" 00000000:00000000 00:00000000 00000000 0 0 4 1 0000000000000000 100 0 0 10 0\n"
|
||||
" 1: 1360032601010031FF005450E0B284FE:01BB 0000000000000000FFFF000008080808:C0AC 01"
|
||||
" 00000000:00000000 00:00000000 00000000 101 0 5 1 0000000000000000 63 4 26 10 -1\n"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _capture_sandbox(
|
||||
tmp_path: Path,
|
||||
gateway: dict[str, str] | None = None,
|
||||
*,
|
||||
recorded: dict[str, str] | None = None,
|
||||
fail_ssh: bool = False,
|
||||
alias: str | None = None,
|
||||
) -> tuple[Path, Path, Path, dict[str, str]]:
|
||||
"""A copy of the capture script, a fake gateway, and a PATH whose `ssh` reaches only the fake.
|
||||
|
||||
Returns (script, fixtures dir, stub call log, env). The script is copied into
|
||||
`<tmp>/scripts/` because it derives `tests/fixtures/` from its own location, which is what
|
||||
keeps the repository's fixtures out of reach of a test.
|
||||
"""
|
||||
script = tmp_path / "scripts" / CAPTURE.name
|
||||
script.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copyfile(CAPTURE, script)
|
||||
|
||||
fake = tmp_path / "gateway"
|
||||
for path, body in {**CAPTURE_FAKE_GATEWAY, **(gateway or {})}.items():
|
||||
target = fake / path.lstrip("/")
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(body, encoding="utf-8")
|
||||
|
||||
fixtures = tmp_path / "tests" / "fixtures"
|
||||
fixtures.mkdir(parents=True, exist_ok=True)
|
||||
for name, body in (recorded or {}).items():
|
||||
(fixtures / name).write_text(body, encoding="utf-8")
|
||||
|
||||
stubs = tmp_path / "stubs"
|
||||
stubs.mkdir()
|
||||
stub = stubs / "ssh"
|
||||
stub.write_text(CAPTURE_SSH_STUB, encoding="utf-8")
|
||||
stub.chmod(0o755)
|
||||
call_log = tmp_path / "ssh-calls.log"
|
||||
call_log.write_text("", encoding="utf-8")
|
||||
|
||||
env = {**os.environ,
|
||||
"PATH": f"{stubs}{os.pathsep}{os.environ['PATH']}",
|
||||
"STUB_LOG": str(call_log),
|
||||
"FAKE_REMOTE": str(fake),
|
||||
"FAIL_SSH": "1" if fail_ssh else "0"}
|
||||
if alias is not None:
|
||||
env["GATEWAY_SSH"] = alias
|
||||
return script, fixtures, call_log, env
|
||||
|
||||
|
||||
def _capture(tmp_path: Path, **kwargs: object) -> subprocess.CompletedProcess[str]:
|
||||
script, _fixtures, _log, env = _capture_sandbox(tmp_path, **kwargs) # type: ignore[arg-type]
|
||||
return subprocess.run([BASH, str(script)], capture_output=True, text=True, env=env, check=False)
|
||||
|
||||
|
||||
def _fixture_body(path: Path) -> str:
|
||||
"""A fixture without its `#` provenance block — the bytes the host answered."""
|
||||
return "".join(line for line in path.read_text().splitlines(keepends=True)
|
||||
if not line.startswith("#"))
|
||||
|
||||
|
||||
def test_the_gateway_capture_exists_and_asks_for_no_privilege() -> None:
|
||||
"""Six world-readable reads over an ssh key: nothing here needs root or a rewrite tool."""
|
||||
assert CAPTURE.is_file(), "scripts/capture-gateway-config.sh is missing (phase 28 task 04)"
|
||||
assert os.access(CAPTURE, os.X_OK)
|
||||
text = CAPTURE.read_text()
|
||||
for path in CAPTURE_TARGETS:
|
||||
assert path in text, f"{path} is no longer captured"
|
||||
code = "\n".join(line for line in text.splitlines() if not line.lstrip().startswith("#"))
|
||||
assert "sudo" not in code, (
|
||||
"the capture must read exactly what dashboards-agent can read; a pass that needed root "
|
||||
"would record more than the agent can see (the smartctl note in capture-fixtures.sh)"
|
||||
)
|
||||
for tool in ("sed ", "awk ", " tr ", "cut "):
|
||||
assert tool not in text, f"{tool.strip()} would be a text filter between the read and a fixture"
|
||||
|
||||
|
||||
def test_the_gateway_capture_dials_the_alias_and_nothing_else(tmp_path: Path) -> None:
|
||||
"""No hardcoded address: every remote command goes to `$GATEWAY_SSH`, default `gateway-ops`."""
|
||||
text = CAPTURE.read_text()
|
||||
assert 'GATEWAY_SSH="${GATEWAY_SSH:-gateway-ops}"' in text
|
||||
# Comments may name an address (`0.0.0.0` is a shape the parser has to survive); code may not.
|
||||
code = "\n".join(line for line in text.splitlines() if not line.lstrip().startswith("#"))
|
||||
stray = [a for a in re.findall(r"\b(?:\d{1,3}\.){3}\d{1,3}\b", code)
|
||||
if not a.startswith("203.0.113.")]
|
||||
assert not stray, f"the script dials or embeds an address: {stray}"
|
||||
|
||||
_script, _fixtures, log, env = _capture_sandbox(tmp_path)
|
||||
subprocess.run([BASH, str(_script)], capture_output=True, text=True, env=env, check=False)
|
||||
assert _dialled_hosts(log) == {"gateway-ops"}
|
||||
|
||||
other = _capture_sandbox(tmp_path / "alias", alias="lab-gw")
|
||||
subprocess.run([BASH, str(other[0])], capture_output=True, text=True, env=other[3], check=False)
|
||||
assert _dialled_hosts(other[2]) == {"lab-gw"}
|
||||
|
||||
|
||||
def _dialled_hosts(log: Path) -> set[str]:
|
||||
"""The host argument of every ssh the stub was called for — options are not hosts."""
|
||||
hosts = set()
|
||||
for line in log.read_text().splitlines():
|
||||
tokens = [token for token in line.split() if not token.startswith("-") and "=" not in token]
|
||||
if tokens:
|
||||
hosts.add(tokens[0])
|
||||
return hosts
|
||||
|
||||
|
||||
def test_the_gateway_capture_records_every_document_of_the_tree_it_read(tmp_path: Path) -> None:
|
||||
"""The happy path, run: one fixture per remote document, include globs and all.
|
||||
|
||||
Two files in `conf.d` is deliberate. The script reads the include globs out of the captured
|
||||
main config rather than a hardcoded pair of directories, so a gateway that adds a vhost file
|
||||
gets it captured, and `files` in the payload stays the tree the config actually merges.
|
||||
"""
|
||||
result = _capture(tmp_path)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
fixtures = tmp_path / "tests" / "fixtures"
|
||||
expected = {"gateway_haproxy.cfg", "gateway_nginx_main.conf",
|
||||
"gateway_nginx_conf_d_http.conf", "gateway_nginx_conf_d_tls.conf",
|
||||
"gateway_nginx_stream_d_stream.conf",
|
||||
"gateway_proc_net_tcp.txt", "gateway_proc_net_tcp6.txt"}
|
||||
assert {p.name for p in fixtures.iterdir()} == expected, result.stdout
|
||||
assert _fixture_body(fixtures / "gateway_haproxy.cfg") == \
|
||||
CAPTURE_FAKE_GATEWAY["/etc/haproxy/haproxy.cfg"]
|
||||
assert _fixture_body(fixtures / "gateway_nginx_conf_d_tls.conf") == \
|
||||
CAPTURE_FAKE_GATEWAY["/etc/nginx/conf.d/tls.conf"]
|
||||
# `mime.types` is included by the config and absent on the host: no fixture, no complaint —
|
||||
# the same absence-the-agent-reports rule, one level up.
|
||||
assert not (fixtures / "gateway_nginx_mime.types").exists()
|
||||
|
||||
|
||||
def test_the_gateway_capture_writes_the_domains_it_read_verbatim(tmp_path: Path) -> None:
|
||||
"""Nothing is scrubbed from a config, and the domains are the proof.
|
||||
|
||||
The `server_name` and the `proxy_pass` target beside it are the ingress map, and the
|
||||
`ssl_certificate_key` is a path rather than a key — so the file the fixture holds is the file
|
||||
the gateway runs, byte for byte. The one exception in this script is the socket tables'
|
||||
globally routable addresses, which `capture-fixtures.sh`'s identifier rule requires.
|
||||
"""
|
||||
_capture(tmp_path)
|
||||
vhosts = (tmp_path / "tests" / "fixtures" / "gateway_nginx_conf_d_http.conf").read_text()
|
||||
assert "server_name nextcloud.reeseapps.com;" in vhosts
|
||||
assert "proxy_pass http://nextcloud.reeselink.com:11000;" in vhosts
|
||||
assert "ssl_certificate_key /etc/letsencrypt/live/nextcloud.reeseapps.com/privkey.pem;" in vhosts
|
||||
assert "SCRUB" not in vhosts and "REDACTED" not in vhosts
|
||||
assert next((line for line in vhosts.splitlines()
|
||||
if line.startswith("# Verbatim /etc/nginx/conf.d/http.conf from gateway-ops")),
|
||||
None), "every fixture says which remote document it is"
|
||||
|
||||
|
||||
def test_a_dead_gateway_leaves_the_recorded_fixtures_alone(tmp_path: Path) -> None:
|
||||
"""The `slots_gufo.json` rule: a capture script must never turn six good fixtures into six
|
||||
empty files, because then the parser tests pass against nothing (or fail for the wrong
|
||||
reason) and the ground truth is gone."""
|
||||
keep = {"gateway_haproxy.cfg": "# old\nfrontend kept\n bind :::22 v4v6\n",
|
||||
"gateway_proc_net_tcp.txt": " 0: 00000000:1EFF 00000000:0000 0A x\n"}
|
||||
result = _capture(tmp_path, recorded=keep, fail_ssh=True)
|
||||
assert result.returncode == 1, result.stdout
|
||||
for name, body in keep.items():
|
||||
assert (tmp_path / "tests" / "fixtures" / name).read_text() == body, \
|
||||
f"{name} was overwritten by a read that failed"
|
||||
for name in keep:
|
||||
assert f"✗ {name}" in result.stdout, result.stdout
|
||||
assert "Connection refused" in result.stdout, "one line names the reason, per fixture"
|
||||
assert sorted(p.name for p in (tmp_path / "tests" / "fixtures").iterdir()) == sorted(keep), \
|
||||
"a failed capture leaves no temp files behind"
|
||||
|
||||
|
||||
def test_an_empty_read_is_refused_the_way_a_failed_one_is(tmp_path: Path) -> None:
|
||||
"""`cat` exiting 0 on an empty file is still not a capture."""
|
||||
keep = {"gateway_proc_net_tcp.txt": " 0: 00000000:1EFF 00000000:0000 0A keep me\n"}
|
||||
result = _capture(tmp_path, gateway={"/proc/net/tcp": ""}, recorded=keep)
|
||||
assert result.returncode == 1, result.stdout
|
||||
assert "gateway_proc_net_tcp.txt" in result.stdout
|
||||
assert "empty" in result.stdout
|
||||
assert (tmp_path / "tests" / "fixtures" / "gateway_proc_net_tcp.txt").read_text() == \
|
||||
next(iter(keep.values()))
|
||||
|
||||
|
||||
def test_an_include_dir_with_no_fixture_name_is_reported_and_fails(tmp_path: Path) -> None:
|
||||
"""A config that moved its vhosts is a loud failure, not a fixture set that quietly stopped
|
||||
being the tree the agent merges."""
|
||||
moved = {"/etc/nginx/nginx.conf": (
|
||||
"http {\n include /etc/nginx/sites-enabled/*.conf;\n}\n"
|
||||
"stream {\n include /etc/nginx/stream.d/*.conf;\n}\n")}
|
||||
result = _capture(tmp_path, gateway=moved)
|
||||
assert result.returncode == 1, result.stdout
|
||||
assert "/etc/nginx/sites-enabled/*.conf" in result.stdout
|
||||
assert not (tmp_path / "tests" / "fixtures" / "gateway_nginx_sites_enabled_web.conf").exists()
|
||||
|
||||
|
||||
def test_a_config_with_no_glob_include_says_the_vhost_fixtures_were_not_refreshed(
|
||||
tmp_path: Path) -> None:
|
||||
keep = {"gateway_nginx_conf_d_http.conf": "# keep\nserver {\n listen 4443;\n}\n"}
|
||||
result = _capture(tmp_path, gateway={"/etc/nginx/nginx.conf": "user nginx;\nevents { }\n"},
|
||||
recorded=keep)
|
||||
assert result.returncode == 1, result.stdout
|
||||
assert "no glob include" in result.stdout
|
||||
assert (tmp_path / "tests" / "fixtures" / "gateway_nginx_conf_d_http.conf").read_text() == \
|
||||
next(iter(keep.values()))
|
||||
|
||||
|
||||
def test_the_socket_capture_publishes_no_globally_routable_endpoint(tmp_path: Path) -> None:
|
||||
"""The one edit the capture makes, run rather than asserted in prose.
|
||||
|
||||
Two rules at once, which is what makes the socket fixtures both usable and publishable: a
|
||||
globally routable address (including one behind `::ffff:`) becomes documentation space, and
|
||||
everything else — every port, every private/loopback/wildcard word, every other column, and
|
||||
the column widths themselves — is byte-for-byte what the kernel answered. A tx/rx queue pair
|
||||
is spelled exactly like an address column and must survive untouched, which is why the edit
|
||||
reads two columns and not the line.
|
||||
"""
|
||||
_capture(tmp_path)
|
||||
fixtures = tmp_path / "tests" / "fixtures"
|
||||
v4 = _fixture_body(fixtures / "gateway_proc_net_tcp.txt").splitlines()
|
||||
v6 = _fixture_body(fixtures / "gateway_proc_net_tcp6.txt").splitlines()
|
||||
raw_v4 = CAPTURE_FAKE_GATEWAY["/proc/net/tcp"].splitlines()
|
||||
raw_v6 = CAPTURE_FAKE_GATEWAY["/proc/net/tcp6"].splitlines()
|
||||
assert len(v4) == len(raw_v4) and len(v6) == len(raw_v6), "a row was added or lost"
|
||||
|
||||
assert v4[1] == raw_v4[1], "a wildcard listener must survive untouched"
|
||||
assert "08080808:911A" not in "".join(v4), "8.8.8.8 is still in the fixture"
|
||||
assert "017100CB:911A" in v4[2], v4[2] # 203.0.113.1, the port kept
|
||||
assert len(v4[2]) == len(raw_v4[2]), "the replacement must be exactly as wide as the original"
|
||||
assert v4[3] == raw_v4[3], (
|
||||
"an already-documentation address and the tx/rx queue pair on that row both moved: only "
|
||||
"the two address columns may be looked at")
|
||||
assert "08080808:08080808" in v4[3], v4[3]
|
||||
|
||||
assert "1360032601010031FF005450E0B284FE" not in "".join(v6), "a global v6 is left in"
|
||||
assert "B80D0120000000000000000001000000:01BB" in v6[2], v6[2] # 2001:db8::1, port kept
|
||||
assert "FFFF0000017100CB:C0AC" in v6[2] or "FFFF0000" in v6[2], \
|
||||
"a v4-mapped row must stay v4-mapped, or the parser's branch for it stops being exercised"
|
||||
assert v6[1] == raw_v6[1], "the wildcard v6 listener moved"
|
||||
assert len(v6[2]) == len(raw_v6[2])
|
||||
|
||||
|
||||
def test_the_socket_capture_is_idempotent_so_a_re_run_does_not_churn(tmp_path: Path) -> None:
|
||||
"""Re-recording the same moment must not diff, or every capture moves the fixture twice.
|
||||
|
||||
Documentation space is not globally routable, so the second pass leaves it alone — the same
|
||||
property that makes the drive-serial scrub re-runnable.
|
||||
"""
|
||||
once = tmp_path / "once"
|
||||
_capture(once)
|
||||
again = _capture_sandbox(tmp_path / "twice",
|
||||
gateway={"/proc/net/tcp": _fixture_body(
|
||||
once / "tests" / "fixtures" / "gateway_proc_net_tcp.txt")})
|
||||
result = subprocess.run([BASH, str(again[0])], capture_output=True, text=True,
|
||||
env=again[3], check=False)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
assert _fixture_body(again[1] / "gateway_proc_net_tcp.txt") == _fixture_body(
|
||||
once / "tests" / "fixtures" / "gateway_proc_net_tcp.txt")
|
||||
|
||||
Reference in New Issue
Block a user