phase: 28_gateway_agent_proxy
**Phase 28 final pass — complete, 100 % green.** All five tasks were already `complete/`; this pass verified the criteria, re-ran the live half, and did the docs pass. - Gate `bash .agents/validate.sh` **rc=0**: `1650 passed, 15 skipped`, coverage **98.97 %** (floor 94), `ruff` clean, `pyright` **0 errors**, `node --check` clean, hygiene clean (`/tmp/validate-done.log`). - Live: `pytest -m live --live` **11 passed**; `./scripts/dev-server.sh status` nine sources `ok`; mTLS re-read of `gateway` → agent **1.2.0**, all three sections `available: true`, 6/6 frontends routed, 11 nginx servers + `$targetBackend`, **19** listeners, no core errors (`data/logs/28-finalpass-live.out`); installed agent file byte-identical to this tree (`b0e9ebf1b718…`). - Criteria: 1 ✅ pytest/coverage · 2 ✅ three sections over a captured tree, `false`/`null` + one `errors` line each elsewhere with `cpu`/`mem`/`disk_io`/`system` intact · 3 ✅ `AGENT_VERSION` 1.2.0 + SCHEMA names all three · 4 ✅ six fixtures verbatim, capture refuses a bad read, 116 socket address columns hold no globally routable endpoint · 5 ✅ gateway runs 1.2.0 and answers over mTLS (transcript `data/logs/28-task05-rollout.md`, `28-finalpass-live.out`) · 6 ✅ validate · 7 ✅ additive: HEAD's 1.1.0 and this 1.2.0 `collect()`ed over one `--root` are key-for-key equal plus the three new sections, `--no-proxy` equal exactly (`data/logs/verify-28-final.py` → 41 PASS, `28-final-verify.out`). - Docs fixed (README/AGENTS/Containerfile, all re-green): README's opening said "Four dashboards" above a five-row table and said the new sections are "stored in the snapshot" (they are in the **agent's** document — no `/api` payload, no `sample` row, nothing charts them until phase 29); `Containerfile`'s header still said "three dashboards"; AGENTS's coverage figure (98.94 → **98.97 %**), invariant 6 gained the captured-socket-table identifier rule, the agent bullet gained "phase 28 stops at the wire", and the fleet record names the stash-and-compare control plus the volatile listener count. Full record: `data/logs/NOTES-28-final.md`. - Deviations: none functional; no git index/history touched, phase files untouched (`PLAN.md` and `complete/` unchanged). `uv pip list --outdated` still lists 4 packages (fastapi 0.153→0.161 etc.) — advisory only, dependencies untouched by this phase, left for a deliberate pass. - Next pending phase: `.agents/phases/todo/29_gateway_dashboard_page/` (the page that consumes these three sections).
This commit is contained in:
@@ -0,0 +1,8 @@
|
|||||||
|
**Phase 28 final pass — complete, 100 % green.** All five tasks were already `complete/`; this pass verified the criteria, re-ran the live half, and did the docs pass.
|
||||||
|
|
||||||
|
- Gate `bash .agents/validate.sh` **rc=0**: `1650 passed, 15 skipped`, coverage **98.97 %** (floor 94), `ruff` clean, `pyright` **0 errors**, `node --check` clean, hygiene clean (`/tmp/validate-done.log`).
|
||||||
|
- Live: `pytest -m live --live` **11 passed**; `./scripts/dev-server.sh status` nine sources `ok`; mTLS re-read of `gateway` → agent **1.2.0**, all three sections `available: true`, 6/6 frontends routed, 11 nginx servers + `$targetBackend`, **19** listeners, no core errors (`data/logs/28-finalpass-live.out`); installed agent file byte-identical to this tree (`b0e9ebf1b718…`).
|
||||||
|
- Criteria: 1 ✅ pytest/coverage · 2 ✅ three sections over a captured tree, `false`/`null` + one `errors` line each elsewhere with `cpu`/`mem`/`disk_io`/`system` intact · 3 ✅ `AGENT_VERSION` 1.2.0 + SCHEMA names all three · 4 ✅ six fixtures verbatim, capture refuses a bad read, 116 socket address columns hold no globally routable endpoint · 5 ✅ gateway runs 1.2.0 and answers over mTLS (transcript `data/logs/28-task05-rollout.md`, `28-finalpass-live.out`) · 6 ✅ validate · 7 ✅ additive: HEAD's 1.1.0 and this 1.2.0 `collect()`ed over one `--root` are key-for-key equal plus the three new sections, `--no-proxy` equal exactly (`data/logs/verify-28-final.py` → 41 PASS, `28-final-verify.out`).
|
||||||
|
- Docs fixed (README/AGENTS/Containerfile, all re-green): README's opening said "Four dashboards" above a five-row table and said the new sections are "stored in the snapshot" (they are in the **agent's** document — no `/api` payload, no `sample` row, nothing charts them until phase 29); `Containerfile`'s header still said "three dashboards"; AGENTS's coverage figure (98.94 → **98.97 %**), invariant 6 gained the captured-socket-table identifier rule, the agent bullet gained "phase 28 stops at the wire", and the fleet record names the stash-and-compare control plus the volatile listener count. Full record: `data/logs/NOTES-28-final.md`.
|
||||||
|
- Deviations: none functional; no git index/history touched, phase files untouched (`PLAN.md` and `complete/` unchanged). `uv pip list --outdated` still lists 4 packages (fastapi 0.153→0.161 etc.) — advisory only, dependencies untouched by this phase, left for a deliberate pass.
|
||||||
|
- Next pending phase: `.agents/phases/todo/29_gateway_dashboard_page/` (the page that consumes these three sections).
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
@@ -0,0 +1,71 @@
|
|||||||
|
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 21%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
.......s................................................................ [ 34%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 56%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
....................................sssssssssss......................... [ 73%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
.......................sss.............................................. [ 99%]
|
||||||
|
......... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover Missing
|
||||||
|
------------------------------------------------------------------
|
||||||
|
src/dashboards/__init__.py 3 0 100%
|
||||||
|
src/dashboards/__main__.py 14 1 93% 32
|
||||||
|
src/dashboards/alerts.py 176 0 100%
|
||||||
|
src/dashboards/api.py 440 0 100%
|
||||||
|
src/dashboards/app.py 186 0 100%
|
||||||
|
src/dashboards/auth.py 265 0 100%
|
||||||
|
src/dashboards/certs.py 292 0 100%
|
||||||
|
src/dashboards/collectors.py 206 4 98% 304, 308-309, 326
|
||||||
|
src/dashboards/config.py 120 0 100%
|
||||||
|
src/dashboards/debug.py 14 0 100%
|
||||||
|
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||||
|
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||||
|
src/dashboards/rates.py 73 0 100%
|
||||||
|
src/dashboards/severity.py 49 0 100%
|
||||||
|
src/dashboards/sources/__init__.py 0 0 100%
|
||||||
|
src/dashboards/sources/disks.py 134 0 100%
|
||||||
|
src/dashboards/sources/host.py 209 0 100%
|
||||||
|
src/dashboards/sources/llm.py 178 1 99% 285
|
||||||
|
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||||
|
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||||
|
src/dashboards/store.py 313 2 99% 553-554
|
||||||
|
------------------------------------------------------------------
|
||||||
|
TOTAL 3496 36 99%
|
||||||
|
Required test coverage of 94% reached. Total coverage: 98.97%
|
||||||
|
1650 passed, 15 skipped, 1 warning in 157.81s (0:02:37)
|
||||||
|
== uv run ruff check src tests agent ==
|
||||||
|
All checks passed!
|
||||||
|
== uv run pyright ==
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
== node --check src/dashboards/web/static/*.js ==
|
||||||
|
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||||
|
coverage 7.16.2 7.17.0 wheel
|
||||||
|
fastapi 0.153.0 0.161.1 wheel
|
||||||
|
pycparser 3.11 3.12 wheel
|
||||||
|
ruff 0.17.1 0.17.2 wheel
|
||||||
+11
@@ -0,0 +1,11 @@
|
|||||||
|
**Task 28/01 complete — green.**
|
||||||
|
|
||||||
|
- `agent/host_agent.py`: new `haproxy` snapshot section — `haproxy_config` argument (+`--haproxy-config`, `--no-proxy`), module-level `parse_haproxy_config()` / `_scan_haproxy_config()`, `Collector.haproxy()`, wired into `snapshot()` before `errors`. Shape matches the task's assumptions; `:::22`→`addr "::"`, port `22`; a portless target keeps `port: null`; `use_backend %[…]` contributes no name; `listen` becomes the pair haproxy defines; design-rules + SCHEMA docstring updated (`AGENT_VERSION` left at 1.1.0 — task 04 owns the bump).
|
||||||
|
- Three-way `available` pinned: `true` / `false` (missing file, one `haproxy: … not found …` line) / `null` (mode-000, a directory, oversized, non-haproxy text) — each exactly **one** line, snapshot and `/health` intact. `--no-proxy` removes the key and the line, Collector-level and through `main()`.
|
||||||
|
- `src/dashboards/collectors.py`: `ADDON_ERROR_PREFIXES = ("smart", "haproxy")` — defect found while implementing. The task mandates the absence line, but `degraded` (that list) is `alerts.py`'s input, so shipping it unfiltered would have put a red banner + overview WARN on every host with no haproxy after the upgrade. Filtered there, still verbatim in `agent.errors` for phase 29.
|
||||||
|
- Tests: 19 new (17 `tests/test_agent.py`, 2 `tests/test_collectors.py`); three existing `errors == []` assertions now filter add-on lines via `core_errors()`.
|
||||||
|
- `tests/fixtures/gateway_haproxy.cfg`: **read-only** ssh capture, body byte-identical to `gateway:/etc/haproxy/haproxy.cfg` (mode 0644 — so the agent user can read it; no credential in it). Parsed end-to-end: 6 frontends / 6 backends, all `default_backend`s resolve.
|
||||||
|
- Docs: README agent section, AGENTS.md agent + `collectors.py` bullets, gateway's fleet row, pytest tally.
|
||||||
|
- Gate: `bash .agents/validate.sh` **rc=0** — 1568 passed / 15 skipped, coverage 98.94 % (floor 94), ruff clean, pyright 0 errors, `node --check` clean. Only the pre-existing advisory (fastapi 0.153.0→0.159.1; no dependency touched).
|
||||||
|
- Dev server restarted: 9 sources `ok`; `/api/system/snapshot` unchanged on the live fleet (agents are 1.1.0, so no `haproxy` key at all). Evidence: `data/logs/NOTES-28-haproxy.md`.
|
||||||
|
- No live change made to any host; no git index/history touched; phase files unmodified. Cosmetic note for task 05/29: after a fleet upgrade, a non-haproxy host's collapsed "More detail" banner will print the absence line — same as a smart line today; alerts are unaffected.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+66
@@ -0,0 +1,66 @@
|
|||||||
|
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 9%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
.........s.............................................................. [ 31%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 50%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 63%]
|
||||||
|
........................................................................ [ 68%]
|
||||||
|
..........................sssssssssss................................... [ 72%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
.............sss....................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover Missing
|
||||||
|
------------------------------------------------------------------
|
||||||
|
src/dashboards/__init__.py 3 0 100%
|
||||||
|
src/dashboards/__main__.py 14 1 93% 32
|
||||||
|
src/dashboards/alerts.py 176 0 100%
|
||||||
|
src/dashboards/api.py 440 0 100%
|
||||||
|
src/dashboards/app.py 186 0 100%
|
||||||
|
src/dashboards/auth.py 265 0 100%
|
||||||
|
src/dashboards/certs.py 292 0 100%
|
||||||
|
src/dashboards/collectors.py 206 5 98% 296-297, 301-302, 319
|
||||||
|
src/dashboards/config.py 120 0 100%
|
||||||
|
src/dashboards/debug.py 14 0 100%
|
||||||
|
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||||
|
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||||
|
src/dashboards/rates.py 73 0 100%
|
||||||
|
src/dashboards/severity.py 49 0 100%
|
||||||
|
src/dashboards/sources/__init__.py 0 0 100%
|
||||||
|
src/dashboards/sources/disks.py 134 0 100%
|
||||||
|
src/dashboards/sources/host.py 209 0 100%
|
||||||
|
src/dashboards/sources/llm.py 178 1 99% 285
|
||||||
|
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||||
|
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||||
|
src/dashboards/store.py 313 2 99% 553-554
|
||||||
|
------------------------------------------------------------------
|
||||||
|
TOTAL 3496 37 99%
|
||||||
|
Required test coverage of 94% reached. Total coverage: 98.94%
|
||||||
|
1568 passed, 15 skipped, 1 warning in 165.97s (0:02:45)
|
||||||
|
== uv run ruff check src tests agent ==
|
||||||
|
All checks passed!
|
||||||
|
== uv run pyright ==
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
== node --check src/dashboards/web/static/*.js ==
|
||||||
|
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||||
|
fastapi 0.153.0 0.159.1 wheel
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
**Implemented (phase 28 task 02 — agent `nginx` section)**
|
||||||
|
- `agent/host_agent.py`: additive `nginx` section — `Collector.nginx(errors)` + `nginx_include_files()` + pure `nginx_include_paths()` / `parse_nginx_config()`; brace-block scanner (`_nginx_tokens`/`_nginx_statements`/`_collect_nginx`) so quoted `;`/`{`, comments, nested `server`/`location`/`upstream`/`map` all parse; `--nginx-config` flag; shared `_config_path`/`read_config`/`_config_label` now used by both proxy sections; SCHEMA docstring + design-rules bullet updated.
|
||||||
|
- Three-way `available` with exactly one `nginx:` line per arm ("not found" pinned); includes followed **one level**, bounded 8 files / 1 MB (excess dropped with one line naming the count, section stays `available: true`); `files` = main + merged docs in the config's own spelling; `proxy_pass $var` served with `variable` + `target: null` (never merged); `--no-proxy` now removes `haproxy` **and** `nginx`.
|
||||||
|
- `collectors.ADDON_ERROR_PREFIXES` grew `nginx` in the same commit (AGENTS.md rule) + new pin that the agent-side and dashboard-side lists are equal (that pin was claimed but missing — defect fixed).
|
||||||
|
- Verified the parser read-only against the **real** gateway tree (`cat` over ssh): 11 servers, both maps, 17 `$targetBackend` entries, the `127.0.0.1:8081` literal, stream `proxy_pass $targetBackend` unmerged. No live change made (install is task 05).
|
||||||
|
|
||||||
|
**Tests / lint** (`uv run …`)
|
||||||
|
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1603 passed, 15 skipped, 98.94%** (was 1568; 34 new nginx cases, one 6-way parametrise over malformed files).
|
||||||
|
- `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → 0 errors.
|
||||||
|
- `uv run pytest -m live --live` → 11 passed; `./scripts/dev-server.sh status` → 9 sources `ok`.
|
||||||
|
- Agent-file coverage via `coverage run --branch --source=agent`: 94%, new code fully covered except the unreachable defensive `except` in `_config_label` (>90% criterion met).
|
||||||
|
|
||||||
|
**Docs**: `README.md` (nginx add-on paragraph), `AGENTS.md` (agent bullet for the `nginx` section, `ADDON_ERROR_PREFIXES` list, gateway fleet row, test tally). Transcript: `data/logs/NOTES-28-nginx.md` (gitignored).
|
||||||
|
|
||||||
|
**Deviations**: map entry keys carry `variable`/`key` with the `$` dropped on both sides of the proxy_pass↔map join (one canonical spelling instead of the task's mixed `$targetBackend`/`targetBackend`); map values that are not addresses report `target: null` rather than a host-shaped guess. `uv pip list --outdated` still reports a pre-existing `fastapi 0.153.0 → 0.159.1` (validate.sh prints it as an advisory only; deps untouched by this task).
|
||||||
|
|
||||||
|
**Next pending task**: `28_gateway_agent_proxy/03_socket_section.md`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+67
@@ -0,0 +1,67 @@
|
|||||||
|
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 26%]
|
||||||
|
............................................s........................... [ 31%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
.............................................................sssssssssss [ 71%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 80%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
................................................sss..................... [ 97%]
|
||||||
|
.................................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover Missing
|
||||||
|
------------------------------------------------------------------
|
||||||
|
src/dashboards/__init__.py 3 0 100%
|
||||||
|
src/dashboards/__main__.py 14 1 93% 32
|
||||||
|
src/dashboards/alerts.py 176 0 100%
|
||||||
|
src/dashboards/api.py 440 0 100%
|
||||||
|
src/dashboards/app.py 186 0 100%
|
||||||
|
src/dashboards/auth.py 265 0 100%
|
||||||
|
src/dashboards/certs.py 292 0 100%
|
||||||
|
src/dashboards/collectors.py 206 5 98% 302-303, 307-308, 325
|
||||||
|
src/dashboards/config.py 120 0 100%
|
||||||
|
src/dashboards/debug.py 14 0 100%
|
||||||
|
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||||
|
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||||
|
src/dashboards/rates.py 73 0 100%
|
||||||
|
src/dashboards/severity.py 49 0 100%
|
||||||
|
src/dashboards/sources/__init__.py 0 0 100%
|
||||||
|
src/dashboards/sources/disks.py 134 0 100%
|
||||||
|
src/dashboards/sources/host.py 209 0 100%
|
||||||
|
src/dashboards/sources/llm.py 178 1 99% 285
|
||||||
|
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||||
|
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||||
|
src/dashboards/store.py 313 2 99% 553-554
|
||||||
|
------------------------------------------------------------------
|
||||||
|
TOTAL 3496 37 99%
|
||||||
|
Required test coverage of 94% reached. Total coverage: 98.94%
|
||||||
|
1603 passed, 15 skipped, 1 warning in 162.84s (0:02:42)
|
||||||
|
== uv run ruff check src tests agent ==
|
||||||
|
All checks passed!
|
||||||
|
== uv run pyright ==
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
== node --check src/dashboards/web/static/*.js ==
|
||||||
|
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||||
|
fastapi 0.153.0 0.159.1 wheel
|
||||||
+15
@@ -0,0 +1,15 @@
|
|||||||
|
Task 03 (`sockets` section) is complete.
|
||||||
|
|
||||||
|
**Implemented** — `agent/host_agent.py`:
|
||||||
|
- `parse_proc_net_tcp(text, family)` keeps only `0A` LISTEN → `{family, addr, port}` and `01` ESTABLISHED → `{family, local, remote}`; every other state is skipped silently. `_proc_net_addr` unrolls `/proc`'s per-32-bit-word little-endian spelling through `socket.inet_ntop`, so wildcards stay literal `0.0.0.0` / `::` and a bad word yields `addr: null` (never `0`); a bad port is `null` too.
|
||||||
|
- `Collector.read_proc_table()` bounds each table to `PROC_NET_MAX_LINES` (4096, `/proc`'s header among them) and reports the excess as **one** line naming the path; `Collector.sockets(errors)` reads v4/v6 independently — either answering gives `available: true`, a missing `tcp6` is silent (kernel without IPv6), a denied table always costs one line, neither answering gives `available: null` plus one combined line. There is deliberately **no `false` arm** (`false` beside empty lists would read "no listeners").
|
||||||
|
- `document["sockets"]` is attached inside `Collector.snapshot()` only when `proxy_enabled`, wrapped in `_safe`, so an unexpected exception costs one `sockets:` line and nothing else — `/snapshot` and `/health` never fail. `--no-proxy` removes the key with no error line.
|
||||||
|
- `src/dashboards/collectors.py`: `ADDON_ERROR_PREFIXES = ("smart", "haproxy", "nginx", "sockets")` so absence/errors do not turn `/system` red.
|
||||||
|
|
||||||
|
**Tests** — `tests/test_agent.py`: v4+v6 parsing incl. wildcard `0.0.0.0`/`::`, every non-`0A`/`01` state skipped, malformed endpoint words, header line ignored, `--root` fake-tree reads, additivity (only `ts`/`duration_ms`/`sockets`/`errors` move), both-missing → one line naming both paths, missing `tcp6` silent, denied table costs one line, truncation reported once, `--no-proxy` (payload, `--once`, `/schema`), `_safe` fallback, `inet_ntop` raising, and a real-`/proc` shape test. `tests/test_collectors.py`: `sockets:` lines excluded from `degraded`, prefix tuple pinned.
|
||||||
|
|
||||||
|
**Verification** — `bash .agents/validate.sh`: **1630 passed, 15 skipped**; coverage **98.94%** (floor 94% met); `ruff` clean; `pyright` 0 errors. Real read-only `--once` on this box: 14 listeners / 8 established, matching `ss -ltnH` exactly, no error lines. No live-infrastructure changes; no git index/history commands.
|
||||||
|
|
||||||
|
**Docs** — README agent section + AGENTS.md `sockets` rules and the add-on-prefix sentence.
|
||||||
|
|
||||||
|
**Next pending task, not started**: `04_version_and_capture.md`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+67
@@ -0,0 +1,67 @@
|
|||||||
|
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 21%]
|
||||||
|
........................................................................ [ 26%]
|
||||||
|
.......................................................................s [ 30%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 39%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 56%]
|
||||||
|
........................................................................ [ 61%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
................sssssssssss............................................. [ 74%]
|
||||||
|
........................................................................ [ 78%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
...sss....................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover Missing
|
||||||
|
------------------------------------------------------------------
|
||||||
|
src/dashboards/__init__.py 3 0 100%
|
||||||
|
src/dashboards/__main__.py 14 1 93% 32
|
||||||
|
src/dashboards/alerts.py 176 0 100%
|
||||||
|
src/dashboards/api.py 440 0 100%
|
||||||
|
src/dashboards/app.py 186 0 100%
|
||||||
|
src/dashboards/auth.py 265 0 100%
|
||||||
|
src/dashboards/certs.py 292 0 100%
|
||||||
|
src/dashboards/collectors.py 206 5 98% 303-304, 308-309, 326
|
||||||
|
src/dashboards/config.py 120 0 100%
|
||||||
|
src/dashboards/debug.py 14 0 100%
|
||||||
|
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||||
|
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||||
|
src/dashboards/rates.py 73 0 100%
|
||||||
|
src/dashboards/severity.py 49 0 100%
|
||||||
|
src/dashboards/sources/__init__.py 0 0 100%
|
||||||
|
src/dashboards/sources/disks.py 134 0 100%
|
||||||
|
src/dashboards/sources/host.py 209 0 100%
|
||||||
|
src/dashboards/sources/llm.py 178 1 99% 285
|
||||||
|
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||||
|
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||||
|
src/dashboards/store.py 313 2 99% 553-554
|
||||||
|
------------------------------------------------------------------
|
||||||
|
TOTAL 3496 37 99%
|
||||||
|
Required test coverage of 94% reached. Total coverage: 98.94%
|
||||||
|
1630 passed, 15 skipped, 1 warning in 449.51s (0:07:29)
|
||||||
|
== uv run ruff check src tests agent ==
|
||||||
|
All checks passed!
|
||||||
|
== uv run pyright ==
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
== node --check src/dashboards/web/static/*.js ==
|
||||||
|
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||||
|
fastapi 0.153.0 0.159.1 wheel
|
||||||
+13
@@ -0,0 +1,13 @@
|
|||||||
|
**Task 04 complete — `AGENT_VERSION` 1.2.0, the SCHEMA, and the gateway captures.**
|
||||||
|
|
||||||
|
**Agent** — `agent/host_agent.py`: `AGENT_VERSION` `1.1.0` → `1.2.0`; the `GET /schema` block already names `haproxy`/`nginx`/`sockets` (each with its `available`), `--no-proxy`, `--haproxy-config`, `--nginx-config`. `tests/test_agent.py` pins the constant plus all of that; the phase-15 TLS pin keeps its schema assertions and no longer competes over the number.
|
||||||
|
|
||||||
|
**Capture** — new `scripts/capture-gateway-config.sh` (`GATEWAY_SSH`, default `gateway-ops`): one `ssh … cat` per document (`scp` would copy `/proc/net/tcp` as 0 bytes), no sudo, haproxy.cfg + `nginx.conf` **plus every document its own glob `include` names** + `/proc/net/tcp{,6}`, provenance header naming the remote path and script, and a refusal (`exit 1`, nothing written, temp removed) for a failed/empty read, an unnameable include dir or a missing glob. Ran it: 7 fixtures now in `tests/fixtures/` (haproxy, nginx main + `conf_d_http` + `stream_d_stream`, both socket tables).
|
||||||
|
|
||||||
|
**Deviation from the task's "scrubs nothing" assumption** — configs stay verbatim (domains, `proxy_pass` targets, key *paths*); the two socket tables now replace **only globally routable endpoints** with documentation space (RFC 5737/3849), idempotent and column-width-preserving, because a global IPv6 here is EUI-64 (MAC-derived) and an inbound visitor's address is not ours to publish — the smartctl serial rule (invariant 6). Documented in README + AGENTS.md and pinned in both test files.
|
||||||
|
|
||||||
|
**Tests** — `test_agent.py`: 7 tests over the captures (header/lineage, the merged include tree, SNI map + `proxy_pass $var` unmerged, every table row decodes, config↔socket join on 22/443/4443/8443/9120, no globally routable endpoint, one collector answering all three sections). `test_deploy_scripts.py`: 10 pins (alias dialed and nothing else, no `sudo`/IP in code, full tree recorded, domains verbatim, dead gateway + empty read leave fixtures alone, the two refusal messages, substitution behaviour + idempotence).
|
||||||
|
|
||||||
|
**Live probe (read-only)** — `data/logs/verify-28-task04.py` → `28-task04-probe.out`: gateway `active`, installed **1.1.0**, CA-only `/health` 200, `/snapshot` 403 `client certificate required`, mTLS snapshot has every pre-1.2.0 section with live numbers and **none** of the three new ones → **PASS: no behaviour change**.
|
||||||
|
|
||||||
|
**Gate** — `bash .agents/validate.sh`: **1650 passed, 15 skipped**, coverage **98.94%**, ruff clean, pyright 0 errors, `node --check` clean, hygiene clean (only the pre-existing `fastapi` advisory); `pytest -m live --live` with `.env.dev`: **11 passed**. Docs updated in README + AGENTS.md (capture script, the three capture rules, the fleet read). Nothing staged or committed; new script/fixtures are untracked for task 05.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+68
@@ -0,0 +1,68 @@
|
|||||||
|
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 21%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
.......s................................................................ [ 34%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 56%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
....................................sssssssssss......................... [ 73%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
.......................sss.............................................. [ 99%]
|
||||||
|
......... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover Missing
|
||||||
|
------------------------------------------------------------------
|
||||||
|
src/dashboards/__init__.py 3 0 100%
|
||||||
|
src/dashboards/__main__.py 14 1 93% 32
|
||||||
|
src/dashboards/alerts.py 176 0 100%
|
||||||
|
src/dashboards/api.py 440 0 100%
|
||||||
|
src/dashboards/app.py 186 0 100%
|
||||||
|
src/dashboards/auth.py 265 0 100%
|
||||||
|
src/dashboards/certs.py 292 0 100%
|
||||||
|
src/dashboards/collectors.py 206 5 98% 303-304, 308-309, 326
|
||||||
|
src/dashboards/config.py 120 0 100%
|
||||||
|
src/dashboards/debug.py 14 0 100%
|
||||||
|
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||||
|
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||||
|
src/dashboards/rates.py 73 0 100%
|
||||||
|
src/dashboards/severity.py 49 0 100%
|
||||||
|
src/dashboards/sources/__init__.py 0 0 100%
|
||||||
|
src/dashboards/sources/disks.py 134 0 100%
|
||||||
|
src/dashboards/sources/host.py 209 0 100%
|
||||||
|
src/dashboards/sources/llm.py 178 1 99% 285
|
||||||
|
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||||
|
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||||
|
src/dashboards/store.py 313 2 99% 553-554
|
||||||
|
------------------------------------------------------------------
|
||||||
|
TOTAL 3496 37 99%
|
||||||
|
Required test coverage of 94% reached. Total coverage: 98.94%
|
||||||
|
1650 passed, 15 skipped, 1 warning in 457.28s (0:07:37)
|
||||||
|
== uv run ruff check src tests agent ==
|
||||||
|
All checks passed!
|
||||||
|
== uv run pyright ==
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
== node --check src/dashboards/web/static/*.js ==
|
||||||
|
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||||
|
fastapi 0.153.0 0.159.1 wheel
|
||||||
+16
@@ -0,0 +1,16 @@
|
|||||||
|
**Task 05 complete — gateway host now runs agent 1.2.0, verified over mTLS.**
|
||||||
|
|
||||||
|
- Re-staged `/home/ops/agent-install/` on `gateway-ops` (gap-10 rule): `host_agent.py` 1.2.0 (`b0e9ebf1…`, md5 `9e062e4f…`) + installer (`ffbcd91f…`), old files snapshotted as `*-20261022-backup` (names the sudo grant doesn't name).
|
||||||
|
- Installed via the granted path with the unit's own args (`--bind 10.1.1.27 --port 9120`); **re-run guard printed nothing** — `systemctl cat` byte-identical before/after (`26bd812…`); bundle + token reused, not rewritten.
|
||||||
|
- `sudo systemctl restart dashboards-agent` → `active`, pid 20708, `https … client cert required, rev 2`.
|
||||||
|
- Verified: CA-only `--status` answers `{"ok":true,…,"version":"1.2.0"}` and still 403s `/snapshot`; mTLS probe (`data/logs/verify-28-task05.py` → `28-task05-verify.out`, exit 0) asserts version 1.2.0, all three new keys, `haproxy.available=true` (6 routed frontends/6 backends), `nginx.available=true` (11 servers, `$targetBackend` map, 19 entries), `sockets.available=true` (17 listens incl. **443**), and every pre-1.2.0 section intact with live numbers, no core error.
|
||||||
|
- Dashboard side untouched: `./scripts/dev-server.sh status` → **nine sources `ok`**; `/api/system/snapshot` gateway row `agent.version 1.2.0`, no `degraded`; `/api/disks/snapshot` still 2 hosts.
|
||||||
|
- Transcript: `data/logs/28-task05-rollout.md` (+ `28-task05-raw.log`) — hashes, ports and counts only, no config body, no address, no credential.
|
||||||
|
|
||||||
|
**Gates:** `bash .agents/validate.sh` → **exit 0**: `uv run pytest --cov --cov-fail-under=94` **1650 passed / 15 skipped**, coverage **98.94 %**; `ruff check src tests agent` clean; `pyright` clean; `node --check` clean; hygiene clean. Re-run after doc edits `uv run pytest -q` → exit 0; doc pins `tests/test_auth_docs.py tests/test_deploy_scripts.py` → **141 passed**; `uv run pytest -m live --live` → **11 passed** (read-only).
|
||||||
|
|
||||||
|
**Docs:** `AGENTS.md` fleet record (gateway on 1.2.0, other two still 1.1.0 and why that costs nothing; the rollout procedure, byte-identical unit, verification pointers) + gap 11's end-state line; `README.md` new "Upgrading an already-registered host is one file, not a rollout" paragraph (bundle-on-disk vs `--cert-bundle` flag, silent re-run guard, restart, both probes).
|
||||||
|
|
||||||
|
**Deviations:** no code change (as the task states); `uv pip list --outdated` shows pre-existing `fastapi 0.153.0 → 0.159.1` drift left alone — this task moved no dependencies and a bump belongs in its own change.
|
||||||
|
|
||||||
|
**Next pending:** phase 28 has no further tasks; `29_gateway_dashboard_page/01_…` is next in `todo/`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+68
@@ -0,0 +1,68 @@
|
|||||||
|
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 17%]
|
||||||
|
........................................................................ [ 21%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
.......s................................................................ [ 34%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 51%]
|
||||||
|
........................................................................ [ 56%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
....................................sssssssssss......................... [ 73%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
.......................sss.............................................. [ 99%]
|
||||||
|
......... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover Missing
|
||||||
|
------------------------------------------------------------------
|
||||||
|
src/dashboards/__init__.py 3 0 100%
|
||||||
|
src/dashboards/__main__.py 14 1 93% 32
|
||||||
|
src/dashboards/alerts.py 176 0 100%
|
||||||
|
src/dashboards/api.py 440 0 100%
|
||||||
|
src/dashboards/app.py 186 0 100%
|
||||||
|
src/dashboards/auth.py 265 0 100%
|
||||||
|
src/dashboards/certs.py 292 0 100%
|
||||||
|
src/dashboards/collectors.py 206 5 98% 303-304, 308-309, 326
|
||||||
|
src/dashboards/config.py 120 0 100%
|
||||||
|
src/dashboards/debug.py 14 0 100%
|
||||||
|
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||||
|
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||||
|
src/dashboards/rates.py 73 0 100%
|
||||||
|
src/dashboards/severity.py 49 0 100%
|
||||||
|
src/dashboards/sources/__init__.py 0 0 100%
|
||||||
|
src/dashboards/sources/disks.py 134 0 100%
|
||||||
|
src/dashboards/sources/host.py 209 0 100%
|
||||||
|
src/dashboards/sources/llm.py 178 1 99% 285
|
||||||
|
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||||
|
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||||
|
src/dashboards/store.py 313 2 99% 553-554
|
||||||
|
------------------------------------------------------------------
|
||||||
|
TOTAL 3496 37 99%
|
||||||
|
Required test coverage of 94% reached. Total coverage: 98.94%
|
||||||
|
1650 passed, 15 skipped, 1 warning in 455.76s (0:07:35)
|
||||||
|
== uv run ruff check src tests agent ==
|
||||||
|
All checks passed!
|
||||||
|
== uv run pyright ==
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
== node --check src/dashboards/web/static/*.js ==
|
||||||
|
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||||
|
fastapi 0.153.0 0.159.1 wheel
|
||||||
@@ -116,11 +116,20 @@ page still renders.
|
|||||||
looser), dials `GET /api/certs/<name>/bundle` over HTTP instead of importing `ensure_certs` — so
|
looser), dials `GET /api/certs/<name>/bundle` over HTTP instead of importing `ensure_certs` — so
|
||||||
it cannot mint even by accident — and prints one line per host with no PEM, no token and no
|
it cannot mint even by accident — and prints one line per host with no PEM, no token and no
|
||||||
password in it (`tests/test_deploy_scripts.py` greps the output for all three).
|
password in it (`tests/test_deploy_scripts.py` greps the output for all three).
|
||||||
|
**A captured socket table is the same case** (phase 28): `scripts/capture-gateway-config.sh`
|
||||||
|
keeps every config byte (a `server_name` and its `proxy_pass` are the ingress map the page draws,
|
||||||
|
and a key *path* is a filename, not a key) and rewrites **only** the two address columns of
|
||||||
|
`/proc/net/tcp{,6}`, and only when `ipaddress` calls the address globally routable — a Linux
|
||||||
|
global IPv6 is EUI-64, so it encodes this box's NIC, and an inbound visitor's address was never
|
||||||
|
ours to publish. Documentation space (RFC 5737 / RFC 3849), first-seen order, same width, same
|
||||||
|
`::ffff:` spelling, idempotent; ports, states and every other column stay verbatim.
|
||||||
|
`tests/test_agent.py` fails the day a fixture grows a routable endpoint back.
|
||||||
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
|
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
|
||||||
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
|
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
|
||||||
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
|
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
|
||||||
`scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing
|
`scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing
|
||||||
exporters) rather than trusting a README or a memory of an upstream.
|
exporters, `scripts/capture-gateway-config.sh` for the agent's `haproxy`/`nginx`/`sockets`
|
||||||
|
ground truth) rather than trusting a README or a memory of an upstream.
|
||||||
8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale.
|
8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale.
|
||||||
Check after touching dependencies: `uv pip list --outdated` must come back empty.
|
Check after touching dependencies: `uv pip list --outdated` must come back empty.
|
||||||
Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile
|
Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile
|
||||||
@@ -194,7 +203,8 @@ store.py ── SQLite (WAL): sample(source, host, metric, res, ts) + event ─
|
|||||||
Responsibilities, one line each:
|
Responsibilities, one line each:
|
||||||
|
|
||||||
- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`,
|
- `agent/host_agent.py` — stdlib-only (3.10+), reads `/proc`, `/sys/block`, `/sys/class/drm`,
|
||||||
`statvfs`; serves `GET /snapshot` + `GET /info` + `GET /schema` + `GET /health`, plus an
|
`statvfs` — plus, for the proxy add-on, the haproxy and nginx configs and `/proc/net/tcp{,6}`
|
||||||
|
on the host that has them; serves `GET /snapshot` + `GET /info` + `GET /schema` + `GET /health`, plus an
|
||||||
HTML status page at `GET /`. Refuses to bind off-loopback without a token **or** a certificate
|
HTML status page at `GET /`. Refuses to bind off-loopback without a token **or** a certificate
|
||||||
bundle (exit 2). Counters stay counters: the dashboard computes
|
bundle (exit 2). Counters stay counters: the dashboard computes
|
||||||
rates so the agent never remembers who polled it last.
|
rates so the agent never remembers who polled it last.
|
||||||
@@ -238,6 +248,105 @@ Responsibilities, one line each:
|
|||||||
*system* source red every four minutes over an add-on that page never draws. The refresher
|
*system* source red every four minutes over an add-on that page never draws. The refresher
|
||||||
is single-flight and keeps the old cache on failure, so a bad pass retries on the next
|
is single-flight and keeps the old cache on failure, so a bad pass retries on the next
|
||||||
expiry instead of hammering the disk.
|
expiry instead of hammering the disk.
|
||||||
|
**The proxy-config add-on (phase 28 task 01)** is the `haproxy` section: `Collector.haproxy()`
|
||||||
|
reads `--haproxy-config` (default `/etc/haproxy/haproxy.cfg`, resolved under `--root` so the
|
||||||
|
fake-tree tests read the fake tree, one size-bounded read per snapshot) and
|
||||||
|
`parse_haproxy_config()` turns its text into `{frontends: [{name, binds: [{addr, port}],
|
||||||
|
default_backend, backends}], backends: [{name, servers: [{name, target, port}]}]}`. Three rules
|
||||||
|
keep it honest. (a) `available` is `smartctl_available`'s three-way answer — `true` read and
|
||||||
|
parsed, `false` no file at that path, `null` a file that could not be read or holds no haproxy
|
||||||
|
section — and each non-`true` arm costs exactly **one** `haproxy:` line for the pass, the missing
|
||||||
|
one saying **not found** (pinned wording, the same cross-module contract as smartctl's, because
|
||||||
|
the dashboard reads that word to answer "is haproxy installed?"). (b) The section reports the
|
||||||
|
config's own text and infers nothing about a request: a bind keeps the address as written
|
||||||
|
(`:::22` splits at the **last** colon into `"::"` and `22`, so an IPv6 bind survives, and
|
||||||
|
`bind :8080` keeps `addr: null` because `*` is a spelling the config did not use), a `server`
|
||||||
|
with no port keeps `port: null` and never `0`, a `use_backend` whose target is a fetch
|
||||||
|
expression (`%[req.hdr(host),map_dom(...)]`) contributes **no name** rather than an expression
|
||||||
|
where a backend belongs, and everything outside `bind`/`default_backend`/`use_backend`/`server`
|
||||||
|
is ignored without a word. The frontend→backend routing is therefore the names, which the
|
||||||
|
dashboard joins. (c) A `listen` section is reported as the frontend+backend pair haproxy itself
|
||||||
|
defines it as, so a config written in that shorthand cannot read as "this gateway routes
|
||||||
|
nothing". The section is absent from the document with `--no-proxy` (silence, the `--no-smart`
|
||||||
|
precedent — "not collected" stays distinguishable from "collected and empty"), it is additive in
|
||||||
|
the payload, and nothing in it can fail `/snapshot` or `/health`.
|
||||||
|
`tests/fixtures/gateway_haproxy.cfg` is the verbatim capture the parser is pinned to.
|
||||||
|
**The same add-on's `nginx` section (phase 28 task 02)** reads `--nginx-config` (default
|
||||||
|
`/etc/nginx/nginx.conf`) **plus the files its own `include` lines name, one level deep** — the
|
||||||
|
gateway keeps its vhosts in `conf.d/*.conf` and its SNI routing in `stream.d/*.conf`, so the
|
||||||
|
main config alone is not the picture. `parse_nginx_config(main_text, included)` is the pure
|
||||||
|
half over text, and it needs a real scanner rather than haproxy's indent loop because nginx
|
||||||
|
nests by `{}`: `_nginx_tokens` keeps a `;` inside a quoted value as data (the real config writes
|
||||||
|
`add_header … "max-age=31536000; includeSubDomains" always;`), `_nginx_statements` builds the
|
||||||
|
block tree and raises only for a file nginx itself would refuse to start on, and the walk fills
|
||||||
|
`server` blocks (`listen`, `server_name`, `proxy_pass`), `upstream` blocks and `map` blocks.
|
||||||
|
Same three rules as `haproxy`: `available` is three-way with exactly one `nginx:` line per arm
|
||||||
|
(the missing one saying **not found**, the pinned wording the dashboard reads), `--no-proxy`
|
||||||
|
removes the key in silence, and nothing in the section can fail `/snapshot` or `/health` —
|
||||||
|
including a config that will not tokenize, which is `available: null` plus its one line with
|
||||||
|
every other section intact. What it refuses to do is interpret: `listen` keeps the address as
|
||||||
|
written (`[::]:8443` → `"::"`, a bare `80` → `addr: null`) and ignores the `ssl`/`proxy_protocol`
|
||||||
|
flags, `server_name _` stays the literal `_`, a map's value becomes a `target` only when it is
|
||||||
|
an address (the `$connection_upgrade` map's `upgrade`/`close` are header words, and calling one
|
||||||
|
a host is inventing a fact to fill a field), and a `proxy_pass $var` travels with `variable`
|
||||||
|
(the `$` dropped once here, so both sides of the join spell it identically) and a **null
|
||||||
|
`target`** — the map decides per connection, so a merged target would be rule 2 with a page's
|
||||||
|
clothes on. Include-following is one level and bounded (8 files / 1 MB of included text, the
|
||||||
|
smart pass-budget shape): the excess is dropped with **one** line naming the count and the
|
||||||
|
section stays `available: true`; `files` is the main config then the documents actually merged,
|
||||||
|
spelled the way the config spells them (`/etc/nginx/conf.d/http.conf`, never the path under
|
||||||
|
`--root`, which is also what lets the parser match its own `include` lines); a literal include
|
||||||
|
naming a file this host does not have costs nothing and no line; and an `include` inside an
|
||||||
|
included file is not followed at all — `root=False` makes that structural, not a convention.
|
||||||
|
**The same add-on's `sockets` section (phase 28 task 03)** is the live half: `read_proc_table`
|
||||||
|
reads `/proc/net/tcp` and `/proc/net/tcp6` (resolved under `--root`, the config rule) through at
|
||||||
|
most `PROC_NET_MAX_LINES` = **4096 lines per table** — `/proc`'s own header among them, so 4095
|
||||||
|
sockets — because a gateway's file is one line per socket and the read must not cost the poll
|
||||||
|
somebody else's connection count; the excess is reported in **one** line naming the bound and the
|
||||||
|
section stays `available: true`. `parse_proc_net_tcp(text, family)` keeps two kernel states only:
|
||||||
|
`0A` LISTEN → `{family, addr, port}` and `01` ESTABLISHED →
|
||||||
|
`{family, local: {addr, port}, remote: {addr, port}}`; every other state is a socket in transit
|
||||||
|
(`TIME_WAIT` above all) and is skipped in silence, since drawing it would put two pipes on one
|
||||||
|
connection and bury the live pairs. `_proc_net_addr` unrolls `/proc`'s per-32-bit-word
|
||||||
|
little-endian spelling through `socket.inet_ntop`, so a wildcard keeps its literal `0.0.0.0` /
|
||||||
|
`::` — the value the inbound join compares against — and a word that is not an address is `addr:
|
||||||
|
null` with the port kept, never a guess and never `0`. The two tables are read **independently**:
|
||||||
|
`available` is `true` when either answered (a kernel built without IPv6 has no `tcp6`, which is
|
||||||
|
absence, so no line — the nginx include rule), `null` with one combined `sockets:` line when
|
||||||
|
neither did, and a table that is there and denied costs one line even beside a good one. This
|
||||||
|
section has **no `false` arm on purpose** — `/proc/net/tcp` is not something a host leaves
|
||||||
|
uninstalled, and `false` beside an empty list would read as "no listeners", which is rule 2 — and
|
||||||
|
**no join with the config**: a socket does not know its domain, so phase 29's page joins these
|
||||||
|
ports to the `haproxy`/`nginx` sections and an unclaimed socket stays visible as one.
|
||||||
|
**Task 04 made the three sections a versioned contract with captured ground truth:**
|
||||||
|
`AGENT_VERSION` is **1.2.0** (`tests/test_agent.py` pins the constant and asserts the
|
||||||
|
`GET /schema` block names `haproxy`/`nginx`/`sockets` — each with its `available` flag — plus
|
||||||
|
`--no-proxy`, `--haproxy-config` and `--nginx-config`), and `scripts/capture-gateway-config.sh`
|
||||||
|
re-records the six fixtures the parsers are pinned to over `GATEWAY_SSH` (`gateway-ops`): the
|
||||||
|
haproxy config, the nginx main config **plus every document its own glob `include` names** (so
|
||||||
|
the fixture tree cannot drift from the config's tree while the parser still follows one level),
|
||||||
|
and `/proc/net/tcp{,6}`. Three rules it exists to keep. (a) **No privilege and no copy tool**: one
|
||||||
|
`ssh <alias> cat <path>` per document, because `/proc/net/tcp` reports size 0 and `scp` would copy
|
||||||
|
an empty file, and every path it reads is world-readable — the same read the agent performs as its
|
||||||
|
own user, so a capture that needed sudo would be recording something the agent cannot see.
|
||||||
|
(b) **A bad read is never a fixture**: a nonzero `cat`/`ls`, an empty body or a missing glob include
|
||||||
|
exits non-zero having written nothing, and a half-written file is removed — the pin is that the
|
||||||
|
recorded bytes survive a dead alias and an empty read both. (c) **Configs verbatim, socket tables
|
||||||
|
identifier-free**: the configs keep every `server_name`, `proxy_pass` and key path (a domain is the
|
||||||
|
map the page draws and a key path is a filename, not a key, while an agent-only bundle carries no
|
||||||
|
key at all), but a globally routable address in a socket table becomes a documentation one (RFC
|
||||||
|
5737 `203.0.113.0/24`, RFC 3849 `2001:db8::/32`) through an idempotent column substitution — ports,
|
||||||
|
states, every other column and every space are verbatim, loopback/private/link-local/wildcard
|
||||||
|
addresses are kept, and the IPv4-mapped `::ffff:` spelling survives — because a global IPv6 on
|
||||||
|
Linux is EUI-64 (derived from the NIC MAC) and an inbound visitor's address is not this repo to
|
||||||
|
publish: the same rule that scrubs a drive's serial (invariant 6). Those two tables are volatile by
|
||||||
|
nature, so `tests/test_agent.py` asserts their **shape**, the ports the recorded configs claim, and
|
||||||
|
the absence of a globally routable endpoint — never a particular row, which a re-run may lawfully
|
||||||
|
change. What phase 28 stops at is the wire: `sources/host.py` passes **none** of the three into
|
||||||
|
`/api/system/snapshot` and `store.py` records no series for a proxy or a socket, so the sections
|
||||||
|
live in the agent's own document and reach no page until phase 29's. What the dashboard does read
|
||||||
|
today is their **error lines**, through `collectors.ADDON_ERROR_PREFIXES`, so a host with no proxy
|
||||||
|
config stays out of `/system`'s `degraded` (invariant 3).
|
||||||
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
|
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
|
||||||
env var always wins. Defaults are safe except the sources, which default to unset. The auth
|
env var always wins. Defaults are safe except the sources, which default to unset. The auth
|
||||||
block (phase 11) is `DASH_ADMIN_PASSWORD` (the **seed**, and `_env()` strips, so `VAR=` means
|
block (phase 11) is `DASH_ADMIN_PASSWORD` (the **seed**, and `_env()` strips, so `VAR=` means
|
||||||
@@ -301,11 +410,17 @@ Responsibilities, one line each:
|
|||||||
same sentence on both pages and only the `timeout` differs (the disks poll has to outlast
|
same sentence on both pages and only the `timeout` differs (the disks poll has to outlast
|
||||||
the agent's smartctl pass, and `None` is translated into httpx's sentinel because a
|
the agent's smartctl pass, and `None` is translated into httpx's sentinel because a
|
||||||
literal `None` means *no timeout at all*). Agent `errors`
|
literal `None` means *no timeout at all*). Agent `errors`
|
||||||
become the payload's `degraded` **except** the `smart:` add-on lines — a host without
|
become the payload's `degraded` **except** the add-on lines named by
|
||||||
smartmontools or device read access is unconfigured, not broken (invariant 3); they stay
|
`ADDON_ERROR_PREFIXES` (`smart`, `haproxy`, `nginx`, `sockets`) — a host without smartmontools,
|
||||||
in `agent.errors` for the `/disks` page to explain — which since phase 22 is only the
|
without device read access, without a proxy config, or whose kernel socket table did not answer
|
||||||
device-read case, because a host whose agent said `smartctl` is *missing* is not listed on
|
is unconfigured, not broken (invariant 3), and `degraded` is `alerts.py`'s input, so a fleet-wide
|
||||||
that page at all (`api._disk_visible`).
|
agent upgrade would otherwise put an overview WARN on every box that never ran haproxy or nginx.
|
||||||
|
Those lines stay verbatim in `agent.errors` for the page that owns the add-on — `/disks` explains
|
||||||
|
the `smart:` ones (and since phase 22 a host whose agent said `smartctl` is *missing* is not
|
||||||
|
listed on that page at all, `api._disk_visible`), phase 29's gateway page reads the
|
||||||
|
`haproxy:`/`nginx:`/`sockets:` ones the same way.
|
||||||
|
**A new agent add-on section adds its prefix here in the same commit**, or its absence-report
|
||||||
|
becomes a red banner on a page that has nothing to say about it.
|
||||||
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
|
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
|
||||||
(`gpu.busy_pct@<index>`). `agent_tls_fields()` is the phase-15 passthrough: a TLS agent's
|
(`gpu.busy_pct@<index>`). `agent_tls_fields()` is the phase-15 passthrough: a TLS agent's
|
||||||
`tls`/`bundle_rev` ride into the payload's existing `agent` block, and a token-mode snapshot
|
`tls`/`bundle_rev` ride into the payload's existing `agent` block, and a token-mode snapshot
|
||||||
@@ -979,7 +1094,7 @@ That scheme is the history API's key — treat it as a public interface.
|
|||||||
| ----------------- | ------------------ | -------------------------------------------------------------------- |
|
| ----------------- | ------------------ | -------------------------------------------------------------------- |
|
||||||
| `driveripper` | `driveripper-ops` | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards |
|
| `driveripper` | `driveripper-ops` | Threadripper 24c/48t + Radeon AI PRO R9700, vLLM `:8000`, **upsd**, the dashboards |
|
||||||
| `deskwork` | `deskwork-ops` | Strix Halo (Radeon 8060S iGPU), gufo `:8731` |
|
| `deskwork` | `deskwork-ops` | Strix Halo (Radeon 8060S iGPU), gufo `:8731` |
|
||||||
| `gateway` | `gateway-ops` | router, no GPU |
|
| `gateway` | `gateway-ops` | router, no GPU, the fleet's ingress: haproxy (`/etc/haproxy/haproxy.cfg`, mode `0644`: six `frontend`s on `:::<port> v4v6` → six `backend`s, one `server <name> <domain>:<port> check inter 5s` each — the file the agent's `haproxy` section parses and `tests/fixtures/gateway_haproxy.cfg` captures) and nginx (`/etc/nginx/nginx.conf` plus `conf.d/http.conf` vhosts on `4443`/`8443` and `stream.d/stream.conf`'s `map $ssl_preread_server_name $targetBackend` → `proxy_pass $targetBackend` on `443` — the tree the agent's `nginx` section parses) |
|
||||||
|
|
||||||
- **Transport is direct dial: dev is the prod topology, repo-wide (phase 10 task 05).** The
|
- **Transport is direct dial: dev is the prod topology, repo-wide (phase 10 task 05).** The
|
||||||
designed shape is one `dashboards-agent` **systemd service per host**, installed by `ops`
|
designed shape is one `dashboards-agent` **systemd service per host**, installed by `ops`
|
||||||
@@ -1023,14 +1138,41 @@ That scheme is the history API's key — treat it as a public interface.
|
|||||||
`host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`,
|
`host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`,
|
||||||
`llm:vllm`.
|
`llm:vllm`.
|
||||||
**The fleet is mTLS since 2026-10-21 (phase 24 task 03), so the bearer-token transport is now
|
**The fleet is mTLS since 2026-10-21 (phase 24 task 03), so the bearer-token transport is now
|
||||||
history rather than the designed shape.** All three hosts run `agent/host_agent.py` **1.1.0**
|
history rather than the designed shape.** `gateway` runs `agent/host_agent.py` **1.2.0** (phase 28
|
||||||
from an installed bundle, every unit carries `--cert-bundle /etc/dashboards-agent.certs` (plus
|
task 05, 2026-10-22) and `deskwork`/`driveripper` still run **1.1.0**; the spread costs nothing,
|
||||||
|
because 1.2.0 is purely additive — the three proxy-config sections are new keys and a 1.1.0
|
||||||
|
snapshot simply has none — so those two move at their next re-stage rather than on a flag day.
|
||||||
|
The control that proves "purely additive" is `data/logs/verify-28-final.py`: it stashes the whole
|
||||||
|
working tree, imports both `host_agent.py` files side by side, calls both `collect()`s against
|
||||||
|
the same `--root`, and demands the 1.2.0 document equal the 1.1.0 document key-for-key with
|
||||||
|
`haproxy`/`nginx`/`sockets` added — and 1.2.0 with `--no-proxy` equal it exactly.
|
||||||
|
All three are installed from a bundle, every unit carries `--cert-bundle /etc/dashboards-agent.certs` (plus
|
||||||
`RuntimeDirectory=dashboards-agent`, and still its `--token-file` line), and `HOST_TARGETS` is
|
`RuntimeDirectory=dashboards-agent`, and still its `--token-file` line), and `HOST_TARGETS` is
|
||||||
`"driveripper=https://10.1.0.126:9120 deskwork=https://10.1.0.51:9120
|
`"driveripper=https://10.1.0.126:9120 deskwork=https://10.1.0.51:9120
|
||||||
gateway=https://10.1.1.27:9120"` — **no `http://` entry and no userinfo anywhere**: the
|
gateway=https://10.1.1.27:9120"` — **no `http://` entry and no userinfo anywhere**: the
|
||||||
certificate is the credential (the pre-flip `.env.dev` is backed up in `data/env-backup/`). `agent.tls: true` + `agent.bundle_rev: 2` in
|
certificate is the credential (the pre-flip `.env.dev` is backed up in `data/env-backup/`). `agent.tls: true` + `agent.bundle_rev: 2` in
|
||||||
every payload, `deployed_bundle_rev == bundle_rev` for all three on `/certs`, and
|
every payload, `deployed_bundle_rev == bundle_rev` for all three on `/certs`, and
|
||||||
`./scripts/dev-server.sh status` reads the nine sources `ok`. The per-host order was the safety:
|
`./scripts/dev-server.sh status` reads the nine sources `ok`. **Phase 28 moved the gateway to
|
||||||
|
1.2.0 on 2026-10-22 and only the agent file moved.** Task 04 read it read-only first
|
||||||
|
(`data/logs/verify-28-task04.py` → `28-task04-probe.out`): 1.1.0, `agent.tls: true` + `bundle_rev 2`,
|
||||||
|
every pre-1.2.0 section answering with live numbers and **no** `haproxy`/`nginx`/`sockets` key —
|
||||||
|
the control that made the install provably additive. Task 05 then ran the phase-24 procedure on
|
||||||
|
that one host (re-stage `/home/ops/agent-install/` first with `*-20261022-backup` snapshots, gap 10;
|
||||||
|
`sudo /home/ops/agent-install/deploy/install-agent.sh --bind 10.1.1.27 --port 9120` with the
|
||||||
|
arguments read back off `systemctl cat`; `systemctl restart dashboards-agent`) and verified it two
|
||||||
|
ways: the CA-only `--status` probe answers `version 1.2.0` on `/health` and still refuses
|
||||||
|
`/snapshot` with `client certificate required`, and the dashboard's own mTLS dial
|
||||||
|
(`data/logs/verify-28-task05.py` → `28-task05-verify.out`) reads `agent_version 1.2.0` with all
|
||||||
|
three sections `available: true` (6 frontends all routed / 6 backends, 11 nginx servers and the
|
||||||
|
`$targetBackend` map, 17 listeners including 443) beside every pre-1.2.0 section intact and
|
||||||
|
`host:gateway`/`disks:gateway` `ok`. That listener count is a reading, not a fact: the same probe
|
||||||
|
re-run on 2026-10-22 for this phase's completion pass (`data/logs/verify-28-final.py` →
|
||||||
|
`28-finalpass-live.out`) answered **19** listeners with the config halves unchanged (6/6, 11
|
||||||
|
servers) — `/proc/net/tcp` is what a busy gateway writes every second, which is exactly why no
|
||||||
|
test asserts a row of it. The one fact from that transcript worth carrying to the next
|
||||||
|
host: the bundle **on disk** — not the `--cert-bundle` flag — is what puts the TLS lines into the
|
||||||
|
unit render, so an upgrade re-run that omits the flag is the silent path (no guard refusal) and
|
||||||
|
`systemctl cat` came back byte-identical (`26bd812…`); only `host_agent.py` changed. The per-host order was the safety:
|
||||||
install agent 1.1.0 + bundle → `systemctl restart` → the CA-only `--status` probe answers →
|
install agent 1.1.0 + bundle → `systemctl restart` → the CA-only `--status` probe answers →
|
||||||
*then* that host's URL flips, so no source was red longer than its own host's restart.
|
*then* that host's URL flips, so no source was red longer than its own host's restart.
|
||||||
`data/logs/NOTES-24-rollout.md` is the transcript (sha256 of what was staged where, the unit
|
`data/logs/NOTES-24-rollout.md` is the transcript (sha256 of what was staged where, the unit
|
||||||
@@ -1112,8 +1254,9 @@ That scheme is the history API's key — treat it as a public interface.
|
|||||||
filter removal, nor the re-run guard — running it would have replaced driveripper's `User=root`
|
filter removal, nor the re-run guard — running it would have replaced driveripper's `User=root`
|
||||||
+ `CapabilityBoundingSet` unit with the snapshot's capability-less one, **silently**. That trap
|
+ `CapabilityBoundingSet` unit with the snapshot's capability-less one, **silently**. That trap
|
||||||
is now closed on all three hosts: `/home/ops/agent-install/` is `ops`-owned, so the repo's
|
is now closed on all three hosts: `/home/ops/agent-install/` is `ops`-owned, so the repo's
|
||||||
`deploy/install-agent.sh` (sha256 `ffbcd91f…`) and `agent/host_agent.py` (**1.1.0**) were staged
|
`deploy/install-agent.sh` (sha256 `ffbcd91f…`) and `agent/host_agent.py` (**1.1.0**, re-staged as
|
||||||
over it without root, and each snapshot was left beside the grant as
|
**1.2.0** on `gateway` in phase 28) were staged over it without root, and each snapshot was left
|
||||||
|
beside the grant as
|
||||||
`install-agent.sh.oct18-backup`, which sudo will not run. **Re-staging is therefore the first
|
`install-agent.sh.oct18-backup`, which sudo will not run. **Re-staging is therefore the first
|
||||||
step of any install, not an optional one** — the grant is only as current as the file it points
|
step of any install, not an optional one** — the grant is only as current as the file it points
|
||||||
at. The `systemctl restart *` grant on those two hosts is the other half of an unattended
|
at. The `systemctl restart *` grant on those two hosts is the other half of an unattended
|
||||||
@@ -1233,9 +1376,9 @@ uv run python scripts/dev-certs-export.py --url http://127.0.0.1:8090 \
|
|||||||
# GET /api/certs/<name>/bundle and writes
|
# GET /api/certs/<name>/bundle and writes
|
||||||
# data/certs/<name>.bundle at 0600 — mints nothing
|
# data/certs/<name>.bundle at 0600 — mints nothing
|
||||||
|
|
||||||
uv run pytest # hermetic: 1549 passed + 15 skipped (11 live,
|
uv run pytest # hermetic: 1650 passed + 15 skipped (11 live,
|
||||||
# 4 store_perf), no network
|
# 4 store_perf), no network
|
||||||
uv run pytest --cov # coverage gate (98.94%; floor 94%)
|
uv run pytest --cov # coverage gate (98.97%; floor 94%)
|
||||||
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
|
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
|
||||||
uv run pytest -m store_perf --store-perf # opt-in: the store hot-path timings over the seeded
|
uv run pytest -m store_perf --store-perf # opt-in: the store hot-path timings over the seeded
|
||||||
# store, priced against `store_perf.BUDGETS` (the cached
|
# store, priced against `store_perf.BUDGETS` (the cached
|
||||||
@@ -1270,6 +1413,11 @@ uv run ruff check src tests agent && uv run pyright
|
|||||||
SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
|
SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
|
||||||
# ... plus smartctl per physical disk (needs device read
|
# ... plus smartctl per physical disk (needs device read
|
||||||
# access: plain smartctl, else `sudo -n`; scrubs serials)
|
# access: plain smartctl, else `sudo -n`; scrubs serials)
|
||||||
|
./scripts/capture-gateway-config.sh # ... plus the gateway's proxy tree and socket tables:
|
||||||
|
# haproxy.cfg, nginx.conf + its glob includes, and
|
||||||
|
# /proc/net/tcp{,6} over $GATEWAY_SSH (gateway-ops);
|
||||||
|
# world-readable paths only, and a failed or empty read
|
||||||
|
# never replaces a good fixture
|
||||||
|
|
||||||
bash .agents/validate.sh # the phase gate: the four checks above (pytest
|
bash .agents/validate.sh # the phase gate: the four checks above (pytest
|
||||||
# --cov --cov-fail-under=94, ruff, pyright, plus a
|
# --cov --cov-fail-under=94, ruff, pyright, plus a
|
||||||
@@ -2342,7 +2490,9 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
|||||||
(phase 15 tasks 01-05 landed 2026-10-20; phase 24 task 03 rolled the fleet out 2026-10-21).
|
(phase 15 tasks 01-05 landed 2026-10-20; phase 24 task 03 rolled the fleet out 2026-10-21).
|
||||||
The rollout is done and verified: gateway, deskwork and driveripper each run `agent/host_agent.py`
|
The rollout is done and verified: gateway, deskwork and driveripper each run `agent/host_agent.py`
|
||||||
**1.1.0** with `/etc/dashboards-agent.certs` installed `0640 root:dashboards-agent`, their units
|
**1.1.0** with `/etc/dashboards-agent.certs` installed `0640 root:dashboards-agent`, their units
|
||||||
carry `--cert-bundle` + `RuntimeDirectory=dashboards-agent` (and keep `--token-file`),
|
carry `--cert-bundle` + `RuntimeDirectory=dashboards-agent` (and keep `--token-file`) — the
|
||||||
|
gateway has since moved to **1.2.0** for phase 28's proxy-config sections on the same unit and
|
||||||
|
the same bundle, see the fleet record above —
|
||||||
`HOST_TARGETS` is `https://` for all three with no userinfo, `agent.bundle_rev` reads `2` on
|
`HOST_TARGETS` is `https://` for all three with no userinfo, `agent.bundle_rev` reads `2` on
|
||||||
every host and equals what `/api/certs` issued, and the nine sources are `ok`. The transcript is
|
every host and equals what `/api/certs` issued, and the nine sources are `ok`. The transcript is
|
||||||
`data/logs/NOTES-24-rollout.md`, and task 04 re-read the whole end state **read-only** afterwards
|
`data/logs/NOTES-24-rollout.md`, and task 04 re-read the whole end state **read-only** afterwards
|
||||||
|
|||||||
+1
-1
@@ -1,4 +1,4 @@
|
|||||||
# Homelab dashboards — one image, three dashboards, one port.
|
# Homelab dashboards — one image, four dashboards and their overview, one port.
|
||||||
#
|
#
|
||||||
# podman build -t dashboards .
|
# podman build -t dashboards .
|
||||||
# podman run --rm -p 8090:8090 --env-file dashboards.env \
|
# podman run --rm -p 8090:8090 --env-file dashboards.env \
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
# Homelab Dashboards
|
# Homelab Dashboards
|
||||||
|
|
||||||
Four dashboards, one domain, one process, one SQLite file:
|
Four dashboards and the overview page that points at them — one domain, one process, one SQLite
|
||||||
|
file:
|
||||||
|
|
||||||
| Page | What it answers | Source |
|
| Page | What it answers | Source |
|
||||||
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
|
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
|
||||||
@@ -90,7 +91,8 @@ that exact line. upsd is dialled at
|
|||||||
## The host agent
|
## The host agent
|
||||||
|
|
||||||
`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
|
`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
|
||||||
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one
|
`/sys/block`, `/sys/class/drm` and `statvfs` — and, on the host that fronts the fleet, the
|
||||||
|
haproxy/nginx configs and `/proc/net/tcp{,6}` — and answers `GET /snapshot` (everything, one
|
||||||
JSON document; `?top=N` trims the process lists), `GET /info` (slow-changing identity: OS,
|
JSON document; `?top=N` trims the process lists), `GET /info` (slow-changing identity: OS,
|
||||||
kernel, CPU model, GPU list), `GET /schema` (the schema version plus this document) and
|
kernel, CPU model, GPU list), `GET /schema` (the schema version plus this document) and
|
||||||
`GET /health` (the one route that needs no credential) as JSON, with a plain HTML status
|
`GET /health` (the one route that needs no credential) as JSON, with a plain HTML status
|
||||||
@@ -147,6 +149,81 @@ one installed before phase 09 has no SMART pass to run. Opt out silently with `-
|
|||||||
The drive truth this is all developed against lives in `tests/fixtures/smartctl_driveripper_*.txt`:
|
The drive truth this is all developed against lives in `tests/fixtures/smartctl_driveripper_*.txt`:
|
||||||
16 real drives, recorded with serials scrubbed, each one parsed by a test.
|
16 real drives, recorded with serials scrubbed, each one parsed by a test.
|
||||||
|
|
||||||
|
The `haproxy` section is the same kind of best-effort add-on, for the host that fronts the fleet.
|
||||||
|
The agent reads `/etc/haproxy/haproxy.cfg` (`--haproxy-config` names another path) and reports
|
||||||
|
what the config says: every `frontend` block with its name, each `bind` address:port exactly as
|
||||||
|
written (`:::22` is the address `::` on port `22`) and the backend names it routes to
|
||||||
|
(`default_backend`, plus every named `use_backend`), and every `backend` block with each `server`'s
|
||||||
|
name, target and port — `port: null` when the line names none, never `0`. It interprets nothing:
|
||||||
|
the routing between the two lists is the **names**, so joining them is the page's job (the gateway
|
||||||
|
page is phase 29) and the agent never evaluates a rule. `available` answers three questions the way `smartctl_available` does —
|
||||||
|
`true` read and parsed, `false` no config at that path (this host runs no haproxy), `null` a file
|
||||||
|
is there but could not be read or understood — and each non-`true` answer costs exactly **one**
|
||||||
|
`errors` line for the pass. Like the SMART lines, those lines never make the *system* page amber:
|
||||||
|
a host that never installed haproxy is unconfigured, not broken, and the page that owns the add-on
|
||||||
|
reads the sentence out of `agent.errors`. A failing config read cannot fail `/snapshot` or
|
||||||
|
`/health`, and `--no-proxy` drops the section and the line both. The parser is pinned to
|
||||||
|
`tests/fixtures/gateway_haproxy.cfg`, the verbatim capture from gateway. Neither path is an env
|
||||||
|
var and `deploy/install-agent.sh` passes neither flag — the unit it writes runs on the defaults —
|
||||||
|
so a box whose configs live elsewhere means adding the flag to `$UNIT` after installing.
|
||||||
|
|
||||||
|
The `nginx` section is that add-on's other half, and it reads more than one file: the main config
|
||||||
|
(`/etc/nginx/nginx.conf`, `--nginx-config` names another) **plus the files its own `include` lines
|
||||||
|
name**, one level deep, because that is where nginx keeps its vhosts
|
||||||
|
(`include /etc/nginx/conf.d/*.conf`) and its SNI routing (`/etc/nginx/stream.d/*.conf`). It
|
||||||
|
reports every `server` block with its `listen` address:port pairs (`[::]:8443` is the address `::`
|
||||||
|
on port `8443`; a bare `listen 80` has no address, and the `ssl`/`proxy_protocol` flags are not
|
||||||
|
this section's business), its `server_name` list (the catch-all `_` included verbatim — "any
|
||||||
|
domain" is what the page says about it), and every `proxy_pass`; every `upstream` block with its
|
||||||
|
`server` targets; and every `map` block with its keys. A `proxy_pass $var` is served with the
|
||||||
|
variable's name and `target: null`: nginx resolves that per connection from the `map` that defines
|
||||||
|
it, so the agent reports both facts **unmerged** and leaves the joining to a page — a merged target
|
||||||
|
would be a number the wire never published. `files` names the main config and the included
|
||||||
|
documents actually merged, and the include pass is bounded (8 files / 1 MB): what it could not
|
||||||
|
take is said in one `errors` line naming the count, and the section still reads `true`. The same
|
||||||
|
three-way `available`, the same one line per arm, the same rule that a missing, unreadable or
|
||||||
|
unparseable config degrades only this section — and `--no-proxy` drops the whole family with it,
|
||||||
|
in silence.
|
||||||
|
|
||||||
|
The `sockets` section is that add-on's live half: what is actually open and carrying traffic, as
|
||||||
|
opposed to what a config claims. The agent reads `/proc/net/tcp` and `/proc/net/tcp6` and keeps
|
||||||
|
only the two states a gateway page can draw pipes from — kernel state `0A` LISTEN into `listens`
|
||||||
|
(`{family, addr, port}`) and `01` ESTABLISHED into `established` (`{family, local, remote}`, each
|
||||||
|
end an `{addr, port}`). Every other state is a socket in transit: a `TIME_WAIT` row is a
|
||||||
|
connection that stopped being true before the page painted, and a proxy accumulates thousands of
|
||||||
|
them, so they are skipped without a word. Each table is read through **at most 4096 lines**
|
||||||
|
(`/proc`'s own header among them, so 4095 sockets per table); a longer one reports the rows it read
|
||||||
|
and costs one `errors` line naming the bound, with `available` still `true`. `addr` is the address
|
||||||
|
itself, unrolled from `/proc`'s per-32-bit-word little-endian spelling — `0.0.0.0` and `::` keep
|
||||||
|
the literal wildcard form, because "open on every address" is the fact the inbound join keys on —
|
||||||
|
and a word that is not an address is `null`, never a guess. The tables are read independently: a
|
||||||
|
kernel without IPv6 has no `tcp6` file, which costs nothing and no line, while a table that is
|
||||||
|
there and denied costs one. `available` is `true` when either table answered and `null` when
|
||||||
|
neither did, with one line saying why; there is no `false`, because a table that did not answer is
|
||||||
|
"no answer" and not "this host has no sockets". Nothing here is joined to a config: a socket does
|
||||||
|
not know its domain or its backend, so phase 29's gateway page will be what joins these ports to
|
||||||
|
the `haproxy` and `nginx` sections — where a socket no frontend claims stays visible as one. Until
|
||||||
|
that page exists the three sections live only in the agent's own `/snapshot` document: no `/api`
|
||||||
|
payload grows them, no `sample` row is stored for them, and nothing charts them.
|
||||||
|
|
||||||
|
All three parsers are pinned to captures rather than to descriptions, and
|
||||||
|
`./scripts/capture-gateway-config.sh` re-records them over the `GATEWAY_SSH` alias (default
|
||||||
|
`gateway-ops`): `/etc/haproxy/haproxy.cfg`, `/etc/nginx/nginx.conf` plus every document its own
|
||||||
|
glob `include` lines name (`gateway_nginx_conf_d_*.conf`, `gateway_nginx_stream_d_*.conf`), and
|
||||||
|
`/proc/net/tcp{,6}`. No sudo — those paths are world-readable, which is exactly what the agent
|
||||||
|
reads as itself — and a read that fails, or answers empty, leaves the existing fixture alone and
|
||||||
|
exits 1, so a laptop with no route to the gateway cannot turn six captures into six empty files.
|
||||||
|
The configs are not rewritten (their domains are the ingress map, and key material appears only
|
||||||
|
as the paths nginx is told to read); the socket tables lose only their identifiers, in line with
|
||||||
|
the rule that scrubs a drive's serial: a globally routable address becomes a documentation one
|
||||||
|
(RFC 5737 / RFC 3849), because a global IPv6 on Linux is EUI-64 and an inbound visitor's address
|
||||||
|
is not ours to publish. Every port, state, column and width in those tables is verbatim, and
|
||||||
|
those files are volatile by nature — a proxy in service has a different connection set every
|
||||||
|
second — so the tests assert shape and the ports the configs claim, never a particular row.
|
||||||
|
`AGENT_VERSION` is **1.2.0**, the version that ships all three sections; a host still running
|
||||||
|
1.1.0 answers `GET /schema` with a block that names none of them, and its snapshot simply has no
|
||||||
|
such keys.
|
||||||
|
|
||||||
`deploy/install-agent.sh --fetch-bundle <url> --cert-name <name> --session <token>` is the same
|
`deploy/install-agent.sh --fetch-bundle <url> --cert-name <name> --session <token>` is the same
|
||||||
TLS install with the download done inside the registration run: it asks
|
TLS install with the download done inside the registration run: it asks
|
||||||
`GET <url>/api/certs/<name>/bundle`, writes the answer to a temp file, and hands it to the
|
`GET <url>/api/certs/<name>/bundle`, writes the answer to a temp file, and hands it to the
|
||||||
@@ -338,6 +415,20 @@ for a bundle install, so step 3 is a copy rather than a guess. A host left on `h
|
|||||||
working over its bearer token — the rollout is per host, and one revoked or expired certificate
|
working over its bearer token — the rollout is per host, and one revoked or expired certificate
|
||||||
never blinks the others.
|
never blinks the others.
|
||||||
|
|
||||||
|
**Upgrading an already-registered host to a newer agent is one file, not a rollout.** Re-stage
|
||||||
|
`agent/host_agent.py` into `/home/ops/agent-install/agent/` (the snapshot-first rule applies: a path
|
||||||
|
grant is only as current as the file it names, so copy what you are replacing to
|
||||||
|
`host_agent.py-<date>-backup` first — a name the sudo grant does not name, so sudo will not run it),
|
||||||
|
run the granted path with **the unit's own arguments and nothing else**
|
||||||
|
(`sudo /home/ops/agent-install/deploy/install-agent.sh --bind <addr> --port 9120`), and restart.
|
||||||
|
No `--cert-bundle`, no `--fetch-bundle`, no `--force`: the bundle already on disk is what the unit
|
||||||
|
render reads — the flag only replaces the file — so the render equals the installed unit, the re-run
|
||||||
|
guard prints nothing, and `/etc/dashboards-agent.certs` and the token file keep their bytes. Then
|
||||||
|
the same two checks the registration uses: the CA-only `--status` probe (with `--bind`) answers
|
||||||
|
`{"ok": true, …, "version": "1.2.0"}` on `/health`, and the dashboard's own `https://` dial answers
|
||||||
|
whatever the new version added — `agent_version` plus the `haproxy` / `nginx` / `sockets` sections
|
||||||
|
for a 1.2.0 agent — beside the sections it always had.
|
||||||
|
|
||||||
Two things the first full rollout learned, both cheap to hit and expensive to guess at:
|
Two things the first full rollout learned, both cheap to hit and expensive to guess at:
|
||||||
|
|
||||||
* **A CA row older than the identifier fix must be rotated before any bundle is installed.** A
|
* **A CA row older than the identifier fix must be rotated before any bundle is installed.** A
|
||||||
@@ -953,6 +1044,11 @@ uv run ruff check src tests agent && uv run pyright # lint + types
|
|||||||
SMART_HOSTS="host-a-ops host-b-ops" ./scripts/capture-fixtures.sh
|
SMART_HOSTS="host-a-ops host-b-ops" ./scripts/capture-fixtures.sh
|
||||||
# ... and smartctl per physical disk: plain smartctl,
|
# ... and smartctl per physical disk: plain smartctl,
|
||||||
# `sudo -n` if that is denied; scrubs serials/WWNs
|
# `sudo -n` if that is denied; scrubs serials/WWNs
|
||||||
|
./scripts/capture-gateway-config.sh # ... and the gateway's proxy tree + socket tables:
|
||||||
|
# haproxy.cfg, nginx.conf + its glob includes,
|
||||||
|
# /proc/net/tcp{,6} over $GATEWAY_SSH (gateway-ops).
|
||||||
|
# World-readable paths only; a failed read never
|
||||||
|
# replaces a good fixture
|
||||||
bash .agents/validate.sh # all of the above + `node --check` on every page
|
bash .agents/validate.sh # all of the above + `node --check` on every page
|
||||||
# script + repo hygiene, with the 94% coverage floor
|
# script + repo hygiene, with the 94% coverage floor
|
||||||
```
|
```
|
||||||
|
|||||||
+1070
-6
File diff suppressed because it is too large
Load Diff
Executable
+352
@@ -0,0 +1,352 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Re-record the gateway's proxy configs and kernel socket tables into tests/fixtures/.
|
||||||
|
#
|
||||||
|
# ./scripts/capture-gateway-config.sh # dials the `gateway-ops` ssh alias
|
||||||
|
# GATEWAY_SSH=lab-gw ./scripts/capture-gateway-config.sh
|
||||||
|
#
|
||||||
|
# What lands where — one fixture per remote document, in `tests/fixtures/`:
|
||||||
|
#
|
||||||
|
# gateway_haproxy.cfg cat /etc/haproxy/haproxy.cfg
|
||||||
|
# gateway_nginx_main.conf cat /etc/nginx/nginx.conf
|
||||||
|
# gateway_nginx_conf_d_<n>.conf cat /etc/nginx/conf.d/*.conf
|
||||||
|
# gateway_nginx_stream_d_<n>.conf cat /etc/nginx/stream.d/*.conf
|
||||||
|
# gateway_proc_net_tcp.txt cat /proc/net/tcp
|
||||||
|
# gateway_proc_net_tcp6.txt cat /proc/net/tcp6
|
||||||
|
#
|
||||||
|
# These documents are the ground truth behind `agent/host_agent.py`'s `haproxy`, `nginx` and
|
||||||
|
# `sockets` sections (phase 28) the way `tests/fixtures/nut_<ups>.vars` pins the UPS wire and
|
||||||
|
# `smartctl_driveripper_<dev>.txt` pins the drives. The include globs are read **out of the
|
||||||
|
# captured main config**, so a gateway that moves its vhosts to a third directory grows a third
|
||||||
|
# fixture on the next run instead of keeping a fixture set that quietly stopped being the tree
|
||||||
|
# the agent merges.
|
||||||
|
#
|
||||||
|
# Four rules, three of them `scripts/capture-fixtures.sh`'s:
|
||||||
|
#
|
||||||
|
# * **Verbatim, never hand-written** (AGENTS.md invariant 7). A fixture you typed is a test of
|
||||||
|
# your memory. Each file carries a `#` provenance block naming the remote path it came from.
|
||||||
|
# The header is comment-shaped because `#` is a comment in haproxy's config and in nginx's,
|
||||||
|
# and a `/proc` line whose 4th field is not a kernel state is skipped by the socket parser —
|
||||||
|
# provenance that costs no parser anything, which `tests/test_agent.py` pins by parsing the
|
||||||
|
# fixtures exactly as they sit on disk.
|
||||||
|
# * **The configs are not rewritten at all.** A domain is not an identifier to hide here: a
|
||||||
|
# `server_name` and the `proxy_pass` target beside it *are* the ingress map the gateway page
|
||||||
|
# exists to draw, and a scrubbed fixture is a fixture that tests a config this gateway does
|
||||||
|
# not run. Key material appears in these files only as the paths the proxy is told to read
|
||||||
|
# (`ssl_certificate_key /etc/letsencrypt/…`), which is a filename and not a key. No text
|
||||||
|
# filter of any kind sits between the read and a config fixture —
|
||||||
|
# `tests/test_deploy_scripts.py` runs this script against a fake remote holding a domain and
|
||||||
|
# demands the bytes back.
|
||||||
|
# * **A socket table keeps its shape and loses its identifiers** (AGENTS.md invariant 6, and
|
||||||
|
# the rule `capture-fixtures.sh` already applies to a drive: "Identifiers never go in the
|
||||||
|
# repo" — Serial, LU WWN, and the NVMe EUI-64 *which encodes the serial*). A Linux global
|
||||||
|
# IPv6 is EUI-64: `2603:…:5054:ff:fe84:b2e0` is this box's NIC address, and the other thing
|
||||||
|
# `/proc/net/tcp` carries is the public address of whoever was connected at the moment of
|
||||||
|
# the capture, which is not ours to publish. So any address `ipaddress` calls globally
|
||||||
|
# routable becomes a documentation address (RFC 5737 / RFC 3849) in first-seen order,
|
||||||
|
# re-encoded through the kernel's own per-32-bit-word little-endian spelling. Loopback,
|
||||||
|
# private, link-local, wildcard and already-documentation addresses are left exactly as they
|
||||||
|
# were, so the run is idempotent and the fixture still meets every shape the parser has to
|
||||||
|
# survive — `0.0.0.0`, `::`, `::ffff:a.b.c.d`, 32-hex-digit v6 words. Every port, state,
|
||||||
|
# queue, uid, inode and column width is byte-for-byte what the kernel answered, and only the
|
||||||
|
# two address *columns* are even looked at (a tx/rx queue pair is spelled like an address
|
||||||
|
# and is not one). The tables are volatile anyway — a proxy in service has a different
|
||||||
|
# connection set every second — so the tests assert shape, the two states kept, and the
|
||||||
|
# ports the captured configs claim to bind. Never a particular row.
|
||||||
|
# * **A bad read never replaces a good fixture.** Empty output, an ssh failure or a permission
|
||||||
|
# denial prints one line naming the file and the reason, leaves whatever is on disk alone,
|
||||||
|
# and exits 1 — the `slots_gufo.json` rule, so a laptop with no route to the gateway cannot
|
||||||
|
# turn six verbatim captures into six empty files.
|
||||||
|
#
|
||||||
|
# No sudo, deliberately: haproxy's config is 0644, nginx's tree is 0644 and `/proc/net/tcp{,6}`
|
||||||
|
# is world-readable, which is exactly what `dashboards-agent` reads as itself. A capture that
|
||||||
|
# needed root would record more than the agent can see — the mistake the smartctl pass in
|
||||||
|
# `capture-fixtures.sh` documents in the other direction.
|
||||||
|
#
|
||||||
|
# A file the host stops having is left in place rather than deleted: when the config moves,
|
||||||
|
# remove the orphan fixture in the same commit that updates the parser tests naming it.
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
FIX="$ROOT/tests/fixtures"
|
||||||
|
GATEWAY_SSH="${GATEWAY_SSH:-gateway-ops}"
|
||||||
|
mkdir -p "$FIX"
|
||||||
|
|
||||||
|
# `-n` matters: `capture_included` walks the remote listing on stdin, and an ssh that reads
|
||||||
|
# stdin would eat the rest of that list and quietly capture one vhost file instead of all of them.
|
||||||
|
SSH_OPTS=(-n -o BatchMode=yes -o ConnectTimeout=8)
|
||||||
|
failed=0
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
cat <<'TXT'
|
||||||
|
Usage: ./scripts/capture-gateway-config.sh [-h]
|
||||||
|
|
||||||
|
Re-records the gateway's proxy configs and kernel socket tables into tests/fixtures/:
|
||||||
|
/etc/haproxy/haproxy.cfg, /etc/nginx/nginx.conf plus the documents its own glob includes
|
||||||
|
name, and /proc/net/tcp{,6}. Every read goes over the `$GATEWAY_SSH` ssh alias (default
|
||||||
|
`gateway-ops`) of a world-readable path, and a read that fails leaves the existing fixture
|
||||||
|
alone and exits 1. The configs land unedited; in the socket tables a globally routable
|
||||||
|
address becomes a documentation one and nothing else moves. See the block comment.
|
||||||
|
TXT
|
||||||
|
}
|
||||||
|
|
||||||
|
case "${1:-}" in
|
||||||
|
-h|--help) usage; exit 0 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# The socket tables' one edit, and the only place in this script that touches a captured byte.
|
||||||
|
# Kept as a here-doc in the house style (capture-fixtures.sh embeds two); it is pinned by
|
||||||
|
# running this script against a fake remote in tests/test_deploy_scripts.py.
|
||||||
|
SOCKET_IDENTIFIERS_OUT=$(cat <<'PY'
|
||||||
|
"""Move the globally-routable addresses in a /proc/net/tcp{,6} table to documentation space.
|
||||||
|
|
||||||
|
Every column, every port and every byte of spacing is the kernel's. What changes is an address
|
||||||
|
`ipaddress` calls global, which becomes a documentation address in first-seen order (RFC 5737
|
||||||
|
for v4, RFC 3849 for v6) re-encoded through the kernel's own per-32-bit-word little-endian
|
||||||
|
spelling — so the replacement is exactly as wide as the original and an IPv4-mapped v6 row stays
|
||||||
|
an IPv4-mapped v6 row. AGENTS.md invariant 6 is the reason: this repo's capture rule is that
|
||||||
|
identifiers never go in tracked files, and a global IPv6 on Linux is EUI-64 (it encodes the
|
||||||
|
NIC's MAC) while a visitor's public address belongs to the visitor. Loopback, private,
|
||||||
|
link-local, multicast, wildcard and already-documentation addresses are returned untouched,
|
||||||
|
which is what makes re-running this over its own output change nothing.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import ipaddress
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
DOC_V4 = int(ipaddress.IPv4Address("203.0.113.0"))
|
||||||
|
DOC_V6 = int(ipaddress.IPv6Address("2001:db8::"))
|
||||||
|
|
||||||
|
|
||||||
|
def unroll(word: str) -> bytes:
|
||||||
|
"""`/proc`'s per-word little-endian spelling -> the address's own bytes."""
|
||||||
|
raw = bytes.fromhex(word)
|
||||||
|
return b"".join(raw[i:i + 4][::-1] for i in range(0, len(raw), 4))
|
||||||
|
|
||||||
|
|
||||||
|
def roll(addr: ipaddress.IPv4Address | ipaddress.IPv6Address) -> str:
|
||||||
|
"""The inverse, in the kernel's uppercase."""
|
||||||
|
packed = addr.packed
|
||||||
|
return "".join(packed[i:i + 4][::-1].hex().upper() for i in range(0, len(packed), 4))
|
||||||
|
|
||||||
|
|
||||||
|
class Replacer:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.counters: dict[int, int] = {4: 0, 6: 0}
|
||||||
|
self.mapping: dict[str, str] = {}
|
||||||
|
self.replacements = 0
|
||||||
|
|
||||||
|
def documentation(self, version: int) -> str:
|
||||||
|
self.counters[version] += 1
|
||||||
|
if version == 4:
|
||||||
|
return str(ipaddress.IPv4Address(DOC_V4 + self.counters[version]))
|
||||||
|
return str(ipaddress.IPv6Address(DOC_V6 + self.counters[version]))
|
||||||
|
|
||||||
|
def column(self, token: str) -> str:
|
||||||
|
"""One `<word>:<port>` column: rewrite the word, keep the port and its spelling."""
|
||||||
|
word, _, port = token.rpartition(":")
|
||||||
|
if len(word) not in (8, 32) or any(c not in "0123456789abcdefABCDEF" for c in word):
|
||||||
|
return token
|
||||||
|
try:
|
||||||
|
addr = ipaddress.ip_address(unroll(word))
|
||||||
|
except ValueError:
|
||||||
|
return token
|
||||||
|
# Only IPv6Address has `ipv4_mapped`; a v4 word is its own subject.
|
||||||
|
mapped = getattr(addr, "ipv4_mapped", None)
|
||||||
|
subject = mapped if mapped is not None else addr
|
||||||
|
if not subject.is_global:
|
||||||
|
return token
|
||||||
|
key = str(subject)
|
||||||
|
if key not in self.mapping:
|
||||||
|
self.mapping[key] = self.documentation(subject.version)
|
||||||
|
replacement = self.mapping[key]
|
||||||
|
self.replacements += 1
|
||||||
|
# A v4-mapped row keeps its mapped spelling, so the row that exercises that branch of
|
||||||
|
# the agent's decoder still exercises it after the capture.
|
||||||
|
if mapped is not None:
|
||||||
|
new: ipaddress.IPv4Address | ipaddress.IPv6Address = ipaddress.IPv6Address(
|
||||||
|
f"::ffff:{replacement}")
|
||||||
|
else:
|
||||||
|
new = ipaddress.ip_address(replacement)
|
||||||
|
return f"{roll(new)}:{port}"
|
||||||
|
|
||||||
|
def line(self, text: str) -> str:
|
||||||
|
"""Rewrite columns 1 and 2 only — the two address columns of a socket table."""
|
||||||
|
spans = [(m.start(), m.end(), m.group(0)) for m in re.finditer(r"\S+", text)]
|
||||||
|
if len(spans) < 4:
|
||||||
|
return text
|
||||||
|
out = []
|
||||||
|
last = 0
|
||||||
|
for index, (start, end, token) in enumerate(spans):
|
||||||
|
out.append(text[last:start])
|
||||||
|
out.append(self.column(token) if index in (1, 2) else token)
|
||||||
|
last = end
|
||||||
|
out.append(text[last:])
|
||||||
|
return "".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str]) -> int:
|
||||||
|
replacer = Replacer()
|
||||||
|
with open(argv[1], encoding="utf-8", errors="replace") as handle:
|
||||||
|
for line in handle:
|
||||||
|
sys.stdout.write(replacer.line(line))
|
||||||
|
if replacer.replacements:
|
||||||
|
print(f" ({replacer.replacements} globally-routable address column(s) moved to "
|
||||||
|
f"documentation space; {len(replacer.mapping)} distinct endpoint(s))",
|
||||||
|
file=sys.stderr)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main(sys.argv))
|
||||||
|
PY
|
||||||
|
)
|
||||||
|
|
||||||
|
# header <kind> <remote path> — the provenance block every fixture carries.
|
||||||
|
header() {
|
||||||
|
local kind="$1" path="$2"
|
||||||
|
echo "# Verbatim $path from $GATEWAY_SSH, read read-only over the ssh key by"
|
||||||
|
echo "# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the"
|
||||||
|
echo "# deployed upstream, never guess its shape)."
|
||||||
|
case "$kind" in
|
||||||
|
config)
|
||||||
|
echo "# Nothing here is rewritten: it is this fleet's ingress map, and the domains in it are"
|
||||||
|
echo "# the data — an inbound name and the target it routes to are what these config sections"
|
||||||
|
echo "# exist to report. Key material appears only as the file paths the proxy reads."
|
||||||
|
;;
|
||||||
|
sockets)
|
||||||
|
echo "# One line per TCP socket, in the kernel's own spelling. The only edit is the repo's"
|
||||||
|
echo "# identifier rule (invariant 6, the drive-serial scrub in capture-fixtures.sh): a"
|
||||||
|
echo "# globally routable address became a documentation one (RFC 5737 / RFC 3849), because a"
|
||||||
|
echo "# global IPv6 here is EUI-64 and a visitor's address is not ours to publish. Every port,"
|
||||||
|
echo "# state, queue, uid, inode and column width is verbatim, as are the loopback, private"
|
||||||
|
echo "# and wildcard addresses the gateway's own frontends use."
|
||||||
|
echo "# Volatile by nature: a proxy in service has a different connection set every second, so"
|
||||||
|
echo "# a diff here on re-capture is expected. The tests assert shape, the two states the agent"
|
||||||
|
echo "# keeps, and the ports the captured configs claim to bind — never a particular row."
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# capture <fixture> <kind> <remote path> — one ssh call, one document, written only if the read
|
||||||
|
# answered something. The remote command is a single `cat`: `scp` cannot read /proc at all (a
|
||||||
|
# procfs file reports size 0, so scp copies an empty file), and one mechanism for six fixtures
|
||||||
|
# is one mechanism to pin.
|
||||||
|
capture() {
|
||||||
|
local fixture="$1" kind="$2" path="$3"
|
||||||
|
local tmp="$FIX/.$fixture.tmp" err="$FIX/.$fixture.err"
|
||||||
|
local rc=0
|
||||||
|
ssh "${SSH_OPTS[@]}" "$GATEWAY_SSH" "cat $path" >"$tmp" 2>"$err" || rc=$?
|
||||||
|
if [ "$rc" -ne 0 ] || [ ! -s "$tmp" ]; then
|
||||||
|
local why
|
||||||
|
why="$(head -n 1 "$err" 2>/dev/null)"
|
||||||
|
[ -n "$why" ] || { [ "$rc" -eq 0 ] && why="answered an empty file" || why="ssh exited $rc"; }
|
||||||
|
rm -f "$tmp" "$err"
|
||||||
|
echo " ✗ $fixture — $GATEWAY_SSH:$path unreadable ($why); the fixture on disk is left alone"
|
||||||
|
failed=1
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
rm -f "$err"
|
||||||
|
{
|
||||||
|
header "$kind" "$path"
|
||||||
|
case "$kind" in
|
||||||
|
sockets) python3 -c "$SOCKET_IDENTIFIERS_OUT" "$tmp" ;;
|
||||||
|
*) cat "$tmp" ;;
|
||||||
|
esac
|
||||||
|
} >"$FIX/$fixture"
|
||||||
|
rc=0
|
||||||
|
rm -f "$tmp"
|
||||||
|
echo " ✓ $fixture ($(wc -l <"$FIX/$fixture") lines from $GATEWAY_SSH:$path)"
|
||||||
|
}
|
||||||
|
|
||||||
|
# capture_included <fixture prefix> <remote dir> — every `*.conf` the main config's include line
|
||||||
|
# names, one fixture each. The prefix names the directory the way the parser spells it
|
||||||
|
# (`gateway_nginx_conf_d_` for /etc/nginx/conf.d), the rest is the file's own name:
|
||||||
|
# /etc/nginx/conf.d/http.conf → gateway_nginx_conf_d_http.conf.
|
||||||
|
capture_included() {
|
||||||
|
local prefix="$1" dir="$2"
|
||||||
|
local listing="$FIX/.$prefix.ls" err="$FIX/.$prefix.ls.err"
|
||||||
|
local rc=0
|
||||||
|
ssh "${SSH_OPTS[@]}" "$GATEWAY_SSH" "ls $dir/*.conf" >"$listing" 2>"$err" || rc=$?
|
||||||
|
if [ "$rc" -ne 0 ]; then
|
||||||
|
local why
|
||||||
|
why="$(head -n 1 "$err" 2>/dev/null)"
|
||||||
|
[ -n "$why" ] || { [ "$rc" -eq 0 ] && why="answered an empty list" || why="ssh exited $rc"; }
|
||||||
|
rm -f "$listing" "$err"
|
||||||
|
echo " ✗ ${prefix}* — $GATEWAY_SSH:$dir could not be listed ($why); the fixtures on disk are left alone"
|
||||||
|
failed=1
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
rm -f "$err"
|
||||||
|
local path base
|
||||||
|
while IFS= read -r path; do
|
||||||
|
[ -n "$path" ] || continue
|
||||||
|
base="${path##*/}"
|
||||||
|
# Parameter expansion and nothing else builds the name, so there is no text filter of any
|
||||||
|
# kind between what the host said and what the fixture holds.
|
||||||
|
capture "${prefix}${base%.conf}.conf" config "$path"
|
||||||
|
done <"$listing"
|
||||||
|
rm -f "$listing"
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- haproxy -----------------------------------------------------------------
|
||||||
|
# The file `parse_haproxy_config` is pinned to: frontends on `:::<port> v4v6`, one
|
||||||
|
# `server <name> <domain>:<port> check inter 5s` per backend, and a `use_backend` whose target
|
||||||
|
# is a fetch expression — which is why a name is reported only where the config names one.
|
||||||
|
capture gateway_haproxy.cfg config /etc/haproxy/haproxy.cfg
|
||||||
|
|
||||||
|
# --- nginx -------------------------------------------------------------------
|
||||||
|
# The main config, then the documents its own include lines name one level deep: the same set
|
||||||
|
# `Collector.nginx` merges. A literal include naming a file this host does not have (this
|
||||||
|
# gateway's `include /etc/nginx/mime.types;`) is neither captured nor an error — absence of a
|
||||||
|
# document the parser needs nothing from is the shape the agent already reports as "costs
|
||||||
|
# nothing and no line".
|
||||||
|
if capture gateway_nginx_main.conf config /etc/nginx/nginx.conf; then
|
||||||
|
globs=()
|
||||||
|
while IFS= read -r line; do
|
||||||
|
probe="${line#"${line%%[![:space:]]*}"}" # leading whitespace off, for the two tests below
|
||||||
|
case "$probe" in ''|'#'*) continue ;; esac
|
||||||
|
case "$probe" in include*) ;; *) continue ;; esac
|
||||||
|
spec="${probe#include}"
|
||||||
|
spec="${spec%%;*}"
|
||||||
|
spec="${spec//[[:space:]]/}"
|
||||||
|
# Anything with a `*` is a glob include, and the agent merges it — so it gets a fixture, or
|
||||||
|
# the report below says the script has no name for it. Silent skipping is what would let the
|
||||||
|
# fixture set stop being the tree the agent reads.
|
||||||
|
case "$spec" in *"*)"*) continue ;; esac # not a directive then: a fetch expression
|
||||||
|
case "$spec" in *"*"*) globs+=("$spec") ;; esac
|
||||||
|
done <"$FIX/gateway_nginx_main.conf"
|
||||||
|
if [ ${#globs[@]} -eq 0 ]; then
|
||||||
|
echo " ✗ gateway_nginx_*.conf — the captured main config names no glob include, so the"
|
||||||
|
echo " vhost fixtures were not refreshed: has the tree the agent merges moved?"
|
||||||
|
failed=1
|
||||||
|
fi
|
||||||
|
for spec in "${globs[@]}"; do
|
||||||
|
case "$spec" in
|
||||||
|
/etc/nginx/conf.d/*.conf) capture_included gateway_nginx_conf_d_ "${spec%/*}" ;;
|
||||||
|
/etc/nginx/stream.d/*.conf) capture_included gateway_nginx_stream_d_ "${spec%/*}" ;;
|
||||||
|
*)
|
||||||
|
echo " ✗ $spec — an include directory this script has no fixture name for"
|
||||||
|
echo " (add a case above and a fixture-pinned test in tests/test_agent.py)"
|
||||||
|
failed=1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- the live sockets --------------------------------------------------------
|
||||||
|
# Both tables, always: `available` is decided across them, because a kernel built without IPv6
|
||||||
|
# answers tcp and not tcp6 and that is absence rather than a fault (no line, the include rule).
|
||||||
|
# Keeping only LISTEN and ESTABLISHED is what makes a capture of a busy gateway usable — the
|
||||||
|
# TIME_WAIT rows in here are the ones the parser must skip, which is exactly what
|
||||||
|
# tests/test_agent.py asserts against this file.
|
||||||
|
capture gateway_proc_net_tcp.txt sockets /proc/net/tcp
|
||||||
|
capture gateway_proc_net_tcp6.txt sockets /proc/net/tcp6
|
||||||
|
|
||||||
|
if [ "$failed" -ne 0 ]; then
|
||||||
|
echo
|
||||||
|
echo "capture FAILED — every fixture named above is unchanged on disk. Is $GATEWAY_SSH an ssh"
|
||||||
|
echo "alias that answers (ssh $GATEWAY_SSH true), and are the paths still where they were?"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "done — read the diff before committing the fixtures."
|
||||||
@@ -36,6 +36,22 @@ class UpstreamError(Exception):
|
|||||||
self.status = status
|
self.status = status
|
||||||
|
|
||||||
|
|
||||||
|
#: Agent add-on sections that report their own absence on purpose. Their `errors` lines are
|
||||||
|
#: facts about a source this page does not draw — `smart:` belongs to `/disks`, `haproxy:` and
|
||||||
|
#: `nginx:` to the gateway page (phase 29) — so they never make the **system** payload
|
||||||
|
#: `degraded`, which is the overview's alert input. Without this a fleet-wide agent upgrade
|
||||||
|
#: would put a red "agent degraded" banner and an overview WARN on every host that has no
|
||||||
|
#: haproxy or nginx installed, which is invariant 3's failure mode with an upgrade's clothes on.
|
||||||
|
#: The page that owns the add-on still reads the lines: they stay verbatim in `agent.errors`.
|
||||||
|
#:
|
||||||
|
#: **A new agent add-on section adds its prefix here in the same commit.** The list the agent's
|
||||||
|
#: own tests filter by is pinned equal to this one
|
||||||
|
#: (`tests/test_agent.py::test_the_addon_error_prefixes_are_the_same_list_the_dashboard_filters`),
|
||||||
|
#: so a section that ships its absence-report without a prefix here fails instead of turning
|
||||||
|
#: every host amber.
|
||||||
|
ADDON_ERROR_PREFIXES: tuple[str, ...] = ("smart", "haproxy", "nginx", "sockets")
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# System hosts
|
# System hosts
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
@@ -221,12 +237,15 @@ async def fetch_host_snapshot(client: httpx.AsyncClient, target: Target,
|
|||||||
if body.get("errors"):
|
if body.get("errors"):
|
||||||
# The agent degraded a section (a /proc read failed) but still answered. Say so
|
# The agent degraded a section (a /proc read failed) but still answered. Say so
|
||||||
# in the payload rather than presenting a partial snapshot as a complete one —
|
# in the payload rather than presenting a partial snapshot as a complete one —
|
||||||
# except the SMART add-on: a host without smartmontools or without device read
|
# except the add-on sections: a host without smartmontools or without device read
|
||||||
# access is *unconfigured, not broken* (invariant 3), and its `smart:` lines must
|
# access, without a proxy config on it, or whose kernel published no socket table, is
|
||||||
# not hold the system page amber forever. They stay in the snapshot payload
|
# *unconfigured, not broken* (invariant 3), and those lines must not hold the system page
|
||||||
# (`agent.errors`) where the /disks page explains them in its own panel.
|
# amber forever. They stay in the snapshot payload (`agent.errors`) where the page that
|
||||||
|
# owns the add-on explains them — `/disks` reads the `smart:` ones, phase 29's gateway
|
||||||
|
# page reads the `haproxy:`/`nginx:`/`sockets:` ones.
|
||||||
errors = body["errors"]
|
errors = body["errors"]
|
||||||
core = ([line for line in errors if not str(line).startswith("smart")]
|
core = ([line for line in errors
|
||||||
|
if not str(line).lstrip().lower().startswith(ADDON_ERROR_PREFIXES)]
|
||||||
if isinstance(errors, list) else errors)
|
if isinstance(errors, list) else errors)
|
||||||
if core:
|
if core:
|
||||||
result.payload["degraded"] = core
|
result.payload["degraded"] = core
|
||||||
|
|||||||
Vendored
+57
@@ -0,0 +1,57 @@
|
|||||||
|
# Verbatim /etc/haproxy/haproxy.cfg from gateway-ops, read read-only over the ssh key by
|
||||||
|
# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the
|
||||||
|
# deployed upstream, never guess its shape).
|
||||||
|
# Nothing here is rewritten: it is this fleet's ingress map, and the domains in it are
|
||||||
|
# the data — an inbound name and the target it routes to are what these config sections
|
||||||
|
# exist to report. Key material appears only as the file paths the proxy reads.
|
||||||
|
defaults
|
||||||
|
log stdout local0 info
|
||||||
|
mode tcp
|
||||||
|
option tcplog
|
||||||
|
timeout connect 10s
|
||||||
|
timeout client 30s
|
||||||
|
timeout server 30s
|
||||||
|
timeout tunnel 3600s
|
||||||
|
default-server init-addr last,libc,none
|
||||||
|
|
||||||
|
frontend haproxy-22
|
||||||
|
bind :::22 v4v6
|
||||||
|
default_backend gitea-2222
|
||||||
|
|
||||||
|
frontend haproxy-2000
|
||||||
|
bind :::2000 v4v6
|
||||||
|
default_backend sec-gitea-2000
|
||||||
|
|
||||||
|
frontend haproxy-5201
|
||||||
|
bind :::5201 v4v6
|
||||||
|
default_backend iperf3-5201
|
||||||
|
|
||||||
|
frontend haproxy-8080
|
||||||
|
bind :::8080 v4v6
|
||||||
|
default_backend unifi-external-8080
|
||||||
|
|
||||||
|
frontend haproxy-25565
|
||||||
|
bind :::25565 v4v6
|
||||||
|
default_backend testcraft-25565
|
||||||
|
|
||||||
|
frontend haproxy-25566
|
||||||
|
bind :::25566 v4v6
|
||||||
|
default_backend nimcraft-25566
|
||||||
|
|
||||||
|
backend gitea-2222
|
||||||
|
server gitea reeseapps-gitea.reeselink.com:2222 check inter 5s
|
||||||
|
|
||||||
|
backend sec-gitea-2000
|
||||||
|
server sec-gitea stackexpected-gitea.reeselink.com:2222 check inter 5s
|
||||||
|
|
||||||
|
backend iperf3-5201
|
||||||
|
server iperf3 web.reeselink.com:5201 check inter 5s
|
||||||
|
|
||||||
|
backend unifi-external-8080
|
||||||
|
server unifi unifi-external.reeselink.com:8080 check inter 5s
|
||||||
|
|
||||||
|
backend testcraft-25565
|
||||||
|
server mc1 minecraft.reeselink.com:25565 check inter 5s
|
||||||
|
|
||||||
|
backend nimcraft-25566
|
||||||
|
server mc2 minecraft.reeselink.com:25566 check inter 5s
|
||||||
+214
@@ -0,0 +1,214 @@
|
|||||||
|
# Verbatim /etc/nginx/conf.d/http.conf from gateway-ops, read read-only over the ssh key by
|
||||||
|
# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the
|
||||||
|
# deployed upstream, never guess its shape).
|
||||||
|
# Nothing here is rewritten: it is this fleet's ingress map, and the domains in it are
|
||||||
|
# the data — an inbound name and the target it routes to are what these config sections
|
||||||
|
# exist to report. Key material appears only as the file paths the proxy reads.
|
||||||
|
server {
|
||||||
|
listen 4443 ssl proxy_protocol;
|
||||||
|
listen [::]:4443 ssl proxy_protocol;
|
||||||
|
server_name nextcloud.reeseapps.com;
|
||||||
|
client_max_body_size 0;
|
||||||
|
|
||||||
|
ssl_certificate /etc/letsencrypt/live/nextcloud.reeseapps.com/fullchain.pem;
|
||||||
|
ssl_certificate_key /etc/letsencrypt/live/nextcloud.reeseapps.com/privkey.pem;
|
||||||
|
|
||||||
|
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||||
|
add_header X-Content-Type-Options "nosniff" always;
|
||||||
|
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||||
|
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://nextcloud.reeselink.com:11000;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_set_header Upgrade $http_upgrade;
|
||||||
|
proxy_set_header Connection $connection_upgrade;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 8443 ssl;
|
||||||
|
listen [::]:8443 ssl;
|
||||||
|
server_name nextcloud.reeseapps.com;
|
||||||
|
client_max_body_size 0;
|
||||||
|
|
||||||
|
ssl_certificate /etc/letsencrypt/live/nextcloud.reeseapps.com/fullchain.pem;
|
||||||
|
ssl_certificate_key /etc/letsencrypt/live/nextcloud.reeseapps.com/privkey.pem;
|
||||||
|
|
||||||
|
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||||
|
add_header X-Content-Type-Options "nosniff" always;
|
||||||
|
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||||
|
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://nextcloud.reeselink.com:11001;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_set_header Upgrade $http_upgrade;
|
||||||
|
proxy_set_header Connection $connection_upgrade;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 4443 ssl proxy_protocol;
|
||||||
|
listen [::]:4443 ssl proxy_protocol;
|
||||||
|
server_name gitea.reeseapps.com;
|
||||||
|
client_max_body_size 0;
|
||||||
|
|
||||||
|
ssl_certificate /etc/letsencrypt/live/gitea.reeseapps.com/fullchain.pem;
|
||||||
|
ssl_certificate_key /etc/letsencrypt/live/gitea.reeseapps.com/privkey.pem;
|
||||||
|
|
||||||
|
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||||
|
add_header X-Content-Type-Options "nosniff" always;
|
||||||
|
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||||
|
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://gitea.reeselink.com:3000;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_set_header Upgrade $http_upgrade;
|
||||||
|
proxy_set_header Connection $connection_upgrade;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 4443 ssl proxy_protocol;
|
||||||
|
listen [::]:4443 ssl proxy_protocol;
|
||||||
|
server_name jellyfin.reeseapps.com;
|
||||||
|
client_max_body_size 0;
|
||||||
|
|
||||||
|
ssl_certificate /etc/letsencrypt/live/jellyfin.reeseapps.com/fullchain.pem;
|
||||||
|
ssl_certificate_key /etc/letsencrypt/live/jellyfin.reeseapps.com/privkey.pem;
|
||||||
|
|
||||||
|
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||||
|
add_header X-Content-Type-Options "nosniff" always;
|
||||||
|
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||||
|
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://jellyfin.reeselink.com:8096;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_set_header Upgrade $http_upgrade;
|
||||||
|
proxy_set_header Connection $connection_upgrade;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 8443 ssl;
|
||||||
|
listen [::]:8443 ssl;
|
||||||
|
server_name alicia-ai-terminology.reeseapps.com;
|
||||||
|
|
||||||
|
ssl_certificate /etc/letsencrypt/live/alicia-ai-terminology.reeseapps.com/fullchain.pem;
|
||||||
|
ssl_certificate_key /etc/letsencrypt/live/alicia-ai-terminology.reeseapps.com/privkey.pem;
|
||||||
|
|
||||||
|
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||||
|
add_header X-Content-Type-Options "nosniff" always;
|
||||||
|
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||||
|
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://127.0.0.1:8081;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_set_header Upgrade $http_upgrade;
|
||||||
|
proxy_set_header Connection $connection_upgrade;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 4443 ssl proxy_protocol;
|
||||||
|
listen [::]:4443 ssl proxy_protocol;
|
||||||
|
server_name connors-legos.reeseapps.com;
|
||||||
|
|
||||||
|
ssl_certificate /etc/letsencrypt/live/connors-legos.reeseapps.com/fullchain.pem;
|
||||||
|
ssl_certificate_key /etc/letsencrypt/live/connors-legos.reeseapps.com/privkey.pem;
|
||||||
|
|
||||||
|
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||||
|
add_header X-Content-Type-Options "nosniff" always;
|
||||||
|
add_header X-Frame-Options "SAMEORIGIN" always;
|
||||||
|
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://bricktracker.reeselink.com:3333;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_set_header Upgrade $http_upgrade;
|
||||||
|
proxy_set_header Connection $connection_upgrade;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 4443 ssl proxy_protocol;
|
||||||
|
listen [::]:4443 ssl proxy_protocol;
|
||||||
|
server_name keycloak.reeseapps.com;
|
||||||
|
client_max_body_size 0;
|
||||||
|
|
||||||
|
ssl_certificate /etc/letsencrypt/live/keycloak.reeseapps.com/fullchain.pem;
|
||||||
|
ssl_certificate_key /etc/letsencrypt/live/keycloak.reeseapps.com/privkey.pem;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://keycloak.reeselink.com:8080;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_set_header Upgrade $http_upgrade;
|
||||||
|
proxy_set_header Connection $connection_upgrade;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 8443 ssl;
|
||||||
|
listen [::]:8443 ssl;
|
||||||
|
server_name admin.keycloak.reeseapps.com;
|
||||||
|
client_max_body_size 0;
|
||||||
|
|
||||||
|
ssl_certificate /etc/letsencrypt/live/admin.keycloak.reeseapps.com/fullchain.pem;
|
||||||
|
ssl_certificate_key /etc/letsencrypt/live/admin.keycloak.reeseapps.com/privkey.pem;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://keycloak.reeselink.com:8080;
|
||||||
|
proxy_set_header Host $host:$server_port;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_set_header X-Forwarded-Port $server_port;
|
||||||
|
proxy_set_header X-Forwarded-Host $host:$server_port;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 4443 ssl proxy_protocol;
|
||||||
|
listen [::]:4443 ssl proxy_protocol;
|
||||||
|
server_name emmaleaf.reeseapps.com;
|
||||||
|
client_max_body_size 10m;
|
||||||
|
|
||||||
|
ssl_certificate /etc/letsencrypt/live/emmaleaf.reeseapps.com/fullchain.pem;
|
||||||
|
ssl_certificate_key /etc/letsencrypt/live/emmaleaf.reeseapps.com/privkey.pem;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://emmaleaf.reeselink.com:80;
|
||||||
|
proxy_ssl_verify off;
|
||||||
|
proxy_set_header Host $host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto $scheme;
|
||||||
|
proxy_set_header Upgrade $http_upgrade;
|
||||||
|
proxy_set_header Connection $connection_upgrade;
|
||||||
|
}
|
||||||
|
}
|
||||||
+63
@@ -0,0 +1,63 @@
|
|||||||
|
# Verbatim /etc/nginx/nginx.conf from gateway-ops, read read-only over the ssh key by
|
||||||
|
# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the
|
||||||
|
# deployed upstream, never guess its shape).
|
||||||
|
# Nothing here is rewritten: it is this fleet's ingress map, and the domains in it are
|
||||||
|
# the data — an inbound name and the target it routes to are what these config sections
|
||||||
|
# exist to report. Key material appears only as the file paths the proxy reads.
|
||||||
|
user nginx;
|
||||||
|
worker_processes auto;
|
||||||
|
|
||||||
|
error_log /var/log/nginx/error.log notice;
|
||||||
|
pid /run/nginx.pid;
|
||||||
|
|
||||||
|
|
||||||
|
events {
|
||||||
|
worker_connections 1024;
|
||||||
|
}
|
||||||
|
|
||||||
|
stream {
|
||||||
|
include /etc/nginx/stream.d/*.conf;
|
||||||
|
}
|
||||||
|
|
||||||
|
http {
|
||||||
|
real_ip_header proxy_protocol;
|
||||||
|
set_real_ip_from 127.0.0.1;
|
||||||
|
set_real_ip_from 10.0.0.0/8;
|
||||||
|
|
||||||
|
include /etc/nginx/mime.types;
|
||||||
|
default_type application/octet-stream;
|
||||||
|
|
||||||
|
log_format detailed '$remote_addr:$remote_port [$time_local] "$server_name" "$request" '
|
||||||
|
'$status "$body_bytes_sent" "$http_referer" "$http_user_agent" '
|
||||||
|
'$request_time "$upstream_addr" "$upstream_response_time" '
|
||||||
|
'$ssl_protocol/$ssl_cipher';
|
||||||
|
|
||||||
|
access_log /var/log/nginx/access.log detailed;
|
||||||
|
|
||||||
|
sendfile on;
|
||||||
|
tcp_nopush on;
|
||||||
|
tcp_nodelay on;
|
||||||
|
|
||||||
|
keepalive_timeout 65;
|
||||||
|
|
||||||
|
gzip on;
|
||||||
|
|
||||||
|
server_tokens off;
|
||||||
|
|
||||||
|
ssl_protocols TLSv1.2 TLSv1.3;
|
||||||
|
ssl_session_cache shared:SSL:10m;
|
||||||
|
ssl_session_timeout 1d;
|
||||||
|
|
||||||
|
map $http_upgrade $connection_upgrade {
|
||||||
|
default upgrade;
|
||||||
|
'' close;
|
||||||
|
}
|
||||||
|
server {
|
||||||
|
listen 80;
|
||||||
|
listen [::]:80;
|
||||||
|
server_name _;
|
||||||
|
return 301 https://$host$request_uri;
|
||||||
|
}
|
||||||
|
|
||||||
|
include /etc/nginx/conf.d/*.conf;
|
||||||
|
}
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
# Verbatim /etc/nginx/stream.d/stream.conf from gateway-ops, read read-only over the ssh key by
|
||||||
|
# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the
|
||||||
|
# deployed upstream, never guess its shape).
|
||||||
|
# Nothing here is rewritten: it is this fleet's ingress map, and the domains in it are
|
||||||
|
# the data — an inbound name and the target it routes to are what these config sections
|
||||||
|
# exist to report. Key material appears only as the file paths the proxy reads.
|
||||||
|
map $ssl_preread_server_name $targetBackend {
|
||||||
|
default 127.0.0.1:4443;
|
||||||
|
homepage.reeseapps.com reeseapps-homepage.reeselink.com:443;
|
||||||
|
ntfy.reeseapps.com reeseapps-ntfy.reeselink.com:443;
|
||||||
|
gitea.reeseapps.com reeseapps-gitea.reeselink.com:443;
|
||||||
|
aipi.reeseapps.com reeseapps-litellm.reeselink.com:443;
|
||||||
|
chatreesept.reeseapps.com reeseapps-openwebui.reeselink.com:443;
|
||||||
|
keycloak.reeseapps.com reeseapps-keycloak.reeselink.com:443;
|
||||||
|
forgejo.reeseapps.com reeseapps-forgejo.reeselink.com:443;
|
||||||
|
brain.reeseapps.com reeseapps-brain-of-reese.reeselink.com:443;
|
||||||
|
brain-of-code.reeseapps.com reeseapps-brain-of-code.reeselink.com:443;
|
||||||
|
immich.reeseapps.com reeseapps-immich.reeselink.com:443;
|
||||||
|
|
||||||
|
stackexpected.com stackexpected.reeselink.com:443;
|
||||||
|
gitea.stackexpected.com stackexpected-gitea.reeselink.com:443;
|
||||||
|
keycloak.stackexpected.com stackexpected-keycloak.reeselink.com:443;
|
||||||
|
nextcloud.stackexpected.com stackexpected-nextcloud.reeselink.com:443;
|
||||||
|
hedgedoc.stackexpected.com stackexpected-hedgedoc.reeselink.com:443;
|
||||||
|
wellsfamilycounseling.reeseapps.com wfc.reeselink.com:443;
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 443;
|
||||||
|
listen [::]:443;
|
||||||
|
|
||||||
|
proxy_connect_timeout 1800s;
|
||||||
|
proxy_timeout 1800s;
|
||||||
|
proxy_protocol on;
|
||||||
|
|
||||||
|
proxy_pass $targetBackend;
|
||||||
|
ssl_preread on;
|
||||||
|
|
||||||
|
resolver 10.1.0.1;
|
||||||
|
}
|
||||||
+58
@@ -0,0 +1,58 @@
|
|||||||
|
# Verbatim /proc/net/tcp from gateway-ops, read read-only over the ssh key by
|
||||||
|
# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the
|
||||||
|
# deployed upstream, never guess its shape).
|
||||||
|
# One line per TCP socket, in the kernel's own spelling. The only edit is the repo's
|
||||||
|
# identifier rule (invariant 6, the drive-serial scrub in capture-fixtures.sh): a
|
||||||
|
# globally routable address became a documentation one (RFC 5737 / RFC 3849), because a
|
||||||
|
# global IPv6 here is EUI-64 and a visitor's address is not ours to publish. Every port,
|
||||||
|
# state, queue, uid, inode and column width is verbatim, as are the loopback, private
|
||||||
|
# and wildcard addresses the gateway's own frontends use.
|
||||||
|
# Volatile by nature: a proxy in service has a different connection set every second, so
|
||||||
|
# a diff here on re-capture is expected. The tests assert shape, the two states the agent
|
||||||
|
# keeps, and the ports the captured configs claim to bind — never a particular row.
|
||||||
|
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
|
||||||
|
0: 00000000:20FB 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9978 1 0000000000000000 100 0 0 10 0
|
||||||
|
1: 00000000:08AE 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7797 1 0000000000000000 100 0 0 10 0
|
||||||
|
2: 00000000:0050 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9974 1 0000000000000000 100 0 0 10 0
|
||||||
|
3: 00000000:01BB 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9972 1 0000000000000000 100 0 0 10 0
|
||||||
|
4: 00000000:115B 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9976 1 0000000000000000 100 0 0 10 0
|
||||||
|
5: 1B01010A:23A0 00000000:0000 0A 00000000:00000000 00:00000000 00000000 994 0 9479 1 0000000000000000 100 0 0 10 0
|
||||||
|
6: 1B01010A:BE2A B001010A:01BB 06 00000000:00000000 03:000013F5 00000000 0 0 0 3 0000000000000000
|
||||||
|
7: 0100007F:115B 0100007F:818C 01 00000000:00000000 00:00000000 00000000 101 0 62860 1 0000000000000000 20 4 30 10 -1
|
||||||
|
8: 1B01010A:BE2E B001010A:01BB 06 00000000:00000000 03:00001407 00000000 0 0 0 3 0000000000000000
|
||||||
|
9: 1B01010A:23A0 0204A8C0:C626 06 00000000:00000000 03:00000820 00000000 0 0 0 3 0000000000000000
|
||||||
|
10: 0100007F:115B 0100007F:C096 01 00000000:00000000 00:00000000 00000000 101 0 49558 1 0000000000000000 20 4 26 10 -1
|
||||||
|
11: 0100007F:115B 0100007F:B502 06 00000000:00000000 03:00000841 00000000 0 0 0 3 0000000000000000
|
||||||
|
12: 1B01010A:8290 A401010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 11850 1 0000000000000000 20 4 28 10 -1
|
||||||
|
13: 1B01010A:01BB 017100CB:90FA 01 00000000:00000000 00:00000000 00000000 101 0 21476 1 0000000000000000 35 4 31 10 -1
|
||||||
|
14: 1B01010A:23A0 0204A8C0:992E 06 00000000:00000000 03:00000C0A 00000000 0 0 0 3 0000000000000000
|
||||||
|
15: 1B01010A:CB4C B001010A:01BB 06 00000000:00000000 03:00000ECB 00000000 0 0 0 3 0000000000000000
|
||||||
|
16: 1B01010A:9C42 2700010A:01BB 06 00000000:00000000 03:00001259 00000000 0 0 0 3 0000000000000000
|
||||||
|
17: 1B01010A:01BB 0100010A:B2AC 01 00000000:00000000 00:00000000 00000000 101 0 77631 1 0000000000000000 20 4 31 10 -1
|
||||||
|
18: 1B01010A:01BB E600020A:BBE8 06 00000000:00000000 03:00000992 00000000 0 0 0 3 0000000000000000
|
||||||
|
19: 0100007F:C99E 0100007F:115B 06 00000000:00000000 03:000016BE 00000000 0 0 0 3 0000000000000000
|
||||||
|
20: 1B01010A:C1FE 2700010A:01BB 06 00000000:00000000 03:00000662 00000000 0 0 0 3 0000000000000000
|
||||||
|
21: 1B01010A:01BB E600020A:B5C4 06 00000000:00000000 03:000005EE 00000000 0 0 0 3 0000000000000000
|
||||||
|
22: 1B01010A:840E B001010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 21477 1 0000000000000000 20 4 30 10 -1
|
||||||
|
23: 1B01010A:D22A 4E01010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 77632 1 0000000000000000 21 4 30 10 -1
|
||||||
|
24: 1B01010A:01BB 017100CB:956A 01 00000000:00000000 00:00000000 00000000 101 0 62858 1 0000000000000000 29 4 30 10 -1
|
||||||
|
25: 0100007F:115B 0100007F:B4F6 06 00000000:00000000 03:000005ED 00000000 0 0 0 3 0000000000000000
|
||||||
|
26: 1B01010A:23A0 0204A8C0:8EE6 06 00000000:00000000 03:00001408 00000000 0 0 0 3 0000000000000000
|
||||||
|
27: 1B01010A:01BB EC01010A:94DE 01 00000000:00000000 00:00000000 00000000 101 0 11849 1 0000000000000000 20 4 31 10 -1
|
||||||
|
28: 0100007F:818C 0100007F:115B 01 00000000:00000000 00:00000000 00000000 101 0 62859 1 0000000000000000 21 4 24 10 -1
|
||||||
|
29: 1B01010A:23A0 0204A8C0:D5DE 06 00000000:00000000 03:0000020F 00000000 0 0 0 3 0000000000000000
|
||||||
|
30: 0100007F:115B 0100007F:B4F8 06 00000000:00000000 03:000005F6 00000000 0 0 0 3 0000000000000000
|
||||||
|
31: 1B01010A:A828 7600010A:63DD 06 00000000:00000000 03:00000F7F 00000000 0 0 0 3 0000000000000000
|
||||||
|
32: 0100007F:C096 0100007F:115B 01 00000000:00000000 00:00000000 00000000 101 0 49557 1 0000000000000000 20 4 24 10 -1
|
||||||
|
33: 1B01010A:D444 FE00010A:2AF8 01 00000000:00000000 00:00000000 00000000 101 0 49559 1 0000000000000000 20 4 30 10 -1
|
||||||
|
34: 1B01010A:9032 B001010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 11954 1 0000000000000000 20 4 30 10 -1
|
||||||
|
35: 1B01010A:977E 2700010A:01BB 06 00000000:00000000 03:00000C72 00000000 0 0 0 3 0000000000000000
|
||||||
|
36: 0100007F:115B 0100007F:B4EA 06 00000000:00000000 03:000005F0 00000000 0 0 0 3 0000000000000000
|
||||||
|
37: 1B01010A:08AE 0204A8C0:8290 01 00000048:00000000 01:00000017 00000000 0 0 86478 4 0000000000000000 24 8 15 10 -1
|
||||||
|
38: 1B01010A:9ED0 2700010A:01BB 06 00000000:00000000 03:0000170F 00000000 0 0 0 3 0000000000000000
|
||||||
|
39: 1B01010A:0050 027100CB:843E 01 0000018C:00000000 01:00000E57 00000007 101 0 84306 2 0000000000000000 4864 4 31 1 7
|
||||||
|
40: 1B01010A:CB5C B001010A:01BB 06 00000000:00000000 03:00000ED6 00000000 0 0 0 3 0000000000000000
|
||||||
|
41: 1B01010A:9776 2700010A:01BB 06 00000000:00000000 03:00000875 00000000 0 0 0 3 0000000000000000
|
||||||
|
42: 1B01010A:C59C FE00010A:2AF8 01 00000000:00000000 00:00000000 00000000 101 0 62861 1 0000000000000000 21 4 30 10 -1
|
||||||
|
43: 1B01010A:9154 2700010A:01BB 06 00000000:00000000 03:00001572 00000000 0 0 0 3 0000000000000000
|
||||||
|
44: 1B01010A:23A0 0204A8C0:9C42 06 00000000:00000000 03:00000FF3 00000000 0 0 0 3 0000000000000000
|
||||||
+26
@@ -0,0 +1,26 @@
|
|||||||
|
# Verbatim /proc/net/tcp6 from gateway-ops, read read-only over the ssh key by
|
||||||
|
# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the
|
||||||
|
# deployed upstream, never guess its shape).
|
||||||
|
# One line per TCP socket, in the kernel's own spelling. The only edit is the repo's
|
||||||
|
# identifier rule (invariant 6, the drive-serial scrub in capture-fixtures.sh): a
|
||||||
|
# globally routable address became a documentation one (RFC 5737 / RFC 3849), because a
|
||||||
|
# global IPv6 here is EUI-64 and a visitor's address is not ours to publish. Every port,
|
||||||
|
# state, queue, uid, inode and column width is verbatim, as are the loopback, private
|
||||||
|
# and wildcard addresses the gateway's own frontends use.
|
||||||
|
# Volatile by nature: a proxy in service has a different connection set every second, so
|
||||||
|
# a diff here on re-capture is expected. The tests assert shape, the two states the agent
|
||||||
|
# keeps, and the ports the captured configs claim to bind — never a particular row.
|
||||||
|
sl local_address remote_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
|
||||||
|
0: 00000000000000000000000000000000:20FB 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9979 1 0000000000000000 100 0 0 10 0
|
||||||
|
1: 00000000000000000000000000000000:08AE 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7799 1 0000000000000000 100 0 0 10 0
|
||||||
|
2: 00000000000000000000000000000000:1451 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9181 1 0000000000000000 100 0 0 10 0
|
||||||
|
3: 00000000000000000000000000000000:0050 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9975 1 0000000000000000 100 0 0 10 0
|
||||||
|
4: 00000000000000000000000000000000:0016 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9179 1 0000000000000000 100 0 0 10 0
|
||||||
|
5: 00000000000000000000000000000000:01BB 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9973 1 0000000000000000 100 0 0 10 0
|
||||||
|
6: 00000000000000000000000000000000:115B 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9977 1 0000000000000000 100 0 0 10 0
|
||||||
|
7: 00000000000000000000000000000000:63DE 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9184 1 0000000000000000 100 0 0 10 0
|
||||||
|
8: 00000000000000000000000000000000:63DD 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9183 1 0000000000000000 100 0 0 10 0
|
||||||
|
9: 00000000000000000000000000000000:07D0 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9180 1 0000000000000000 100 0 0 10 0
|
||||||
|
10: 00000000000000000000000000000000:1F90 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9182 1 0000000000000000 100 0 0 10 0
|
||||||
|
11: B80D0120000000000000000001000000:01BB 2377D3FDF349E0970000000003000000:C0AC 01 00000000:00000000 00:00000000 00000000 101 0 49556 1 0000000000000000 80 4 26 10 -1
|
||||||
|
12: B80D0120000000000000000001000000:01BB 2377D3FDF349E0970000000003000000:A120 01 00000000:00000000 00:00000000 00000000 101 0 11953 1 0000000000000000 59 4 30 7 7
|
||||||
+1662
-11
File diff suppressed because it is too large
Load Diff
@@ -249,6 +249,55 @@ async def test_real_degradations_still_show_when_smart_noise_is_alongside() -> N
|
|||||||
assert result.payload["degraded"] == ["temperatures: no hwmon sensors"]
|
assert result.payload["degraded"] == ["temperatures: no hwmon sensors"]
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_host_without_a_proxy_config_does_not_degrade_the_system_source() -> None:
|
||||||
|
"""The proxy add-on's absence lines, same rule and same reason as `smart:` above: two of this
|
||||||
|
fleet's three hosts run no haproxy and no nginx, so an agent upgrade must not put a
|
||||||
|
`degraded` payload (and therefore an overview alert, `alerts.py` reads this key) on a box that
|
||||||
|
never had either. The lines survive in `agent.errors` for the page that owns them — phase 29's
|
||||||
|
gateway source reads them the way `/disks` reads `smart_errors`.
|
||||||
|
|
||||||
|
`collectors.ADDON_ERROR_PREFIXES` is the one list of these prefixes; this asserts every
|
||||||
|
name by behaviour, so dropping any of them fails here, and `tests/test_agent.py` pins both
|
||||||
|
the wording of the lines the agent actually writes and the equality of the two lists.
|
||||||
|
"""
|
||||||
|
missing_haproxy = ("haproxy: /etc/haproxy/haproxy.cfg not found — haproxy is not installed "
|
||||||
|
"here, or its config lives elsewhere (--haproxy-config); --no-proxy "
|
||||||
|
"opts out")
|
||||||
|
missing_nginx = ("nginx: /etc/nginx/nginx.conf not found — nginx is not installed here, or "
|
||||||
|
"its config lives elsewhere (--nginx-config); --no-proxy opts out")
|
||||||
|
no_sockets = ("sockets: /proc/net/tcp not found, /proc/net/tcp6 not found — this host "
|
||||||
|
"publishes no kernel socket table; --no-proxy opts out")
|
||||||
|
body = agent_body()
|
||||||
|
body["errors"] = [missing_haproxy, missing_nginx, no_sockets,
|
||||||
|
"smart: smartctl 7.5 not found"]
|
||||||
|
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
return httpx.Response(200, json=body)
|
||||||
|
|
||||||
|
async with client_for(handler) as client:
|
||||||
|
result = await fetch_host_snapshot(client, Target("gw", "http://agent.test"),
|
||||||
|
HostCollector(RateEstimator()))
|
||||||
|
assert "degraded" not in result.payload, "unconfigured is not broken (invariant 3)"
|
||||||
|
for prefix in ("haproxy:", "nginx:", "sockets:"):
|
||||||
|
assert any(prefix in str(e) for e in result.payload["agent"]["errors"]), \
|
||||||
|
"the sentence survives for the page that explains it"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_a_real_degradation_shows_alongside_the_proxy_addon_noise() -> None:
|
||||||
|
"""Filtering the add-ons must not swallow a genuine half-snapshot beside them."""
|
||||||
|
body = agent_body()
|
||||||
|
body["errors"] = ["nginx: /etc/nginx/nginx.conf is there but could not be read: boom",
|
||||||
|
"temperatures: no hwmon sensors"]
|
||||||
|
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
return httpx.Response(200, json=body)
|
||||||
|
|
||||||
|
async with client_for(handler) as client:
|
||||||
|
result = await fetch_host_snapshot(client, Target("gw", "http://agent.test"),
|
||||||
|
HostCollector(RateEstimator()))
|
||||||
|
assert result.payload["degraded"] == ["temperatures: no hwmon sensors"]
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# physical disks (the /disks source's fetch half)
|
# physical disks (the /disks source's fetch half)
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
|
|||||||
@@ -1,4 +1,12 @@
|
|||||||
"""Contract tests for the scripts an operator runs: `deploy/` and `scripts/dev-*.sh`/`.py`.
|
"""Contract tests for the scripts an operator runs: `deploy/`, `scripts/dev-*.sh`/`.py`, and
|
||||||
|
`scripts/capture-gateway-config.sh`.
|
||||||
|
|
||||||
|
The capture script gets the same treatment as the installer because it can do the same kind of
|
||||||
|
damage quietly: pointed at a gateway it cannot read, it is six empty files where six verbatim
|
||||||
|
captures were, and every parser test that reads them then passes against nothing. So it is run
|
||||||
|
here, against a stub `ssh` and a fake remote tree, and the assertions are about what lands on
|
||||||
|
disk — including the one edit it makes (a globally routable socket address becomes documentation
|
||||||
|
space) and the thing it never does (rewrite a config).
|
||||||
|
|
||||||
There is no shell test framework here — and adding one would buy nothing a subprocess does
|
There is no shell test framework here — and adding one would buy nothing a subprocess does
|
||||||
not — but phase 10 rests on a sentence that lives in **two** files: the sudoers grant in
|
not — but phase 10 rests on a sentence that lives in **two** files: the sudoers grant in
|
||||||
@@ -46,6 +54,7 @@ from dashboards.store import SampleStore
|
|||||||
|
|
||||||
ROOT = Path(__file__).resolve().parent.parent
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
DEPLOY = ROOT / "deploy"
|
DEPLOY = ROOT / "deploy"
|
||||||
|
SCRIPTS = ROOT / "scripts"
|
||||||
INSTALLER = DEPLOY / "install-agent.sh"
|
INSTALLER = DEPLOY / "install-agent.sh"
|
||||||
SUDOERS = DEPLOY / "agent-sudoers"
|
SUDOERS = DEPLOY / "agent-sudoers"
|
||||||
BASH = shutil.which("bash") or "/bin/bash"
|
BASH = shutil.which("bash") or "/bin/bash"
|
||||||
@@ -2334,3 +2343,379 @@ def test_a_local_bundle_still_validates_with_its_own_flag_name(tmp_path: Path) -
|
|||||||
assert result.returncode == 2
|
assert result.returncode == 2
|
||||||
assert result.stderr.startswith("--cert-bundle"), result.stderr
|
assert result.stderr.startswith("--cert-bundle"), result.stderr
|
||||||
assert "--fetch-bundle" not in result.stderr
|
assert "--fetch-bundle" not in result.stderr
|
||||||
|
|
||||||
|
|
||||||
|
# --- the gateway capture (phase 28 task 04) ------------------------------------------------
|
||||||
|
|
||||||
|
#: The script that records `tests/fixtures/gateway_*` — the ground truth the agent's `haproxy`,
|
||||||
|
#: `nginx` and `sockets` parsers are pinned to. It has no Python-side coverage to speak of (the
|
||||||
|
#: project's coverage source is `src/dashboards`), so the pins here **run it**, against a fake
|
||||||
|
#: remote reached through a stubbed `ssh`, the way the installer sandbox stubs `install`.
|
||||||
|
CAPTURE = SCRIPTS / "capture-gateway-config.sh"
|
||||||
|
|
||||||
|
#: The remote documents, and the fixture each one becomes. A new include directory on the gateway
|
||||||
|
#: means a new entry here *and* a new `case` in the script — the script refuses silently-unknown
|
||||||
|
#: globs for exactly this reason.
|
||||||
|
CAPTURE_TARGETS = (
|
||||||
|
"/etc/haproxy/haproxy.cfg",
|
||||||
|
"/etc/nginx/nginx.conf",
|
||||||
|
"/etc/nginx/conf.d/*.conf",
|
||||||
|
"/etc/nginx/stream.d/*.conf",
|
||||||
|
"/proc/net/tcp",
|
||||||
|
"/proc/net/tcp6",
|
||||||
|
)
|
||||||
|
|
||||||
|
#: The stub `ssh`. It drops the options the script passes, records every call, and answers `cat`
|
||||||
|
#: and `ls` from `$FAKE_REMOTE` as if that directory were the remote filesystem's `/`. Nothing
|
||||||
|
#: here can reach a network: the point is to exercise the script's own branches (the include
|
||||||
|
#: globs, the refusal rule, the socket-table edit) with no gateway in sight.
|
||||||
|
CAPTURE_SSH_STUB = """#!/bin/sh
|
||||||
|
printf '%s\\n' "$*" >> "$STUB_LOG"
|
||||||
|
while [ $# -gt 0 ]; do
|
||||||
|
case "$1" in
|
||||||
|
-o) shift 2 ;;
|
||||||
|
-n) shift ;;
|
||||||
|
*) break ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
host="$1"; shift
|
||||||
|
command="$*"
|
||||||
|
if [ "${FAIL_SSH:-0}" = "1" ]; then
|
||||||
|
echo "ssh: connect to host $host port 22: Connection refused" >&2
|
||||||
|
exit 255
|
||||||
|
fi
|
||||||
|
case "$command" in
|
||||||
|
"cat "*)
|
||||||
|
path="${command#cat }"
|
||||||
|
file="$FAKE_REMOTE$path"
|
||||||
|
[ -f "$file" ] || { echo "cat: $path: No such file or directory" >&2; exit 1; }
|
||||||
|
cat "$file"
|
||||||
|
;;
|
||||||
|
"ls "*)
|
||||||
|
spec="${command#ls }"
|
||||||
|
dir="${spec%/*}"
|
||||||
|
glob="${spec##*/}"
|
||||||
|
found=0
|
||||||
|
for file in "$FAKE_REMOTE$dir"/$glob; do
|
||||||
|
[ -e "$file" ] || continue
|
||||||
|
found=1
|
||||||
|
printf '%s\\n' "${file#"$FAKE_REMOTE"}"
|
||||||
|
done
|
||||||
|
[ "$found" = 1 ] || { echo "ls: cannot access '$spec': No such file or directory" >&2; exit 2; }
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "stub ssh: unsupported remote command: $command" >&2
|
||||||
|
exit 2
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
"""
|
||||||
|
|
||||||
|
#: A gateway tree small enough to assert on whole, with the two shapes the parsers exist for: a
|
||||||
|
#: `use_backend` whose target is a fetch expression, and a stream `proxy_pass` of a map variable.
|
||||||
|
CAPTURE_FAKE_GATEWAY: dict[str, str] = {
|
||||||
|
"/etc/haproxy/haproxy.cfg": (
|
||||||
|
"global\n maxconn 4096\n"
|
||||||
|
"defaults\n mode tcp\n"
|
||||||
|
"frontend haproxy-22\n bind :::22 v4v6\n default_backend gitea-2222\n"
|
||||||
|
"frontend haproxy-8443\n bind :::8443 v4v6\n"
|
||||||
|
" use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/maps,b0)\n"
|
||||||
|
"backend gitea-2222\n server gitea gitea.reeselink.com:2222 check inter 5s\n"
|
||||||
|
),
|
||||||
|
"/etc/nginx/nginx.conf": (
|
||||||
|
"user nginx;\n"
|
||||||
|
"stream {\n include /etc/nginx/stream.d/*.conf;\n}\n"
|
||||||
|
"http {\n"
|
||||||
|
" include /etc/nginx/mime.types;\n"
|
||||||
|
" server { listen 80; listen [::]:80; server_name _; return 301 https://$host; }\n"
|
||||||
|
" include /etc/nginx/conf.d/*.conf;\n}\n"
|
||||||
|
),
|
||||||
|
"/etc/nginx/conf.d/http.conf": (
|
||||||
|
"server {\n"
|
||||||
|
" listen 4443 ssl proxy_protocol;\n"
|
||||||
|
" server_name nextcloud.reeseapps.com;\n"
|
||||||
|
" ssl_certificate_key /etc/letsencrypt/live/nextcloud.reeseapps.com/privkey.pem;\n"
|
||||||
|
" location / { proxy_pass http://nextcloud.reeselink.com:11000; }\n}\n"
|
||||||
|
),
|
||||||
|
"/etc/nginx/conf.d/tls.conf": (
|
||||||
|
"server {\n listen 8443 ssl;\n server_name brain.reeseapps.com;\n"
|
||||||
|
" location / { proxy_pass http://reeseapps-brain-of-reese.reeselink.com:443; }\n}\n"
|
||||||
|
),
|
||||||
|
"/etc/nginx/stream.d/stream.conf": (
|
||||||
|
"map $ssl_preread_server_name $targetBackend {\n"
|
||||||
|
" default 127.0.0.1:4443;\n"
|
||||||
|
" brain.example reeseapps-brain-of-reese.reeselink.com:443;\n}\n"
|
||||||
|
"server {\n listen 443;\n proxy_pass $targetBackend;\n}\n"
|
||||||
|
),
|
||||||
|
# 8.8.8.8 and 2603:6013::1 are globally routable, `::ffff:8.8.8.8` is a visitor behind a v6
|
||||||
|
# socket, and 203.0.113.7 is already documentation space. The tx/rx queue pair on the last row
|
||||||
|
# is spelled like an address column and is not one.
|
||||||
|
"/proc/net/tcp": (
|
||||||
|
" sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid"
|
||||||
|
" timeout inode\n"
|
||||||
|
" 0: 00000000:1EFF 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0"
|
||||||
|
" 0 1 1 0000000000000000 100 0 0 10 0\n"
|
||||||
|
" 1: 1B01010A:01BB 08080808:911A 01 00000000:00000000 00:00000000 00000000 101"
|
||||||
|
" 0 2 1 00000000d22dedba 20 4 28 10 -1\n"
|
||||||
|
" 2: 0100007F:23A0 017100CB:C0A8 06 08080808:08080808 00:00000000 00000000 0"
|
||||||
|
" 0 3 3 0000000000000000 20 0 0 10 0\n"
|
||||||
|
),
|
||||||
|
"/proc/net/tcp6": (
|
||||||
|
" sl local_address remote_address"
|
||||||
|
" st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode\n"
|
||||||
|
" 0: 00000000000000000000000000000000:01BB 00000000000000000000000000000000:0000 0A"
|
||||||
|
" 00000000:00000000 00:00000000 00000000 0 0 4 1 0000000000000000 100 0 0 10 0\n"
|
||||||
|
" 1: 1360032601010031FF005450E0B284FE:01BB 0000000000000000FFFF000008080808:C0AC 01"
|
||||||
|
" 00000000:00000000 00:00000000 00000000 101 0 5 1 0000000000000000 63 4 26 10 -1\n"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _capture_sandbox(
|
||||||
|
tmp_path: Path,
|
||||||
|
gateway: dict[str, str] | None = None,
|
||||||
|
*,
|
||||||
|
recorded: dict[str, str] | None = None,
|
||||||
|
fail_ssh: bool = False,
|
||||||
|
alias: str | None = None,
|
||||||
|
) -> tuple[Path, Path, Path, dict[str, str]]:
|
||||||
|
"""A copy of the capture script, a fake gateway, and a PATH whose `ssh` reaches only the fake.
|
||||||
|
|
||||||
|
Returns (script, fixtures dir, stub call log, env). The script is copied into
|
||||||
|
`<tmp>/scripts/` because it derives `tests/fixtures/` from its own location, which is what
|
||||||
|
keeps the repository's fixtures out of reach of a test.
|
||||||
|
"""
|
||||||
|
script = tmp_path / "scripts" / CAPTURE.name
|
||||||
|
script.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
shutil.copyfile(CAPTURE, script)
|
||||||
|
|
||||||
|
fake = tmp_path / "gateway"
|
||||||
|
for path, body in {**CAPTURE_FAKE_GATEWAY, **(gateway or {})}.items():
|
||||||
|
target = fake / path.lstrip("/")
|
||||||
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
target.write_text(body, encoding="utf-8")
|
||||||
|
|
||||||
|
fixtures = tmp_path / "tests" / "fixtures"
|
||||||
|
fixtures.mkdir(parents=True, exist_ok=True)
|
||||||
|
for name, body in (recorded or {}).items():
|
||||||
|
(fixtures / name).write_text(body, encoding="utf-8")
|
||||||
|
|
||||||
|
stubs = tmp_path / "stubs"
|
||||||
|
stubs.mkdir()
|
||||||
|
stub = stubs / "ssh"
|
||||||
|
stub.write_text(CAPTURE_SSH_STUB, encoding="utf-8")
|
||||||
|
stub.chmod(0o755)
|
||||||
|
call_log = tmp_path / "ssh-calls.log"
|
||||||
|
call_log.write_text("", encoding="utf-8")
|
||||||
|
|
||||||
|
env = {**os.environ,
|
||||||
|
"PATH": f"{stubs}{os.pathsep}{os.environ['PATH']}",
|
||||||
|
"STUB_LOG": str(call_log),
|
||||||
|
"FAKE_REMOTE": str(fake),
|
||||||
|
"FAIL_SSH": "1" if fail_ssh else "0"}
|
||||||
|
if alias is not None:
|
||||||
|
env["GATEWAY_SSH"] = alias
|
||||||
|
return script, fixtures, call_log, env
|
||||||
|
|
||||||
|
|
||||||
|
def _capture(tmp_path: Path, **kwargs: object) -> subprocess.CompletedProcess[str]:
|
||||||
|
script, _fixtures, _log, env = _capture_sandbox(tmp_path, **kwargs) # type: ignore[arg-type]
|
||||||
|
return subprocess.run([BASH, str(script)], capture_output=True, text=True, env=env, check=False)
|
||||||
|
|
||||||
|
|
||||||
|
def _fixture_body(path: Path) -> str:
|
||||||
|
"""A fixture without its `#` provenance block — the bytes the host answered."""
|
||||||
|
return "".join(line for line in path.read_text().splitlines(keepends=True)
|
||||||
|
if not line.startswith("#"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_gateway_capture_exists_and_asks_for_no_privilege() -> None:
|
||||||
|
"""Six world-readable reads over an ssh key: nothing here needs root or a rewrite tool."""
|
||||||
|
assert CAPTURE.is_file(), "scripts/capture-gateway-config.sh is missing (phase 28 task 04)"
|
||||||
|
assert os.access(CAPTURE, os.X_OK)
|
||||||
|
text = CAPTURE.read_text()
|
||||||
|
for path in CAPTURE_TARGETS:
|
||||||
|
assert path in text, f"{path} is no longer captured"
|
||||||
|
code = "\n".join(line for line in text.splitlines() if not line.lstrip().startswith("#"))
|
||||||
|
assert "sudo" not in code, (
|
||||||
|
"the capture must read exactly what dashboards-agent can read; a pass that needed root "
|
||||||
|
"would record more than the agent can see (the smartctl note in capture-fixtures.sh)"
|
||||||
|
)
|
||||||
|
for tool in ("sed ", "awk ", " tr ", "cut "):
|
||||||
|
assert tool not in text, f"{tool.strip()} would be a text filter between the read and a fixture"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_gateway_capture_dials_the_alias_and_nothing_else(tmp_path: Path) -> None:
|
||||||
|
"""No hardcoded address: every remote command goes to `$GATEWAY_SSH`, default `gateway-ops`."""
|
||||||
|
text = CAPTURE.read_text()
|
||||||
|
assert 'GATEWAY_SSH="${GATEWAY_SSH:-gateway-ops}"' in text
|
||||||
|
# Comments may name an address (`0.0.0.0` is a shape the parser has to survive); code may not.
|
||||||
|
code = "\n".join(line for line in text.splitlines() if not line.lstrip().startswith("#"))
|
||||||
|
stray = [a for a in re.findall(r"\b(?:\d{1,3}\.){3}\d{1,3}\b", code)
|
||||||
|
if not a.startswith("203.0.113.")]
|
||||||
|
assert not stray, f"the script dials or embeds an address: {stray}"
|
||||||
|
|
||||||
|
_script, _fixtures, log, env = _capture_sandbox(tmp_path)
|
||||||
|
subprocess.run([BASH, str(_script)], capture_output=True, text=True, env=env, check=False)
|
||||||
|
assert _dialled_hosts(log) == {"gateway-ops"}
|
||||||
|
|
||||||
|
other = _capture_sandbox(tmp_path / "alias", alias="lab-gw")
|
||||||
|
subprocess.run([BASH, str(other[0])], capture_output=True, text=True, env=other[3], check=False)
|
||||||
|
assert _dialled_hosts(other[2]) == {"lab-gw"}
|
||||||
|
|
||||||
|
|
||||||
|
def _dialled_hosts(log: Path) -> set[str]:
|
||||||
|
"""The host argument of every ssh the stub was called for — options are not hosts."""
|
||||||
|
hosts = set()
|
||||||
|
for line in log.read_text().splitlines():
|
||||||
|
tokens = [token for token in line.split() if not token.startswith("-") and "=" not in token]
|
||||||
|
if tokens:
|
||||||
|
hosts.add(tokens[0])
|
||||||
|
return hosts
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_gateway_capture_records_every_document_of_the_tree_it_read(tmp_path: Path) -> None:
|
||||||
|
"""The happy path, run: one fixture per remote document, include globs and all.
|
||||||
|
|
||||||
|
Two files in `conf.d` is deliberate. The script reads the include globs out of the captured
|
||||||
|
main config rather than a hardcoded pair of directories, so a gateway that adds a vhost file
|
||||||
|
gets it captured, and `files` in the payload stays the tree the config actually merges.
|
||||||
|
"""
|
||||||
|
result = _capture(tmp_path)
|
||||||
|
assert result.returncode == 0, result.stdout + result.stderr
|
||||||
|
fixtures = tmp_path / "tests" / "fixtures"
|
||||||
|
expected = {"gateway_haproxy.cfg", "gateway_nginx_main.conf",
|
||||||
|
"gateway_nginx_conf_d_http.conf", "gateway_nginx_conf_d_tls.conf",
|
||||||
|
"gateway_nginx_stream_d_stream.conf",
|
||||||
|
"gateway_proc_net_tcp.txt", "gateway_proc_net_tcp6.txt"}
|
||||||
|
assert {p.name for p in fixtures.iterdir()} == expected, result.stdout
|
||||||
|
assert _fixture_body(fixtures / "gateway_haproxy.cfg") == \
|
||||||
|
CAPTURE_FAKE_GATEWAY["/etc/haproxy/haproxy.cfg"]
|
||||||
|
assert _fixture_body(fixtures / "gateway_nginx_conf_d_tls.conf") == \
|
||||||
|
CAPTURE_FAKE_GATEWAY["/etc/nginx/conf.d/tls.conf"]
|
||||||
|
# `mime.types` is included by the config and absent on the host: no fixture, no complaint —
|
||||||
|
# the same absence-the-agent-reports rule, one level up.
|
||||||
|
assert not (fixtures / "gateway_nginx_mime.types").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_gateway_capture_writes_the_domains_it_read_verbatim(tmp_path: Path) -> None:
|
||||||
|
"""Nothing is scrubbed from a config, and the domains are the proof.
|
||||||
|
|
||||||
|
The `server_name` and the `proxy_pass` target beside it are the ingress map, and the
|
||||||
|
`ssl_certificate_key` is a path rather than a key — so the file the fixture holds is the file
|
||||||
|
the gateway runs, byte for byte. The one exception in this script is the socket tables'
|
||||||
|
globally routable addresses, which `capture-fixtures.sh`'s identifier rule requires.
|
||||||
|
"""
|
||||||
|
_capture(tmp_path)
|
||||||
|
vhosts = (tmp_path / "tests" / "fixtures" / "gateway_nginx_conf_d_http.conf").read_text()
|
||||||
|
assert "server_name nextcloud.reeseapps.com;" in vhosts
|
||||||
|
assert "proxy_pass http://nextcloud.reeselink.com:11000;" in vhosts
|
||||||
|
assert "ssl_certificate_key /etc/letsencrypt/live/nextcloud.reeseapps.com/privkey.pem;" in vhosts
|
||||||
|
assert "SCRUB" not in vhosts and "REDACTED" not in vhosts
|
||||||
|
assert next((line for line in vhosts.splitlines()
|
||||||
|
if line.startswith("# Verbatim /etc/nginx/conf.d/http.conf from gateway-ops")),
|
||||||
|
None), "every fixture says which remote document it is"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_dead_gateway_leaves_the_recorded_fixtures_alone(tmp_path: Path) -> None:
|
||||||
|
"""The `slots_gufo.json` rule: a capture script must never turn six good fixtures into six
|
||||||
|
empty files, because then the parser tests pass against nothing (or fail for the wrong
|
||||||
|
reason) and the ground truth is gone."""
|
||||||
|
keep = {"gateway_haproxy.cfg": "# old\nfrontend kept\n bind :::22 v4v6\n",
|
||||||
|
"gateway_proc_net_tcp.txt": " 0: 00000000:1EFF 00000000:0000 0A x\n"}
|
||||||
|
result = _capture(tmp_path, recorded=keep, fail_ssh=True)
|
||||||
|
assert result.returncode == 1, result.stdout
|
||||||
|
for name, body in keep.items():
|
||||||
|
assert (tmp_path / "tests" / "fixtures" / name).read_text() == body, \
|
||||||
|
f"{name} was overwritten by a read that failed"
|
||||||
|
for name in keep:
|
||||||
|
assert f"✗ {name}" in result.stdout, result.stdout
|
||||||
|
assert "Connection refused" in result.stdout, "one line names the reason, per fixture"
|
||||||
|
assert sorted(p.name for p in (tmp_path / "tests" / "fixtures").iterdir()) == sorted(keep), \
|
||||||
|
"a failed capture leaves no temp files behind"
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_empty_read_is_refused_the_way_a_failed_one_is(tmp_path: Path) -> None:
|
||||||
|
"""`cat` exiting 0 on an empty file is still not a capture."""
|
||||||
|
keep = {"gateway_proc_net_tcp.txt": " 0: 00000000:1EFF 00000000:0000 0A keep me\n"}
|
||||||
|
result = _capture(tmp_path, gateway={"/proc/net/tcp": ""}, recorded=keep)
|
||||||
|
assert result.returncode == 1, result.stdout
|
||||||
|
assert "gateway_proc_net_tcp.txt" in result.stdout
|
||||||
|
assert "empty" in result.stdout
|
||||||
|
assert (tmp_path / "tests" / "fixtures" / "gateway_proc_net_tcp.txt").read_text() == \
|
||||||
|
next(iter(keep.values()))
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_include_dir_with_no_fixture_name_is_reported_and_fails(tmp_path: Path) -> None:
|
||||||
|
"""A config that moved its vhosts is a loud failure, not a fixture set that quietly stopped
|
||||||
|
being the tree the agent merges."""
|
||||||
|
moved = {"/etc/nginx/nginx.conf": (
|
||||||
|
"http {\n include /etc/nginx/sites-enabled/*.conf;\n}\n"
|
||||||
|
"stream {\n include /etc/nginx/stream.d/*.conf;\n}\n")}
|
||||||
|
result = _capture(tmp_path, gateway=moved)
|
||||||
|
assert result.returncode == 1, result.stdout
|
||||||
|
assert "/etc/nginx/sites-enabled/*.conf" in result.stdout
|
||||||
|
assert not (tmp_path / "tests" / "fixtures" / "gateway_nginx_sites_enabled_web.conf").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_config_with_no_glob_include_says_the_vhost_fixtures_were_not_refreshed(
|
||||||
|
tmp_path: Path) -> None:
|
||||||
|
keep = {"gateway_nginx_conf_d_http.conf": "# keep\nserver {\n listen 4443;\n}\n"}
|
||||||
|
result = _capture(tmp_path, gateway={"/etc/nginx/nginx.conf": "user nginx;\nevents { }\n"},
|
||||||
|
recorded=keep)
|
||||||
|
assert result.returncode == 1, result.stdout
|
||||||
|
assert "no glob include" in result.stdout
|
||||||
|
assert (tmp_path / "tests" / "fixtures" / "gateway_nginx_conf_d_http.conf").read_text() == \
|
||||||
|
next(iter(keep.values()))
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_socket_capture_publishes_no_globally_routable_endpoint(tmp_path: Path) -> None:
|
||||||
|
"""The one edit the capture makes, run rather than asserted in prose.
|
||||||
|
|
||||||
|
Two rules at once, which is what makes the socket fixtures both usable and publishable: a
|
||||||
|
globally routable address (including one behind `::ffff:`) becomes documentation space, and
|
||||||
|
everything else — every port, every private/loopback/wildcard word, every other column, and
|
||||||
|
the column widths themselves — is byte-for-byte what the kernel answered. A tx/rx queue pair
|
||||||
|
is spelled exactly like an address column and must survive untouched, which is why the edit
|
||||||
|
reads two columns and not the line.
|
||||||
|
"""
|
||||||
|
_capture(tmp_path)
|
||||||
|
fixtures = tmp_path / "tests" / "fixtures"
|
||||||
|
v4 = _fixture_body(fixtures / "gateway_proc_net_tcp.txt").splitlines()
|
||||||
|
v6 = _fixture_body(fixtures / "gateway_proc_net_tcp6.txt").splitlines()
|
||||||
|
raw_v4 = CAPTURE_FAKE_GATEWAY["/proc/net/tcp"].splitlines()
|
||||||
|
raw_v6 = CAPTURE_FAKE_GATEWAY["/proc/net/tcp6"].splitlines()
|
||||||
|
assert len(v4) == len(raw_v4) and len(v6) == len(raw_v6), "a row was added or lost"
|
||||||
|
|
||||||
|
assert v4[1] == raw_v4[1], "a wildcard listener must survive untouched"
|
||||||
|
assert "08080808:911A" not in "".join(v4), "8.8.8.8 is still in the fixture"
|
||||||
|
assert "017100CB:911A" in v4[2], v4[2] # 203.0.113.1, the port kept
|
||||||
|
assert len(v4[2]) == len(raw_v4[2]), "the replacement must be exactly as wide as the original"
|
||||||
|
assert v4[3] == raw_v4[3], (
|
||||||
|
"an already-documentation address and the tx/rx queue pair on that row both moved: only "
|
||||||
|
"the two address columns may be looked at")
|
||||||
|
assert "08080808:08080808" in v4[3], v4[3]
|
||||||
|
|
||||||
|
assert "1360032601010031FF005450E0B284FE" not in "".join(v6), "a global v6 is left in"
|
||||||
|
assert "B80D0120000000000000000001000000:01BB" in v6[2], v6[2] # 2001:db8::1, port kept
|
||||||
|
assert "FFFF0000017100CB:C0AC" in v6[2] or "FFFF0000" in v6[2], \
|
||||||
|
"a v4-mapped row must stay v4-mapped, or the parser's branch for it stops being exercised"
|
||||||
|
assert v6[1] == raw_v6[1], "the wildcard v6 listener moved"
|
||||||
|
assert len(v6[2]) == len(raw_v6[2])
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_socket_capture_is_idempotent_so_a_re_run_does_not_churn(tmp_path: Path) -> None:
|
||||||
|
"""Re-recording the same moment must not diff, or every capture moves the fixture twice.
|
||||||
|
|
||||||
|
Documentation space is not globally routable, so the second pass leaves it alone — the same
|
||||||
|
property that makes the drive-serial scrub re-runnable.
|
||||||
|
"""
|
||||||
|
once = tmp_path / "once"
|
||||||
|
_capture(once)
|
||||||
|
again = _capture_sandbox(tmp_path / "twice",
|
||||||
|
gateway={"/proc/net/tcp": _fixture_body(
|
||||||
|
once / "tests" / "fixtures" / "gateway_proc_net_tcp.txt")})
|
||||||
|
result = subprocess.run([BASH, str(again[0])], capture_output=True, text=True,
|
||||||
|
env=again[3], check=False)
|
||||||
|
assert result.returncode == 0, result.stdout + result.stderr
|
||||||
|
assert _fixture_body(again[1] / "gateway_proc_net_tcp.txt") == _fixture_body(
|
||||||
|
once / "tests" / "fixtures" / "gateway_proc_net_tcp.txt")
|
||||||
|
|||||||
Reference in New Issue
Block a user