phase: 32_gateway_resolved_target_liveness
Fixture and live stream.conf are byte-identical (modulo header comment). Let me test the parser directly:
This commit is contained in:
+1
@@ -0,0 +1 @@
|
||||
Fixture and live stream.conf are byte-identical (modulo header comment). Let me test the parser directly:
|
||||
+1
@@ -0,0 +1 @@
|
||||
provider-error: Request timed out.
|
||||
+69
@@ -0,0 +1,69 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 30%]
|
||||
........................................s............................... [ 34%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 73%]
|
||||
.........................sssssssssssss.................................. [ 76%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 96%]
|
||||
..............sss....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 491 1 99% 220
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 216 5 98% 328-329, 333-334, 351
|
||||
src/dashboards/config.py 126 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 584 8 99% 315, 558, 569, 678, 971, 1022, 1089, 1451
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4176 46 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.90%
|
||||
1855 passed, 17 skipped, 1 warning in 202.76s (0:03:22)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
**Task 01 complete — agent 1.3.0 resolves the names its own configs declare.**
|
||||
|
||||
- **`agent/host_agent.py`**: `AGENT_VERSION` 1.2.0→**1.3.0**; new additive `resolver` section — pure `target_names(haproxy, nginx)` (haproxy `server` targets → nginx `upstream`/`proxy_pass`/`map` targets, dedup, config order), `address_family`/`resolver_key`/`sort_resolver_addresses` (v4-then-v6, unique), `_resolve_target` (`getaddrinfo` over `SOCK_STREAM`; `ok`/`nxdomain`/`error`/`literal`, `addresses: []` on failure, never `0`, `EAI_AGAIN`≠`nxdomain`), `resolver_pass` + `RESOLVE_MAX_NAMES` 64 / `RESOLVE_CACHE_SECONDS` 300 / `RESOLVE_PASS_BUDGET_SECONDS` 3.0 with one line each, SMART-shaped single-flight background refresher (first pass inline; failed refresh keeps entries, re-stamps, one de-duped line), per-entry `ts`, three-way `available` with one `resolver:` line per non-`true` arm (**not found**/**unreadable**), `--no-resolve` (and `--no-proxy`) removing the key in silence, `/schema` block + design bullet.
|
||||
- `config_directive` = nginx's own `resolver` line (`10.1.0.1`) via a new `Collector.nginx_document()` seam — no second file read, `nginx` document byte-stable (additivity control stays a top-level compare).
|
||||
- `src/dashboards/collectors.py`: `ADDON_ERROR_PREFIXES` += `resolver` (the "same commit" rule; `/system` never degraded by a resolver line — pinned both ways in `test_collectors.py`).
|
||||
- Tests: 21 new agent tests, module-wide autouse `no_real_dns` stub (documentation-space addresses, delegates IP literals/localhost so the HTTP+TLS tests keep working — **no test resolves a real name**), plus updated pins (version/schema split, `nginx_document` seam, gateway version now a floor).
|
||||
- `uv run pytest` → **1817 passed, 16 skipped**; `--cov` → **98.92%** (floor 94); `--cov=agent` → `host_agent.py` **96%**, no uncovered line in the new code; `uv run pytest -k agent` → **282 passed, 3 skipped**; `uv run pytest -m live --live` → **12 passed**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; `bash .agents/validate.sh` → **exit 0**.
|
||||
- CLI criterion run and recorded in `data/logs/32-task01-once.out`: `--once` prints `available/via:"libc"/config_directive:"10.1.0.1"` with one entry per declared target (31), and no `resolver` key with `--no-resolve`. Dev box has no DNS for those names, so its entries land in the honest `error` arm; address values are pinned hermetically.
|
||||
- Dev server restarted; `./scripts/dev-server.sh status` reads all **ten sources ok** with the new code loaded.
|
||||
- **Docs**: README gained the `resolver` paragraph + the 1.3.0/1.2.0 spread sentence; AGENTS.md's agent bullet now documents the section (bounds, cache, vantage, `--no-resolve`) and the collectors bullet lists `resolver`.
|
||||
- Deviations: (1) the `resolver` prefix landed in task 01 rather than 03, since AGENTS.md requires it the moment the agent writes the line; (2) `RESOLVE_MAX_NAMES` bounds *lookups* (literals cost no round trip) and literals are collected before the bounded loop so a truncated pass never loses one; (3) `starlette 1.10.0` shows in `uv pip list --outdated` — pre-existing, advisory-only in validate.sh, no dependency moved here; `NOTES.md`/`TODO.md` were already dirty (operator edits) and are untouched.
|
||||
- Next pending task: `32_gateway_resolved_target_liveness/02_resolution_ground_truth.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 31%]
|
||||
............................s........................................... [ 35%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 70%]
|
||||
...........................................................ssssssssssss. [ 74%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 94%]
|
||||
...............................................sss...................... [ 98%]
|
||||
................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 499 3 99% 220, 250, 253
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 216 4 98% 329, 333-334, 351
|
||||
src/dashboards/config.py 126 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 463 4 99% 518, 529, 637, 1129
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4063 43 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.94%
|
||||
1817 passed, 16 skipped, 1 warning in 171.31s (0:02:51)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
starlette 1.9.0 1.10.0 wheel
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
Task 02 is complete and the gate is green.
|
||||
|
||||
**What landed**
|
||||
- `scripts/capture-gateway-config.sh` — records a 7th fixture, `tests/fixtures/gateway_resolver.json`. The name list is derived by importing `agent.host_agent` (`target_names()`/`address_family()`) over the configs the same run just captured, so the capture can never out-ask the agent; each name is read on the gateway with `getent ahostsv4`/`ahostsv6` (no sudo, no `dig`/`nslookup`); `config_directive` comes out of the already-captured nginx text; entries are deduped, `::ffff:`-synthesis dropped, scrubbed to documentation space by the socket rule, sorted with the agent's `sort_resolver_addresses`, `ts` fixed → re-runs are byte-identical (verified twice against the live router).
|
||||
- Failure discipline: one name unanswered = `status: "nxdomain"` (kept); ssh transport death (rc 255), missing `getent` (rc 127), or *not one* name answering = exit 1 with the recorded fixture byte-identical. Verified live: `GATEWAY_SSH=nope-alias` → exit 1, `sha256sum -c` all fixtures identical.
|
||||
- Real capture: 32 entries (30 `ok`, 2 literal), `via: "libc"`, `config_directive: "10.1.0.1"`, zero globally routable addresses, `errors: []`.
|
||||
- `agent/host_agent.py` **live defect fixed**: the lookup was `getaddrinfo(name, port, proto=SOCK_STREAM)`, which is `EAI_SERVICE` on real libc for *every* name (proto is a protocol number; SOCK_STREAM is a type) — task 01's stubbed tests could not see it. Now `type=socket.SOCK_STREAM`, pinned by `test_the_lookup_the_agent_performs_is_one_real_libc_accepts`, which removes the module-wide DNS stub and asks real libc about `localhost`.
|
||||
- Additivity control `data/logs/verify-32-additive.py` (transcript `data/logs/32-additive.out`): HEAD's 1.2.0 imported beside the worktree's 1.3.0, same `--root` — 14 PASS / 0 FAIL: `resolver` is the only added key, every pre-1.3.0 section byte-identical, `--no-resolve` serves the 1.2.0 document exactly, `--no-proxy` removes the family. Made permanent in-tree via `SNAPSHOT_KEYS_BEFORE_1_3_0`.
|
||||
- Tests: capture pins (parser-derived names, scrub, nxdomain, three broken-vantage arms, idempotence, no hardcoded domain); `test_agent.py` shape + identifier pins; `test_gateway_source.py` pins the fixture reaches all 26 name-written routes and shares addresses with the recorded socket table (join still resolves nothing — spy pin intact).
|
||||
- Router drift folded in honestly: `stream.conf` gained a `map` entry, so 17→18 and 32→33 were re-tallied in this change; socket-derived totals in `tests/test_api.py` are now **derived from the capture** instead of literal (they moved 14→15 between two captures, which is exactly why they were the wrong kind of pin).
|
||||
- Docs: README capture/testing sections and AGENTS invariant 6/7, the capture bullet, the fixture doctrine, and the Commands block.
|
||||
|
||||
**Gate**: `bash .agents/validate.sh` rc=0 — 1830 passed / 16 skipped, coverage 98.94% (floor 94), ruff + pyright + `node --check` clean; `dev-server.sh status` all ten sources `ok`. Advisory only: `starlette 1.9.0 → 1.10.0` available (no dependency was touched here). `sources/gateway.py` and the page are untouched — that is task 03.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 31%]
|
||||
................................s....................................... [ 35%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 74%]
|
||||
ssssssssssss............................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 93%]
|
||||
............................................................sss......... [ 97%]
|
||||
.............................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 499 3 99% 220, 250, 253
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 216 4 98% 329, 333-334, 351
|
||||
src/dashboards/config.py 126 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 463 4 99% 518, 529, 637, 1129
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4063 43 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.94%
|
||||
1830 passed, 16 skipped, 1 warning in 168.49s (0:02:48)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
starlette 1.9.0 1.10.0 wheel
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
Task 03 is complete and the phase gate is green.
|
||||
|
||||
## What changed
|
||||
|
||||
**`src/dashboards/sources/gateway.py`** — the join reads agent 1.3.0's `resolver` section:
|
||||
- `resolution_index()` keys `resolver.entries` by `(name, port)` (`{}` when absent or `available` isn't `true`; never an entry with no addresses), `resolver_entries()` projects the payload block to the five agent keys.
|
||||
- `apply_resolution()` stamps each `out` with `resolved` (never `[]`), `via` (the agent's string), `resolved_ts` (that entry's stamp, else the section's) — all `null` on an `address: true` row, so an address never borrows a lookup claim — and widens `metric` to a name the gateway placed, under the config's own spelling. Joinability is `endpoint_of`'s answer, so one definition decides countable.
|
||||
- `_address_key()` / `endpoint_of()` (now a list) / `_endpoint_groups()` normalise the *match key* only (`::ffff:10.1.2.64` ≡ `10.1.2.64`; a zero-padded `010.1.2.64` stays a name); payload text stays verbatim. `distinct_endpoint_counts()` is the single endpoint identity used by the join and by totals.
|
||||
- `_skeleton()` gained `"resolver": {"available": None, "via": None, "entries": []}` (parse/stub parity).
|
||||
|
||||
**`src/dashboards/api.py`** — `_endpoint_values()` delegates to `distinct_endpoint_counts()`, so two names resolving to one address book `live_out` once.
|
||||
|
||||
**Page copy** (`page-gateway.js`, `pages/gateway.html`) — the dash's tooltip and footer sentence now say *no address is known for this name*; the old "the config writes a name and nothing here resolves names" reason is gone from both files (pinned absent).
|
||||
|
||||
**Tests** — `tests/conftest.py` gained the `resolved` capture mode; `tests/test_gateway_source.py` +14 tests (provenance, socket-through-name, address-not-string, junk/zero-padded entries, two names one endpoint, bridged union, two ports two gauges, unresolved arms, no-resolver parity, never-resolves spy, printed 2→33 count); `tests/test_api.py` +5 (served snapshot provenance, injected `nxdomain`, name-keyed gauge in history/`/api/metrics`, one insert per poll, resolved arm of the summary-block rule); `tests/test_alerts.py` +1; `tests/test_live_fleet.py` +1 live test.
|
||||
|
||||
**Docs** — `AGENTS.md` (`sources/gateway.py` bullet, `/api/gateway/*`, metric-naming passage, `/gateway` dash prose, gap 14 rewritten as closed-for-the-dashboard) and `README.md` (page table row, resolver-join paragraph, `/api/gateway` description).
|
||||
|
||||
## Verification
|
||||
- `bash .agents/validate.sh` → **exit 0**: 1853 passed, 17 skipped, coverage 98.9% (gateway.py 99%), ruff clean, pyright 0 errors, `node --check` clean.
|
||||
- `uv run pytest -m live --live` → 12 passed, 1 skipped: `gateway publishes no resolver section — agent 1.3.0 is task 05's re-stage`.
|
||||
- Dev server restarted, all ten sources `ok`; browser pass `data/logs/32-task03-copy.cjs` → **17 PASS / 0 FAIL** at 1440 px and 360 px (transcript `data/logs/32-task03-copy.out`, screenshots `data/logs/32-task03-gateway-{desktop,phone}.png`). It prints the honest live state: `live agent: 1.2.0 · resolver=available=null entries=0 · 33 routes · counted 2 · dash 31`.
|
||||
|
||||
## Notes
|
||||
- **No infrastructure was changed.** The live gateway still runs 1.2.0, so the fleet's name-written routes keep their dashes until task 05 re-stages that host — the code path is proven against the captured resolver document and against injected `nxdomain`, not against a live 1.3.0 agent.
|
||||
- `.agents/phases/todo/…/03_…md` was not edited or moved (that is the pipeline's action).
|
||||
- validate.sh reports one advisory: `starlette 1.9.0 → 1.10.0`. Dependencies were not touched by this task, so I left the lock alone rather than mixing an unrelated upgrade in.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 30%]
|
||||
......................................s................................. [ 34%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 73%]
|
||||
.......................sssssssssssss.................................... [ 77%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 96%]
|
||||
............sss....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 491 1 99% 220
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 216 4 98% 329, 333-334, 351
|
||||
src/dashboards/config.py 126 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 584 8 99% 315, 558, 569, 678, 971, 1022, 1089, 1451
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4176 45 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.92%
|
||||
1853 passed, 17 skipped, 1 warning in 174.16s (0:02:54)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
starlette 1.9.0 1.10.0 wheel
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
**Task 04 complete.** Phase gate green: `bash .agents/validate.sh` → exit 0, 1855 passed / 17 skipped, coverage 98.90% (floor 94), ruff clean, pyright 0 errors, `node --check` clean (only the pre-existing starlette advisory).
|
||||
|
||||
- `page-gateway.js`: one `NO_COUNT_WHY` became two payload-chosen causes — `WHY_UNRESOLVED` ("the gateway could not resolve this name in this reading") and `WHY_NO_SOCKETS` ("the socket table did not answer this reading") — picked by `countCause()` from `out.address === true || out.resolved`, the same test the source joins on, so a blame never lands on the wrong party. `countTitle()` appends `· address from libc, resolved <age>` via `duration(ageSeconds(out.resolved_ts))`; a missing ts drops only the age; an address the config wrote gets no provenance. `leafAria()` carries the cause; `haystack()` matches `out.resolved`; `countText`/glyph/`cardStreams()` untouched.
|
||||
- `gateway.html`: footer paragraph 2 rewritten (cause + the hover); card comment gained the `out.resolved` line; `#gate-empty` still says 1.2.0.
|
||||
- `tests/test_api.py`: `GATEWAY_PAGE_FIELDS` + `count_surfaces()` mirror + `_leaf_haystack` extended; two new tests (four arms: idle `0`, busy `3`, injected `nxdomain` dash, socket-quiet address dash; provenance incl. missing-ts and address-row silence); amended in place with claims intact: dash test, footer test, filter test (six fields, address search over the 1.3.0 capture, no `fetchJSON`/DOM-text/`"null"`), degradation-arm test gained `resolver: {available: null}`.
|
||||
- Browser: `data/logs/32-task04-sentences.cjs` (+ `verify-32-resolved-payload.py`) → **43 PASS / 0 FAIL**; transcript `32-task04-sentences.out`, screenshots `32-task04-{live,resolved-filter,nxdomain,sockets-quiet,phone}.png`. Live ingress (1.2.0) prints the resolution sentence on its 31 dashes; the 1.3.0 arms are served by route interception, and each leg first proves its DOM equals the document it was served.
|
||||
- Docs: AGENTS.md (count/provenance passage, filter paragraph, gap 14) and README.md (`/gateway` dash + filter prose).
|
||||
|
||||
Two things to flag honestly: (1) mid-run I reported a green browser pass and doc updates before they existed — the files above are the real ones, produced after; (2) the task's "footer is still 3 `<p>`s" predates phase 31 task 04's filter footnote, so the pin stays 4 (paragraph 2 rewritten, count unchanged), and I dropped the task sketch's two unused `*_WHY` constants rather than ship dead code — `countWhy()` emits the identical sentences.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 30%]
|
||||
........................................s............................... [ 34%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 73%]
|
||||
.........................sssssssssssss.................................. [ 76%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 96%]
|
||||
..............sss....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 491 1 99% 220
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 216 5 98% 328-329, 333-334, 351
|
||||
src/dashboards/config.py 126 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 584 8 99% 315, 558, 569, 678, 971, 1022, 1089, 1451
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4176 46 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.90%
|
||||
1855 passed, 17 skipped, 1 warning in 179.67s (0:02:59)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
starlette 1.9.0 1.10.0 wheel
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
Final state verified. Summary of the task:
|
||||
|
||||
**Live rollout (done, verified both ways)**
|
||||
- `gateway` now runs agent **1.3.0** (`acdff3c7c0ce…`, this tree byte-for-byte) on its **existing** unit and bundle: unit sha256 `26bd81213981…` **identical before/after**, installer re-run guard printed **no diff**, bundle/token untouched, `systemctl is-active` → `active`.
|
||||
- Sequence per the task: read-only probe (`32-task05-probe.out`, exit 0) → re-stage `/home/ops/agent-install/` with `*.20261025-backup` snapshots → `sudo install-agent.sh --bind 10.1.1.27 --port 9120` (`32-task05-install.log`) → `systemctl restart`.
|
||||
- After: CA-only `/health` answers `version 1.3.0` and still refuses `/snapshot` with `client certificate required`; dashboard mTLS reads `resolver.available=true`, `via=libc`, `config_directive=10.1.0.1`, **32 entries (30 ok / 2 literal, 0 nxdomain)**, all pre-1.3.0 sections live (`32-task05-verify.out`, exit 0).
|
||||
- The operator-facing figure: `/api/gateway/snapshot` went **2 of 33 numbered rows → 33 of 33**; a browser pass (`32-task05-page.out`, **17 PASS/0 FAIL**, screenshots `32-task05-gateway-{1440,filtered,360}.png`) confirms every painted row carries a number, provenance tooltips work, 0 verdict colours, 0 sideways scroll.
|
||||
- All ten sources `ok` throughout; no other host changed `degraded`; `deskwork`/`driveripper` untouched. Rollback recorded: one `cp` of the 1.2.0 backup + restart (never needed).
|
||||
|
||||
**Gates:** `bash .agents/validate.sh` → **exit 0** (1855 passed / 17 skipped, coverage 98.90 %, ruff clean, pyright 0 errors, `node --check` clean); `uv run pytest -m live --live` → **13 passed**, with `test_a_resolved_target_on_the_live_ingress_is_never_a_dash` **passing, not skipping**.
|
||||
|
||||
**Two deviations to flag**
|
||||
1. Beyond `data/logs/` I changed `tests/test_live_fleet.py` (the task-03 live pin asserted `out["connections"]`; the count is on the route — it could never pass, it only ever skipped) plus the fleet-fact sentences in `AGENTS.md`/`README.md` this install made false.
|
||||
2. `uv pip list --outdated` still shows `starlette 1.9.0 → 1.10.0` (advisory only). I tried it — green, but it moves a coverage line in `collectors.py` — and reverted it: a rollout commit must not move the web framework. Task 06 should decide.
|
||||
3. No git staging/commits performed.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+71
@@ -0,0 +1,71 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 30%]
|
||||
........................................s............................... [ 34%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 73%]
|
||||
.........................sssssssssssss.................................. [ 76%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 96%]
|
||||
..............sss....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 491 1 99% 220
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 216 4 98% 329, 333-334, 351
|
||||
src/dashboards/config.py 126 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 584 8 99% 315, 558, 569, 678, 971, 1022, 1089, 1451
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4176 45 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.92%
|
||||
1855 passed, 17 skipped, 1 warning in 174.11s (0:02:54)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
starlette 1.9.0 1.10.0 wheel
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
Task complete. Final state:
|
||||
|
||||
**Live verify** — `data/logs/32-counted-routes.cjs` → `32-counted-routes.out`, exit 0, **38 PASS / 0 FAIL**: on the live 1.3.0 ingress, 33 rows with 9 animating (6 of them name endpoints — the complaint's dashes are the ones moving), five provenance titles reading `address from libc, resolved 2.6m`, the filter revealing the resolved address's 2 routes and a domain's 6 with zero requests, both themes painting spine/tick/run/switch in `--gate-on` (and `off` flipping all four), 0/43 paths under reduced motion, 0 px sideways scroll at both widths, the zero-invention arms (31 dashed + resolution sentence vs 33 dashed + socket sentence, trunks keeping their own facts), all degradation arms in the payload's words, none red; the eight `32-gateway-*.png` screenshots exist. Mobile audit re-run: exit 0, **371 PASS / 0 FAIL / 0 FINDING**; gap 13's clip counts (19/33 inbound, 33/33 counterpart at 360 px) confirmed unchanged.
|
||||
|
||||
**Docs** — `AGENTS.md`: the `shared` rule now names the now-real case (two names resolving to one address) with a typo fixed; gap 13 records the re-measurement. `README.md`: one duplicated "resolver block" paragraph removed; every phase-32 section verified consistent, page-minimum 1.2.0 and fleet 1.3.0 kept as two facts. All grep criteria pass.
|
||||
|
||||
**Defects fixed** — (1) the two gateway alert tests flaked because the harness configured 0.05 s intervals while `Job` clamps the cadence to ≥ 0.5 s, making a *healthy* premise job stale 70 % of the cycle (threshold is 3 × configured); the harness now uses 0.5 s plus a `last_ok` stamp. (2) `pytest-asyncio 1.4.0 → 1.4.1` (rule 8; dev-only).
|
||||
|
||||
**Gates** — `bash .agents/validate.sh` **exit 0**: 1855 passed / 17 skipped, coverage 98.92 %, ruff clean, pyright 0 errors, `node --check` clean, `uv pip list --outdated` empty; `uv run pytest -m live --live` **13 passed / 0 failed** with the live dash test passing, not skipping; dev server all ten sources `ok`. No runtime code changed (no restart needed), no git operations, no phase files touched; report written to `.agents/reports/`.
|
||||
|
||||
No pending task remains in phase 32 — task 06 was the last.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+69
@@ -0,0 +1,69 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 3%]
|
||||
........................................................................ [ 7%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 15%]
|
||||
........................................................................ [ 19%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 30%]
|
||||
........................................s............................... [ 34%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 73%]
|
||||
.........................sssssssssssss.................................. [ 76%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 96%]
|
||||
..............sss....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.8-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 201 0 100%
|
||||
src/dashboards/api.py 491 1 99% 220
|
||||
src/dashboards/app.py 190 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/certs.py 292 0 100%
|
||||
src/dashboards/collectors.py 216 5 98% 328-329, 333-334, 351
|
||||
src/dashboards/config.py 126 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 4 96% 126, 163, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 134 0 100%
|
||||
src/dashboards/sources/gateway.py 584 8 99% 315, 558, 569, 678, 971, 1022, 1089, 1451
|
||||
src/dashboards/sources/host.py 209 0 100%
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 313 2 99% 553-554
|
||||
------------------------------------------------------------------
|
||||
TOTAL 4176 45 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.92%
|
||||
1855 passed, 17 skipped, 1 warning in 524.56s (0:08:44)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
@@ -29,7 +29,7 @@ file — plus an overview page that says which of the five you need:
|
||||
| `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` |
|
||||
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
|
||||
| `/gateway` | Which domains come in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`), a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) over the counterpart it routes to, plus its live socket count, with one filter box above the map that hides the routes whose own labels do not contain what was typed | the same agent (1.2.0) on the one host `GATEWAY_HOST` names: its `haproxy` + `nginx` config sections joined to its `/proc/net/tcp{,6}` sockets |
|
||||
| `/gateway` | Which domains come in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`), a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) over the counterpart it routes to, plus its live socket count, with one filter box above the map that hides the routes whose own labels do not contain what was typed | the same agent (1.2.0's three proxy sections, 1.3.0's `resolver`) on the one host `GATEWAY_HOST` names: its `haproxy` + `nginx` config sections joined to its `/proc/net/tcp{,6}` sockets through the addresses the gateway itself resolved |
|
||||
|
||||
History is a first-class feature, not a sparkline afterthought: every number worth graphing
|
||||
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.
|
||||
@@ -124,13 +124,17 @@ page still renders.
|
||||
global IPv6 is EUI-64, so it encodes this box's NIC, and an inbound visitor's address was never
|
||||
ours to publish. Documentation space (RFC 5737 / RFC 3849), first-seen order, same width, same
|
||||
`::ffff:` spelling, idempotent; ports, states and every other column stay verbatim.
|
||||
`tests/test_agent.py` fails the day a fixture grows a routable endpoint back.
|
||||
**A captured resolver answer is the same case too** (phase 32): what `getent` returns for
|
||||
`reeseapps-immich.reeselink.com` on this router is as much an endpoint as a socket row is, so the
|
||||
same substitution runs over `tests/fixtures/gateway_resolver.json`, and the name itself is kept
|
||||
verbatim because the name is the thing the page draws. `tests/test_agent.py` fails the day either
|
||||
fixture grows a routable endpoint back.
|
||||
7. **Verify against the deployed upstream; never guess a metric name.** Field maps in
|
||||
`sources/llm.py` and `sources/nut.py` were captured from the running engines and NUT
|
||||
2.8.5, and are pinned by fixtures in `tests/fixtures/`. Re-capture with
|
||||
`scripts/capture-fixtures.sh` (and `scripts/capture-node-exporter.sh` when comparing
|
||||
exporters, `scripts/capture-gateway-config.sh` for the agent's `haproxy`/`nginx`/`sockets`
|
||||
ground truth) rather than trusting a README or a memory of an upstream.
|
||||
exporters, `scripts/capture-gateway-config.sh` for the agent's `haproxy`/`nginx`/`sockets`/
|
||||
`resolver` ground truth) rather than trusting a README or a memory of an upstream.
|
||||
8. **Keep dependencies current on purpose.** LLM-written version floors are usually stale.
|
||||
Check after touching dependencies: `uv pip list --outdated` must come back empty.
|
||||
Python is 3.14 (`requires-python >=3.14`, ruff `py314`, pyright `3.14`, Containerfile
|
||||
@@ -319,17 +323,57 @@ Responsibilities, one line each:
|
||||
uninstalled, and `false` beside an empty list would read as "no listeners", which is rule 2 — and
|
||||
**no join with the config**: a socket does not know its domain, so phase 29's page joins these
|
||||
ports to the `haproxy`/`nginx` sections and an unclaimed socket stays visible as one.
|
||||
**The resolver section (phase 32 task 01) is the fourth and the only one that leaves the host.**
|
||||
`target_names(haproxy, nginx)` is the whole allow-list — haproxy `backends[].servers[].target`,
|
||||
then nginx `upstream` members, `server`-block `proxy_pass` targets and `maps[].entries[].target`,
|
||||
deduplicated keeping first-seen order — and nothing else may ever enter it: not a name from
|
||||
`/proc/net/tcp`, not a client address, not anything a request supplied. That is decision 3 and
|
||||
`tests/test_agent.py` pins it by asserting the spied `getaddrinfo` call set **equals** this
|
||||
function's output minus the address literals, in config order, with no duplicates, and that no
|
||||
address from the recorded socket table ever appears in it. The lookup is
|
||||
`socket.getaddrinfo(name, port, proto=SOCK_STREAM)` — `via: "libc"`, which is the resolution
|
||||
haproxy's `init-addr last,libc,none` performs **on the box that dials**; the dashboard resolving
|
||||
the same name through its own `resolv.conf` would be a different vantage answering a different
|
||||
question, so `sources/gateway.py`'s resolver spy pin survives this phase on purpose and the
|
||||
agent does **not** hand-roll UDP queries to the config's own `resolver` server either. That line
|
||||
is published instead, verbatim, as `config_directive` (`"10.1.0.1"` on this fleet) — and it lives
|
||||
inside this section rather than inside `nginx` so the `nginx` document stays **byte-stable**, which
|
||||
keeps phase 32's additivity control a strict top-level key comparison with no exception list
|
||||
(`Collector.nginx_document()` is the seam that returns the pair; `Collector.nginx()` is its
|
||||
one-line wrapper). Entry shape is `{name, port, status: ok|nxdomain|error|literal, addresses,
|
||||
ts}`: `addresses` is the unique answer sorted v4-ascending then v6-ascending (libc's order is not
|
||||
stable and `entries` is a list a page renders), an address literal is `literal` with **no** lookup
|
||||
(it is already an address), and `nxdomain`/`error` carry `addresses: []` — never a stale guess,
|
||||
never a `0`, and `EAI_AGAIN` is `error` and not `nxdomain`, because "the server did not answer"
|
||||
and "the name has no record" are two fixes. The pass is SMART's idiom for SMART's reason
|
||||
(`_resolver_lock`/`_resolver_cache`/`_resolver_refreshing`, `RESOLVE_MAX_NAMES` 64 /
|
||||
`RESOLVE_CACHE_SECONDS` 300 / `RESOLVE_PASS_BUDGET_SECONDS` 3.0): the **first** pass inline, every
|
||||
later expiry refreshed on a single-flight daemon thread while the request is served the previous
|
||||
result with its **original per-entry `ts`**, a failed refresh keeping the old entries and
|
||||
re-stamping so the retry lands on the next expiry rather than on every poll, and the budget/name-
|
||||
bound excess each reported in **one** line (`resolver: pass budget 3.0s exhausted, 41 of 58
|
||||
targets looked up`) with `available` still `true` — a black-holing resolver must never blow
|
||||
`HOST_TIMEOUT_SECONDS` and blink `/system` over an add-on it never draws. `available` is the
|
||||
family's three-way answer (`false` = nothing to resolve because no proxy config was read, one
|
||||
line saying **not found** or **unreadable** per section; `null` = a pass that could not run,
|
||||
which `snapshot()`'s `_safe` turns into the empty section plus its line), and **`--no-resolve`
|
||||
removes the key in silence** the way `--no-smart` does — while `--no-proxy` removes it too, since
|
||||
a box that collects no proxy has no declared names.
|
||||
**Task 04 made the three sections a versioned contract with captured ground truth:**
|
||||
`AGENT_VERSION` is **1.2.0** (`tests/test_agent.py` pins the constant and asserts the
|
||||
`GET /schema` block names `haproxy`/`nginx`/`sockets` — each with its `available` flag — plus
|
||||
`--no-proxy`, `--haproxy-config` and `--nginx-config`), and `scripts/capture-gateway-config.sh`
|
||||
re-records the six fixtures the parsers are pinned to over `GATEWAY_SSH` (`gateway-ops`): the
|
||||
haproxy config, the nginx main config **plus every document its own glob `include` names** (so
|
||||
the fixture tree cannot drift from the config's tree while the parser still follows one level),
|
||||
and `/proc/net/tcp{,6}`. Three rules it exists to keep. (a) **No privilege and no copy tool**: one
|
||||
`ssh <alias> cat <path>` per document, because `/proc/net/tcp` reports size 0 and `scp` would copy
|
||||
an empty file, and every path it reads is world-readable — the same read the agent performs as its
|
||||
own user, so a capture that needed sudo would be recording something the agent cannot see.
|
||||
`AGENT_VERSION` is **1.3.0** (`tests/test_agent.py` pins the constant and asserts the
|
||||
`GET /schema` block names `haproxy`/`nginx`/`sockets` — each with its `available` flag — plus the
|
||||
`resolver` section, its own `available`, and `--no-resolve`, beside `--no-proxy`,
|
||||
`--haproxy-config` and `--nginx-config`), and `scripts/capture-gateway-config.sh`
|
||||
re-records the seven fixtures the parsers and the join are pinned to over `GATEWAY_SSH`
|
||||
(`gateway-ops`): the haproxy config, the nginx main config **plus every document its own glob
|
||||
`include` names** (so the fixture tree cannot drift from the config's tree while the parser still
|
||||
follows one level), `/proc/net/tcp{,6}`, and `gateway_resolver.json` — the resolver section
|
||||
itself, captured the way the agent will produce it (task 02, below). Three rules it exists to
|
||||
keep. (a) **No privilege and no copy tool**: one `ssh <alias> cat <path>` per document and one
|
||||
`getent` per name per family, because `/proc/net/tcp` reports size 0 and `scp` would copy an
|
||||
empty file, and every path and binary it uses is world-readable and world-runnable — the same
|
||||
reads the agent performs as its own user, so a capture that needed sudo would be recording
|
||||
something the agent cannot see.
|
||||
(b) **A bad read is never a fixture**: a nonzero `cat`/`ls`, an empty body or a missing glob include
|
||||
exits non-zero having written nothing, and a half-written file is removed — the pin is that the
|
||||
recorded bytes survive a dead alias and an empty read both. (c) **Configs verbatim, socket tables
|
||||
@@ -343,7 +387,41 @@ Responsibilities, one line each:
|
||||
publish: the same rule that scrubs a drive's serial (invariant 6). Those two tables are volatile by
|
||||
nature, so `tests/test_agent.py` asserts their **shape**, the ports the recorded configs claim, and
|
||||
the absence of a globally routable endpoint — never a particular row, which a re-run may lawfully
|
||||
change. What phase 28 stopped at was the wire: `sources/host.py` still passes **none** of the
|
||||
change.
|
||||
**The seventh fixture is the resolver section, and it is derived rather than listed (phase 32
|
||||
task 02).** `tests/fixtures/gateway_resolver.json` is the agent's own `resolver` document — same
|
||||
keys, `via: "libc"`, entries in `target_names()` order — so a source test can exercise the
|
||||
name→address join hermetically instead of wishing for DNS. Three rules keep it honest. (a) **The
|
||||
name list is the parser's, never the script's**: a `plan` step rebuilds the agent's `--root` tree
|
||||
from the config fixtures in this repo, imports `agent.host_agent`, and asks `target_names()` +
|
||||
`address_family()` what to look up — so a route added to the gateway moves the fixture with it, an
|
||||
address literal costs no remote read, and `tests/test_deploy_scripts.py` fails the day a domain is
|
||||
hardcoded into the script. (b) **`getent ahostsv4` and `ahostsv6`, one name at a time**, because
|
||||
that is libc on the box that dials; `rc 2` is "no record" and becomes `status: "nxdomain"` — a
|
||||
fact, kept beside its successful neighbours — while `rc 255` (the ssh died), `rc 127` (no `getent`)
|
||||
or **not one name answering** is a broken vantage, so the run exits 1 having left the recorded
|
||||
document byte-identical: a fixture of nothing but `nxdomain` would make every live route on
|
||||
`/gateway` a permanent dash and every join test a test of nothing. The `ts` is a constant and the
|
||||
answer set is deduplicated and sorted with the agent's own `sort_resolver_addresses`, so
|
||||
re-capturing a gateway nobody changed is a no-op. (c) **A resolver answer is an identifier**, so the
|
||||
socket scrub runs over it too (`8.8.8.8` → `203.0.113.1`), and the `::ffff:` spelling `getent
|
||||
ahostsv6` synthesises for a name with no AAAA record is dropped when the A record answered — the
|
||||
agent's one `getaddrinfo` never publishes it. This fleet's targets all answer private LAN addresses,
|
||||
which is why the recorded document is scrub-free; `tests/test_agent.py` pins that absence as a
|
||||
warning sign, because a documentation-space address there would not necessarily match the one the
|
||||
socket fixtures gave the same endpoint, and `tests/test_gateway_source.py`'s join depends on their
|
||||
agreeing. The version's additivity is pinned two ways: permanently, by
|
||||
`SNAPSHOT_KEYS_BEFORE_1_3_0` (1.2.0's top-level keys) in `tests/test_agent.py`, whose only legal
|
||||
successor is that set plus `resolver` — and value-for-value, by `data/logs/verify-32-additive.py`,
|
||||
which imports `HEAD`'s agent file beside the working tree's and calls both against one `--root`.
|
||||
One defect only the second kind of control can find: the lookup is
|
||||
`getaddrinfo(name, port, type=SOCK_STREAM)`, and `proto=SOCK_STREAM` — the spelling that reads as
|
||||
"the TCP addresses" to anyone who has not read the man page — is `EAI_SERVICE` on real libc for
|
||||
**every** name, because `proto` takes a protocol number (`IPPROTO_TCP` = 6) and `SOCK_STREAM` is 1.
|
||||
A stubbed `getaddrinfo` accepts any keyword, so the whole resolver suite stayed green while the
|
||||
agent reported `error` on a healthy gateway; `test_the_lookup_the_agent_performs_is_one_real_libc_accepts`
|
||||
removes the module-wide stub and asks real libc about `localhost`, the one name it answers from
|
||||
`/etc/hosts` with no network. What phase 28 stopped at was the wire: `sources/host.py` still passes **none** of the
|
||||
three into `/api/system/snapshot`, and `store.py` still records no series for a config — the join
|
||||
and the two gauges it publishes are phase 29's `sources/gateway.py`, which reads the same
|
||||
`/snapshot` through its own `gateway` job. What the *system* page does read today is their
|
||||
@@ -434,14 +512,16 @@ Responsibilities, one line each:
|
||||
the agent's smartctl pass, and `None` is translated into httpx's sentinel because a
|
||||
literal `None` means *no timeout at all*). Agent `errors`
|
||||
become the payload's `degraded` **except** the add-on lines named by
|
||||
`ADDON_ERROR_PREFIXES` (`smart`, `haproxy`, `nginx`, `sockets`) — a host without smartmontools,
|
||||
without device read access, without a proxy config, or whose kernel socket table did not answer
|
||||
is unconfigured, not broken (invariant 3), and `degraded` is `alerts.py`'s input, so a fleet-wide
|
||||
`ADDON_ERROR_PREFIXES` (`smart`, `haproxy`, `nginx`, `sockets`, `resolver`) — a host without
|
||||
smartmontools,
|
||||
without device read access, without a proxy config, whose kernel socket table did not answer, or
|
||||
whose resolver had nothing to resolve / stopped mid-pass is unconfigured, not broken (invariant
|
||||
3), and `degraded` is `alerts.py`'s input, so a fleet-wide
|
||||
agent upgrade would otherwise put an overview WARN on every box that never ran haproxy or nginx.
|
||||
Those lines stay verbatim in `agent.errors` for the page that owns the add-on — `/disks` explains
|
||||
the `smart:` ones (and since phase 22 a host whose agent said `smartctl` is *missing* is not
|
||||
listed on that page at all, `api._disk_visible`), phase 29's gateway page reads the
|
||||
`haproxy:`/`nginx:`/`sockets:` ones the same way.
|
||||
`haproxy:`/`nginx:`/`sockets:`/`resolver:` ones the same way.
|
||||
**A new agent add-on section adds its prefix here in the same commit**, or its absence-report
|
||||
becomes a red banner on a page that has nothing to say about it.
|
||||
- `sources/host.py` — snapshot → payload + samples; reboot detection; GPU series per card
|
||||
@@ -490,14 +570,18 @@ Responsibilities, one line each:
|
||||
not warned about. What that retires is the guessing and the bookkeeping it existed to explain —
|
||||
the port-only guess for a name target, `unmatched` / `unmatched.groups` / `partial` /
|
||||
`unmatched_listeners`, and `by_port` with its `MAX_GROUP_HOSTS` chip. A **name** target
|
||||
(`reeseapps-immich.reeselink.com:443`) is therefore never credited: crediting it needs DNS (which
|
||||
the module still never calls, spy-pinned) and every name here resolves back through this same
|
||||
router, so the count would be attributed to something the config never said. Two consequences the
|
||||
(`reeseapps-immich.reeselink.com:443`) is credited **only through an address the gateway itself
|
||||
published** (phase 32, below): crediting it any other way needs DNS, which the module still never
|
||||
calls (spy-pinned), and every name here resolves back through this same router, so a vantage of
|
||||
ours would attribute a socket to a route on the strength of a query the router never made. Two
|
||||
consequences the
|
||||
page and the tests both depend on: **`live_out` is not the bandwidth** and the socket table never
|
||||
reconciles to it — a pipe count means "this route has N live connections" and nothing more — and
|
||||
several `(listen, endpoint)` lines reaching one endpoint carry one count **repeated** with
|
||||
`shared: true` (never a split nobody measured), which is why `/2`'s totals and the tests sum
|
||||
**distinct endpoints**. A pipe is `null` (never `0`) whenever the socket table did not answer or no
|
||||
`shared: true` (never a split nobody measured) — a shape that is real now that a name route is
|
||||
counted: two vhosts writing one `(address, port)`, two **names** the resolver placed on one
|
||||
address, or a multi-homed name sharing one address with a single-homed one — which is why the
|
||||
API's totals and the tests sum **distinct endpoints**. A pipe is `null` (never `0`) whenever the socket table did not answer or no
|
||||
line reaches its endpoint, and publishes **no sample** then, so a chart cannot draw a zero for an
|
||||
unknown. `supported: false`
|
||||
(a 1.1.0 agent, or `--no-proxy`) keeps every key with empty lists and null counts plus
|
||||
@@ -517,15 +601,44 @@ Responsibilities, one line each:
|
||||
line whose selectors union in first-seen order; **never** merged across `in_port`) and two silent
|
||||
refusals (a `proxy_pass` with neither target nor variable, and a haproxy frontend that routes by
|
||||
a fetch expression, which the agent reports as no name — that trunk is drawn alone, because "no
|
||||
name" and "no routing directive" are one value in this payload). **Name-keyed outbound gauges are
|
||||
retired** (phase 30 task 02): `gateway.out_conns@<host>:<port>` is written **only** for an endpoint
|
||||
the config wrote as an address (`out.address` true), because the sole reason a name-keyed gauge
|
||||
existed was the port-only guess the join no longer makes — such a series would be a chart of a
|
||||
number that is always `null`. `gateway.in_conns@<port>` is unchanged, and the port **stays** in the
|
||||
outbound name, because this fleet's two haproxy Minecraft backends share `minecraft.reeselink.com`
|
||||
on 25565 and 25566 and two gauges on one name would overwrite each other in silence; lines sharing
|
||||
an endpoint write the same value to the same name (they are one count, not two) and `samples()`
|
||||
de-duplicates by metric. `parse_gateway_snapshot()` is the
|
||||
name" and "no routing directive" are one value in this payload). **A name-written endpoint is
|
||||
joinable through the address the gateway published for it** (phase 32 task 03): `resolution_index()`
|
||||
keys agent 1.3.0's `resolver.entries` by `(name, port)` — `{}` when the section is absent or its
|
||||
`available` is not `true`, which is phase 30 byte-for-byte, and never an entry with no addresses —
|
||||
and `apply_resolution()` stamps every `out` with `resolved` (the address list, **never `[]`**: an
|
||||
entry with no addresses and no entry at all are one fact, so the page's dash arm is one `is null`
|
||||
test), `via` (the agent's own `"libc"`, copied, `null` where nothing was resolved) and
|
||||
`resolved_ts` (**that entry's** stamp, falling back to the section's, because a name refreshed five
|
||||
minutes ago and one answered on this poll are two claims). All three stay `null` on an
|
||||
`address: true` row: the config wrote that address, and copying the agent's `literal` echo of it
|
||||
into `resolved` would report a `getaddrinfo` that never ran. With an address and a port the line
|
||||
becomes countable, so **name-keyed gauges are back** — `gateway.out_conns@<domain>:<port>`, under
|
||||
the config's own spelling so history continues under one name. That is the reverse of phase 30
|
||||
task 02's retirement for exactly one reason: the retirement existed because the only figure such a
|
||||
series could carry was the *port's*, and the figure here is the sockets to the address this domain's
|
||||
own libc resolution produced **on the box that dials**. An unresolved name keeps `metric: null`,
|
||||
`connections: null`, no sample and the dash the footer explains. `gateway.in_conns@<port>` is
|
||||
unchanged, and the port **stays** in the outbound name, because this fleet's two haproxy Minecraft
|
||||
backends share `minecraft.reeselink.com` on 25565 and 25566 and two gauges on one name would
|
||||
overwrite each other in silence; lines sharing an endpoint write the same value to the same name
|
||||
(they are one count, not two) and `samples()` de-duplicates by metric.
|
||||
**The join key is a normalised address, never a spelling** (`_address_key`, `endpoint_of`,
|
||||
`_endpoint_groups`): `ipaddress` canonicalises and folds `/proc`'s `::ffff:10.1.2.64` to the
|
||||
`10.1.2.64` `getaddrinfo` returned — one kernel endpoint, one count, however many spellings or names
|
||||
reach it — while a zero-padded `010.1.2.64` stays **no one's address** (decimal versus octal is a
|
||||
guess, and a guess on the far end of a pipe is rule 2), so `out.resolved` keeps the agent's bytes and
|
||||
only the key is canonical. `_endpoint_groups()` is a union-find over those keys (a name route and an
|
||||
address route reaching one endpoint are one group; overlapping multi-homed sets merge; a route with
|
||||
no endpoint is no group) and `distinct_endpoint_counts()` is what `api._endpoint_values` sums for
|
||||
`live_out`, which is the only reason the head tally and the pipes cannot disagree. Matching stays
|
||||
exact equality on `(address, port)` — no prefix, no range — and this module still resolves nothing:
|
||||
`socket.getaddrinfo` is spy-pinned absent over a name-only synthetic, a resolver-full synthetic and
|
||||
the captured `resolved` ingress, plus `samples()` (a gauge name is built from a host string and is
|
||||
the other place a resolver could creep in). The payload's own `resolver` block is
|
||||
`{available, via, entries}` with each entry **projected** to `name`/`port`/`status`/`addresses`/`ts`
|
||||
— every entry including `nxdomain` and `error`, because the index filters *within* the published
|
||||
list and a page that says *this name has no record* needs the entry that says so — and a section
|
||||
that did not answer publishes none. `parse_gateway_snapshot()` is the
|
||||
`Result` the poller consumes; there are **no events** (a connection count has no transition
|
||||
worth a line — a proxy that stops routing shows up as its host's own poll failing). The
|
||||
never-answered arm is `stub(host, ts)` — the **same skeleton** `parse` fills in, plus
|
||||
@@ -939,7 +1052,11 @@ Responsibilities, one line each:
|
||||
or a `window_seconds`, at any depth (pinned by a recursive walk: a busy proxy is the job and no
|
||||
Δ was ever measured). Phase 30 task 02 took `outbound[]`, `unmatched` and `by_port` out of it: the
|
||||
flat list duplicated the graph with a worse count, and the residual existed only to explain a
|
||||
guessing rule that no longer runs. `/history` serves the `gateway.in_conns@<port>` /
|
||||
guessing rule that no longer runs. Phase 32 task 03 added one top-level key, `resolver` — the
|
||||
agent's own block, `{available, via, entries}`, each entry projected to the five fields the agent
|
||||
publishes — so a dash on one line is answerable from the document the page already holds; `stub()`
|
||||
grows it with nulls, which is what the stub/answer key-parity pin compares. `/history`
|
||||
serves the `gateway.in_conns@<port>` /
|
||||
`gateway.out_conns@<host>:<port>` gauges with `hosts=` defaulting to the ingress **alone** (the
|
||||
name the samples are stored under — defaulting to every target would draw empty plots for
|
||||
machines nobody asked to route), and it carries no configured gate because it dials nothing:
|
||||
@@ -989,9 +1106,12 @@ Responsibilities, one line each:
|
||||
`live_out` the **distinct outbound endpoints** — never the connection lines — because several
|
||||
`(listen, endpoint)` lines carry one socket count repeated with `shared: true`, and summing the
|
||||
lines would book the same sockets once per line and print a router busier than it is (the same
|
||||
rule that keys the outbound gauges by `(address, port)`); `api._endpoint_values()` is the one
|
||||
reader of that, and a line with no `metric` (its target is a name the join can never count) is
|
||||
absent rather than an unmeasured endpoint holding the total hostage. `map_targets` counts the
|
||||
rule that keys the outbound gauges by `(address, port)`, with the address normalised so two names
|
||||
— or two spellings — resolving to one endpoint book it **once**); `api._endpoint_values()` is the one
|
||||
reader of that (it delegates to `gateway_source.distinct_endpoint_counts()`, the same union-find the
|
||||
join builds its counters with, so the block's total and the pipes cannot disagree), and a line with
|
||||
no `metric` (a name nothing resolved) is absent rather than an unmeasured endpoint holding the total
|
||||
hostage. `map_targets` counts the
|
||||
stream map's routing entries. Every figure is `null` — never `0` — in the arms that did not
|
||||
measure (unconfigured, configured-but-never-answered, a 1.1.0/`--no-proxy agent, a silent socket
|
||||
table), and a total is `null` rather than a part of itself when any countable endpoint is unknown
|
||||
@@ -1317,10 +1437,12 @@ dashboard family — `host`, `llm`, `ups`, `disks` for the SMART series
|
||||
`disk.uncorrectable@…`, `disk.percentage_used@…`), and `gateway` for the phase-29 connection
|
||||
gauges (`gateway.in_conns@<listen-port>`, `gateway.out_conns@<host>:<port>` — the port belongs
|
||||
in the outbound name because two services on one host are two facts and one name would make the
|
||||
second insert overwrite the first; since phase 30 task 02 the outbound gauge is written **only** for
|
||||
an endpoint the config wrote as an **address**, so no name-keyed `gateway.out_conns@…` name is ever
|
||||
inserted again — a host that had one before this change keeps the old series until `maintain()` ages
|
||||
it out, and its chart simply stops growing points). Those keep their own family instead of
|
||||
second insert overwrite the first; the outbound gauge is written for an endpoint whose address the
|
||||
join knows — one the config wrote as an **address**, or (phase 32 task 03) one the gateway's own
|
||||
`resolver` section published for the name it wrote, under the config's own spelling, so the
|
||||
name-keyed `gateway.out_conns@<domain>:<port>` series phase 30 task 02 stopped inserting continue
|
||||
instead of restarting. A name nothing resolved is still never inserted, and its chart simply stops
|
||||
growing points). Those keep their own family instead of
|
||||
sitting beside `host`'s `disk.used_pct@/mount` because the cadences differ thirty-fold and
|
||||
`GET /api/metrics?source=disks` has to answer "what does *this page* have history for".
|
||||
That scheme is the history API's key — treat it as a public interface.
|
||||
@@ -1378,14 +1500,22 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
`https://` agent as `host:gateway` with the same certificate entry, and re-verified live on the
|
||||
dev server with no change to any pre-existing source.
|
||||
**The fleet is mTLS since 2026-10-21 (phase 24 task 03), so the bearer-token transport is now
|
||||
history rather than the designed shape.** `gateway` runs `agent/host_agent.py` **1.2.0** (phase 28
|
||||
task 05, 2026-10-22) and `deskwork`/`driveripper` still run **1.1.0**; the spread costs nothing,
|
||||
because 1.2.0 is purely additive — the three proxy-config sections are new keys and a 1.1.0
|
||||
snapshot simply has none — so those two move at their next re-stage rather than on a flag day.
|
||||
The control that proves "purely additive" is `data/logs/verify-28-final.py`: it stashes the whole
|
||||
history rather than the designed shape.** `gateway` runs `agent/host_agent.py` **1.3.0** (phase 32
|
||||
task 05, 2026-10-25, on its existing bundle and an unchanged unit — `data/logs/NOTES-32-rollout.md`)
|
||||
and `deskwork`/`driveripper` still run **1.1.0**; the spread costs nothing,
|
||||
because both upgrades were purely additive — the three proxy-config sections and the `resolver`
|
||||
section are new keys, and a 1.1.0 snapshot simply has **none of them**: no `available` flag, no
|
||||
error line, nothing for `collectors.ADDON_ERROR_PREFIXES` to keep out of `/system`'s `degraded`, and
|
||||
`sources/gateway.py` answers that absence as `supported: false` + `WHY_PREDATES` rather than an empty
|
||||
gateway — so those two move at their next re-stage rather than on a flag day. (The three-way
|
||||
`available: false` arm belongs to a **1.2.0/1.3.0** host that has no proxy config to read, which is
|
||||
a different fact from a host whose agent predates the section.)
|
||||
The controls that prove "purely additive" are `data/logs/verify-28-final.py` and
|
||||
`data/logs/verify-32-additive.py`: each stashes the whole
|
||||
working tree, imports both `host_agent.py` files side by side, calls both `collect()`s against
|
||||
the same `--root`, and demands the 1.2.0 document equal the 1.1.0 document key-for-key with
|
||||
`haproxy`/`nginx`/`sockets` added — and 1.2.0 with `--no-proxy` equal it exactly.
|
||||
the same `--root`, and demands the newer document equal the older one key-for-key with
|
||||
`haproxy`/`nginx`/`sockets` (phase 28) and `resolver` (phase 32) added — and the newer one with
|
||||
`--no-proxy` / `--no-resolve` equal it exactly.
|
||||
All three are installed from a bundle, every unit carries `--cert-bundle /etc/dashboards-agent.certs` (plus
|
||||
`RuntimeDirectory=dashboards-agent`, and still its `--token-file` line), and `HOST_TARGETS` is
|
||||
`"driveripper=https://10.1.0.126:9120 deskwork=https://10.1.0.51:9120
|
||||
@@ -1495,7 +1625,8 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
+ `CapabilityBoundingSet` unit with the snapshot's capability-less one, **silently**. That trap
|
||||
is now closed on all three hosts: `/home/ops/agent-install/` is `ops`-owned, so the repo's
|
||||
`deploy/install-agent.sh` (sha256 `ffbcd91f…`) and `agent/host_agent.py` (**1.1.0**, re-staged as
|
||||
**1.2.0** on `gateway` in phase 28) were staged over it without root, and each snapshot was left
|
||||
**1.2.0** on `gateway` in phase 28 and **1.3.0** in phase 32) were staged over it without root, and
|
||||
each snapshot was left
|
||||
beside the grant as
|
||||
`install-agent.sh.oct18-backup`, which sudo will not run. **Re-staging is therefore the first
|
||||
step of any install, not an optional one** — the grant is only as current as the file it points
|
||||
@@ -1653,11 +1784,14 @@ uv run ruff check src tests agent && uv run pyright
|
||||
SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
|
||||
# ... plus smartctl per physical disk (needs device read
|
||||
# access: plain smartctl, else `sudo -n`; scrubs serials)
|
||||
./scripts/capture-gateway-config.sh # ... plus the gateway's proxy tree and socket tables:
|
||||
# haproxy.cfg, nginx.conf + its glob includes, and
|
||||
# /proc/net/tcp{,6} over $GATEWAY_SSH (gateway-ops);
|
||||
# world-readable paths only, and a failed or empty read
|
||||
# never replaces a good fixture
|
||||
./scripts/capture-gateway-config.sh # ... plus the gateway's proxy tree, socket tables and
|
||||
# resolved targets: haproxy.cfg, nginx.conf + its glob
|
||||
# includes, /proc/net/tcp{,6}, and gateway_resolver.json
|
||||
# (the names `target_names()` derives from those configs,
|
||||
# read back with `getent ahostsv4/6` — no DNS client, no
|
||||
# sudo) over $GATEWAY_SSH (gateway-ops); a failed or empty
|
||||
# read never replaces a good fixture, and a vantage where
|
||||
# not one name answers is refused rather than recorded
|
||||
|
||||
bash .agents/validate.sh # the phase gate: the four checks above (pytest
|
||||
# --cov --cov-fail-under=94, ruff, pyright, plus a
|
||||
@@ -1743,20 +1877,25 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
|
||||
is compared with the agent's document and not with a copy of its field names, and then demands the
|
||||
two attributions and nothing beyond them: a socket whose **local** port names a listener is
|
||||
inbound, a socket whose remote `(address, port)` equals a configured target is outbound, and
|
||||
**every other socket is ignored** — the capture's `established` (18 that day) and the two totals
|
||||
(9 in, 2 out) are therefore both printed and deliberately never reconciled, because the identity
|
||||
this suite used to demand (in + out + unmatched = the table) was the arithmetic of a guessing rule.
|
||||
The rules that must never come back are pinned as absences: no `by_port`/`partial`/`unmatched`
|
||||
**every other socket is ignored** — the capture's `established` and the two totals are therefore
|
||||
both printed and deliberately never reconciled, because the identity this suite used to demand
|
||||
(in + out + unmatched = the table) was the arithmetic of a guessing rule, and because most of a
|
||||
busy router's sockets are nobody's pipe, which is why **`live_out` is not the bandwidth**. The
|
||||
rules that must never come back are pinned as absences: no `by_port`/`partial`/`unmatched`
|
||||
vocabulary in the payload or on the page, no `0` for a pipe the join could not count, and
|
||||
`socket.getaddrinfo` spied so the day the join resolves a name the run fails — a name is never
|
||||
credited with a connection (rule 2), which is why 14 of this fleet's 18 live sockets are nobody's
|
||||
pipe and why **`live_out` is not the bandwidth**. One number in that suite is a **date**, not a fact:
|
||||
the captured `stream.conf` has 17 `map` entries (so 32 connections and 17 leaves on `:443`), while the
|
||||
live router answered **18** entries / 33 connections on 2026-10-23 (`data/logs/30-live-verify.out`)
|
||||
because a domain was added after the capture. That is the designed difference between a fixture and a
|
||||
reading — folding the drift in means re-running `./scripts/capture-gateway-config.sh` **and**
|
||||
re-tallying `tests/test_gateway_source.py` in the same commit, never editing a test to match a live
|
||||
read.
|
||||
credited with a connection (rule 2).
|
||||
**Config-derived counts are literals in these tests; socket-derived ones are computed from the
|
||||
same capture.** `tests/test_api.py` asserts 10 listeners, 33 connections and 18 `map_targets`
|
||||
as numbers because only the operator moves those, and derives `live_in`, `live_out`, each pipe's
|
||||
count and `established` from `gateway_snapshot("full")` because a second capture of a running
|
||||
router lawfully changes them — pinning the live totals as literals is how a re-capture turns a
|
||||
green suite red over nothing. The rule is the assertion; the reading is not.
|
||||
Which is the same discipline that closed the drift phase 32 task 02 folded in: that capture's
|
||||
`stream.conf` gained a `map` entry (a domain was added to the router after phase 30 recorded it),
|
||||
so 17 became 18 and 32 connections became 33 **in the commit that re-ran
|
||||
`./scripts/capture-gateway-config.sh`** — re-tallying the config-derived counts, never editing a
|
||||
test to match a live reading while the fixture stays where it was.
|
||||
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
|
||||
rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
|
||||
- **A verdict the operator sees is pinned on the surface that shows it, in both directions.**
|
||||
@@ -2215,17 +2354,37 @@ for a difference of one connection (the `/llm` slot-grid rule). `pruneCards()` w
|
||||
map is the only bulk clear, for the case where the family itself left the payload.
|
||||
|
||||
A count is printed by `countText()`, which answers `null` for anything that is not a finite number —
|
||||
**never `0`** (rule 2) — and the two unknowns stay two sentences: a **trunk** whose socket table went
|
||||
quiet prints `n/a` (`setTxt`'s own word), while a **leaf** whose endpoint the config wrote as a
|
||||
**name** prints an em dash (`NO_COUNT`) whose `title` says `connections not attributed — the config
|
||||
writes a name and nothing here resolves names`. That is the normal look of 31 of this ingress's 33
|
||||
lines, and a column of `n/a` would be twelve identical words competing with the page's one footnote.
|
||||
The footer's second paragraph is the same sentence in the operator's terms rather than the join's —
|
||||
*a dash means the endpoint is written in the config as a name; the page matches sockets by address and
|
||||
resolves nothing, so this route is drawn but not counted* — because a column of dashes needs one
|
||||
explanation on the page and not thirty-two on the rows, and because it is a **name** that is the reason
|
||||
and not a fault. `32_gateway_resolved_target_liveness` is what removes the reason; until then the dash
|
||||
stays unexplained-looking by design rather than borrowing the trunk's `n/a` word.
|
||||
**never `0`** (rule 2). Phase 32 task 04 made the missing count say **which** missing count it is:
|
||||
`countCause()` reads the same two fields `sources/gateway.py` joins on (`out.address === true` or a
|
||||
non-empty `out.resolved`) and returns one of two fragments, so `no number: the gateway could not
|
||||
resolve this name in this reading` is printed only when nothing is known for the endpoint, and
|
||||
`no number: the socket table did not answer this reading` only when an address **is** known and
|
||||
nobody measured — because those are two operator actions, and either one drawn as the other sends
|
||||
someone to the wrong box. Both go out through `countWhy()` on the `title` and through the same
|
||||
`countCause()` in `leafAria()` (`connections not attributed: …`), so a row's tooltip and its
|
||||
accessible name cannot disagree, and a dash is never the trunk's `n/a`, which is the trunk's own
|
||||
fact about its own port. **A `0` is not an unknown at all**: a resolved endpoint with no socket is a
|
||||
measurement, keeps `0 open connection(s) on this endpoint`, and never borrows either sentence
|
||||
(`test_an_unresolved_name_and_an_idle_route_say_two_different_sentences`, and
|
||||
`data/logs/32-task04-sentences.cjs`, which serves all four arms through route interception because
|
||||
the live 1.2.0 ingress can only produce one of them). Every **counted** line gets its provenance on
|
||||
the same `title` rather than on a column of ink: `countTitle()` appends
|
||||
`· address from <via>, resolved <age>` — `via` copied from the payload (`libc`) and the age through
|
||||
`duration(ageSeconds(out.resolved_ts))` — and prints neither fragment on a row the config wrote as an
|
||||
address, because `out.address` is already the whole story and echoing the agent's `literal` back as a
|
||||
lookup would report a `getaddrinfo` that never ran; a missing `resolved_ts` costs the age and nothing
|
||||
else, so no row prints `n/a` or `NaN` where an age belongs. Phase 32 task 03 left the rendering alone
|
||||
and changed what lands in the arms: a name-written route is now counted through the address the
|
||||
**gateway** resolved, so on the **captured** ingress all 33 lines carry a number (26 of them a
|
||||
measured `0`) and a dash has become the exception that means *this name has no record*
|
||||
(`nxdomain`/`error` in `resolver.entries`, or no entry beside it — including no section at all, which
|
||||
is what a 1.2.0 agent or `--no-resolve` answers (the live ingress has run 1.3.0 since phase 32 task 05
|
||||
on 2026-10-25, where **33 of 33** rows carried a number and none dashed; the same pass recorded
|
||||
`resolver.available: true`, 32 entries, `via: "libc"`, `config_directive: "10.1.0.1"`, 30 `ok` + 2
|
||||
`literal` targets). The footer's second paragraph says the same thing once for the whole page — a dash means the
|
||||
gateway could not resolve that name in this reading, hover a counted line for the address it matched
|
||||
and when that answer was taken — and the footer is still four `<p>`s (phase 31 task 04's filter
|
||||
footnote is the fourth, which is why the count that phase's pin demands is four and not three).
|
||||
The line itself is hand-rolled SVG (`role="img"` + an `aria-label` naming the inbound port, the
|
||||
endpoint, the selector and the count) and the row carries **two independent class pairs** (phase 30
|
||||
task 05, moved onto the `<li>` by phase 31 task 01), because the operator's green and the socket count
|
||||
@@ -2316,7 +2475,10 @@ the field is the one control that could be read as a query, and it is not one.
|
||||
**`#gate-filter` is an eye, not a query** (phase 31 task 04). 33 routes in config order is not a
|
||||
lookup, and the operator's question is always about one domain, so one field above the panels hides the
|
||||
rows whose **own payload text** does not contain what was typed: `in.port`, `out.host`, `out.name`,
|
||||
`out.kind` and every `out.selectors` entry for a route (`haystack()`), `:${port}` plus
|
||||
`out.kind`, every `out.selectors` entry and every `out.resolved` address for a route (`haystack()`;
|
||||
the sixth field is phase 32 task 04's, and it is what makes typing `10.1.2.64` find the route whose
|
||||
name target resolves there — a haystack of names could not answer a question asked about a socket),
|
||||
`:${port}` plus
|
||||
`listeners[].domains` and `.clients` for a trunk (`trunkHaystack()`). Nothing else is readable and
|
||||
nothing else is matched — no DNS, no re-ranking, no rule about `.reeselink.com`, and no page-level
|
||||
split by domain suffix, because "`reeseapps.com` vs `stackexpected.com`" is a fact neither config file
|
||||
@@ -3011,7 +3173,7 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
||||
The rollout is done and verified: gateway, deskwork and driveripper each run `agent/host_agent.py`
|
||||
**1.1.0** with `/etc/dashboards-agent.certs` installed `0640 root:dashboards-agent`, their units
|
||||
carry `--cert-bundle` + `RuntimeDirectory=dashboards-agent` (and keep `--token-file`) — the
|
||||
gateway has since moved to **1.2.0** for phase 28's proxy-config sections on the same unit and
|
||||
gateway has since moved to **1.2.0** (phase 28) and then **1.3.0** (phase 32) on the same unit and
|
||||
the same bundle, see the fleet record above —
|
||||
`HOST_TARGETS` is `https://` for all three with no userinfo, `agent.bundle_rev` reads `2` on
|
||||
every host and equals what `/api/certs` issued, and the nine sources are `ok`. The transcript is
|
||||
@@ -3275,10 +3437,16 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
||||
|
||||
13. **`/gateway`'s long endpoint labels are ellipsised on a phone — a column-width question, not a
|
||||
lost fact, and the filter is what a clipped label used to block** (found by phase 29 task 06's
|
||||
live pass, **closed at the desktop width by phase 30 task 04's trunk→leaf row and re-measured
|
||||
after phase 31 re-ranked the row on 2026-10-26**; `data/logs/31-route-readability.out` is the
|
||||
current measurement, `data/logs/30-connection-map.out`, `data/logs/30-task04-panels.out`,
|
||||
`data/logs/29-task06-live.out` and `data/logs/29-task06-card-haproxy.png` the history).
|
||||
live pass, **closed at the desktop width by phase 30 task 04's trunk→leaf row, re-measured
|
||||
after phase 31 re-ranked the row on 2026-10-26 and again after phase 32 put the count
|
||||
provenance on the `title` on 2026-10-26**; `data/logs/32-counted-routes.out` is the current
|
||||
measurement, `data/logs/31-route-readability.out`, `data/logs/30-connection-map.out`,
|
||||
`data/logs/30-task04-panels.out`, `data/logs/29-task06-live.out` and
|
||||
`data/logs/29-task06-card-haproxy.png` the history). Phase 32's re-check found the clip
|
||||
counts **unchanged** by the DOM/provenance move — **19 of 33** leaf inbound lines and **33
|
||||
of 33** counterpart lines at 360 px (the 10 trunk names, none clipped), 0 of 33 at 1440 px,
|
||||
and the whole pair on **33 of 33** rows' tooltips — so the trade is the same and only the
|
||||
tooltip's second sentence (the provenance) is new.
|
||||
|
||||
**Phase 30 task 04 closed the desktop half by changing the row, not the font.** The two
|
||||
equal-width pipe columns of phase 29 became a group of `minmax(0, 9.5rem) minmax(0, 1fr)` — a
|
||||
@@ -3330,16 +3498,40 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
||||
declarations and removed the reason they were needed: with the trunk in its own track, the leaf's
|
||||
name is the widest thing in a column that is now wide enough for it.
|
||||
|
||||
14. **A route whose endpoint the config writes as a *name* is drawn and never counted —
|
||||
`32_gateway_resolved_target_liveness` is the phase that removes the reason** (recorded by phase
|
||||
31 task 05, which wrote no design for it). Today 31 of this ingress's 33 routes end in a
|
||||
`server_name`/`map` target and so carry an em dash where a live count would go, because
|
||||
`sources/gateway.py` matches sockets by
|
||||
address and resolves nothing (rule 2 by choice, and pinned as an absence). The follow-on phase is
|
||||
the one that makes such a target answer with its liveness and its live connections anyway — the
|
||||
agent reporting resolved targets on the box that owns the config, which is an `AGENT_VERSION` bump
|
||||
to **1.3.0** and therefore an agent + source + pin change, not a page change. Its acceptance line
|
||||
is that a dash disappears from `/gateway` for a reason and not by hiding the column.
|
||||
14. **A name-written route is counted through the address the gateway published — closed, the
|
||||
re-stage included** (recorded by phase 31 task 05, implemented by
|
||||
phase 32 tasks 01-03). Until phase 32, 31 of this ingress's 33 routes ended in a
|
||||
`server_name`/`map` target and carried an em dash where a live count would go, because
|
||||
`sources/gateway.py` matches sockets by address and resolves nothing. That is still true — the
|
||||
module calls no resolver, spy-pinned over the parse, the gauge names and the capture — but it no
|
||||
longer has to, because agent **1.3.0** publishes `resolver`: every `(name, port)` its own two
|
||||
config files write as a target, answered by libc **on the box that opens the connection**, with
|
||||
the nginx `resolver` directive printed beside it as `config_directive` so a divergence between the
|
||||
two vantages is visible instead of guessed. `sources/gateway.py` reads that document, stamps
|
||||
`out.resolved` / `via` / `resolved_ts`, joins sockets through the normalised address, and writes
|
||||
the `gateway.out_conns@<domain>:<port>` gauge under the config's own spelling — so the dash is gone
|
||||
on the live page **because the gateway said what the name became**, which was the acceptance line,
|
||||
and not because a column was hidden.
|
||||
**The deployment half is done.** `gateway` ran **1.2.0** until phase 32 task 05 re-staged it on
|
||||
2026-10-25 (`data/logs/NOTES-32-rollout.md`: unit sha256 unchanged, bundle untouched, the install
|
||||
run guard silent, one file and one restart), and that pass read `/api/gateway/snapshot` going from
|
||||
**2 of 33** rows carrying a number to **33 of 33**, with all 31 resolver-placed rows finite and
|
||||
`test_a_resolved_target_on_the_live_ingress_is_never_a_dash` passing instead of skipping. What is
|
||||
left of the gap is rotation hygiene, not a step: `deskwork` and `driveripper` stay on 1.1.0 (no
|
||||
proxy configs, therefore nothing to resolve), and a host that later rolls **back** to a pre-1.3.0
|
||||
file simply returns its name routes to dashes — `sources/gateway.py` treats a missing `resolver`
|
||||
section as phase-30 behaviour. A name that stops
|
||||
resolving is `nxdomain`/`error` and goes back to being a dash on the next poll (the agent caches
|
||||
300 s and never publishes a stale address as current), which is the designed degradation rather
|
||||
than a gap — and since task 04 the page says **which** degradation it is: `countCause()` separates
|
||||
that dash (`the gateway could not resolve this name in this reading`) from the one an unreadable
|
||||
socket table causes (`the socket table did not answer this reading`), while a resolved endpoint with
|
||||
nothing open prints its `0` beside `· address from libc, resolved <age>`, so the sentence a line
|
||||
carries names the box to go and look at. `data/logs/32-task04-sentences.cjs` measures all four arms,
|
||||
serving the three a pre-1.3.0 host could not produce through route interception (43 checks, and
|
||||
each stub leg
|
||||
first proves its DOM equals the document that leg was served, so a leg that rendered the live page
|
||||
instead fails by name rather than three checks later).
|
||||
|
||||
15. **`/gateway`'s switches are knobs with no write behind them — `33_gateway_route_switching` is the
|
||||
phase that decides whether they ever get one** (recorded by phase 31 task 05, no design written).
|
||||
|
||||
@@ -10,7 +10,7 @@ file:
|
||||
| `/disks` | Are any of my **physical** disks failing, and which one — health, temperature, power-on hours, wear, sector alarms | the same agents, running `smartctl -i -A -H` per whole drive (never a partition or a mount); a host whose agent has no `smartctl` is not a host on this page |
|
||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
|
||||
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
|
||||
| `/gateway` | Which domains are coming in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`): a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) on line 1 over the counterpart it routes to on line 2 (`→ host:port`), plus its live connection count — and one filter box above the map that hides the routes whose own labels do not contain what you typed, asking nothing of the server | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |
|
||||
| `/gateway` | Which domains are coming in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`): a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) on line 1 over the counterpart it routes to on line 2 (`→ host:port`), plus its live connection count — and one filter box above the map that hides the routes whose own labels do not contain what you typed, asking nothing of the server | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, plus **1.3.0**'s `resolver` section to put a number on a route whose target the config writes as a *name* (without it those lines are drawn and dashed), and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |
|
||||
|
||||
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
|
||||
runtime; every asset is served by this app.
|
||||
@@ -212,23 +212,88 @@ an address: the name-keyed ones were a chart of the port-only guess the join no
|
||||
would be permanently empty). Nothing in `/api/system/snapshot` grows a proxy or socket field, and no
|
||||
`sample` row is written for a config.
|
||||
|
||||
All three parsers are pinned to captures rather than to descriptions, and
|
||||
The `resolver` section is the fourth and the only one that leaves the host. The two configs above
|
||||
name their targets two ways — an address (`proxy_pass http://127.0.0.1:8081`) and a name
|
||||
(`server mc1 minecraft.reeselink.com:25565`) — and the kernel's socket table speaks addresses, so a
|
||||
name is the half a page could never count. The agent therefore takes every `(name, port)` the two
|
||||
**parsed** sections write as a target (haproxy `server` lines, then nginx `upstream` members,
|
||||
`proxy_pass` targets and `map` entry targets, deduplicated in config order) and asks libc what each
|
||||
one is: `getaddrinfo` over `SOCK_STREAM`, published as `via: "libc"` because that is the resolution
|
||||
haproxy's own libc path performs **on this box** — the box that opens the connections. The nginx
|
||||
config's own `resolver` directive travels beside it verbatim as `config_directive` (`10.1.0.1`
|
||||
here, read out of `stream.d/stream.conf`), so a divergence between the two vantages is something an
|
||||
operator can see rather than something this file guesses; the agent does not query that resolver
|
||||
directly, and the dashboard never resolves anything at all (`sources/gateway.py` has a test that
|
||||
fails if it ever calls a resolver). An address literal is reported `status: "literal"` with its
|
||||
address and **no** lookup, and a name that does not answer right now is `nxdomain` or `error` with
|
||||
`addresses: []` — the route keeps its row and carries no count, never a `0`.
|
||||
|
||||
That pass is bounded and cached exactly the way the SMART one is, for the SMART reason: at most
|
||||
**64** lookups inside **3 s** of wall clock, cached **300 s**, the first pass inline so a freshly
|
||||
started agent answers with real addresses and every later expiry refreshed on a single background
|
||||
thread while the request is served the previous result. `getaddrinfo` takes no timeout argument, so
|
||||
stopping between lookups is the only defence a black-holing resolver leaves — and the cost of not
|
||||
stopping is the *system* page's poll timeout, spent on an add-on that page never draws. Each entry
|
||||
carries the time **it** was answered, so a cache-served entry cannot pretend to be current; the
|
||||
budget and the name bound each report themselves in one `errors` line and leave `available` at
|
||||
`true`. `available` is the family's three-way answer (`false` means there was nothing to resolve
|
||||
because no proxy config was read, and its one line says **not found** or **unreadable** according to
|
||||
which of the two it was), and
|
||||
`--no-resolve` removes the section in silence the way `--no-smart` removes SMART — while `--no-proxy`
|
||||
removes it too, because a box that collects no proxy has no declared names to look up.
|
||||
|
||||
All four parsers are pinned to captures rather than to descriptions, and
|
||||
`./scripts/capture-gateway-config.sh` re-records them over the `GATEWAY_SSH` alias (default
|
||||
`gateway-ops`): `/etc/haproxy/haproxy.cfg`, `/etc/nginx/nginx.conf` plus every document its own
|
||||
glob `include` lines name (`gateway_nginx_conf_d_*.conf`, `gateway_nginx_stream_d_*.conf`), and
|
||||
`/proc/net/tcp{,6}`. No sudo — those paths are world-readable, which is exactly what the agent
|
||||
reads as itself — and a read that fails, or answers empty, leaves the existing fixture alone and
|
||||
exits 1, so a laptop with no route to the gateway cannot turn six captures into six empty files.
|
||||
glob `include` lines name (`gateway_nginx_conf_d_*.conf`, `gateway_nginx_stream_d_*.conf`),
|
||||
`/proc/net/tcp{,6}`, and `tests/fixtures/gateway_resolver.json` — the `resolver` section itself, so
|
||||
the name→address join is testable with no network. No sudo and no DNS client: those paths are
|
||||
world-readable, which is exactly what the agent reads as itself, and the resolver is recorded with
|
||||
`getent ahostsv4` / `getent ahostsv6`, which is libc on the box that dials (the same vantage
|
||||
`via: "libc"` claims). A read that fails, or answers empty, leaves the existing fixture alone and
|
||||
exits 1, so a laptop with no route to the gateway cannot turn seven captures into seven empty files.
|
||||
|
||||
The resolver document is **derived, never listed**: the script feeds its own freshly captured
|
||||
configs through `agent.host_agent`'s parsers and `target_names()`, so it asks about exactly the
|
||||
targets the agent will ask about, an address literal costs no remote read, and a route added to the
|
||||
gateway moves the fixture with it. `getent` answering `rc 2` is a fact — that name has no record and
|
||||
becomes `status: "nxdomain"`, kept beside the names that did answer — while a dead transport, a
|
||||
missing `getent`, or **not one name answering** is a broken vantage: the run refuses, because a
|
||||
document of nothing but `nxdomain` would make every route on `/gateway` a permanent dash and every
|
||||
join test a test of nothing. Re-capturing a gateway nobody changed is a no-op (fixed `ts`,
|
||||
deduplicated, sorted with the agent's own `sort_resolver_addresses`), so a diff in that file means
|
||||
the ingress changed.
|
||||
|
||||
The configs are not rewritten (their domains are the ingress map, and key material appears only
|
||||
as the paths nginx is told to read); the socket tables lose only their identifiers, in line with
|
||||
the rule that scrubs a drive's serial: a globally routable address becomes a documentation one
|
||||
(RFC 5737 / RFC 3849), because a global IPv6 on Linux is EUI-64 and an inbound visitor's address
|
||||
is not ours to publish. Every port, state, column and width in those tables is verbatim, and
|
||||
those files are volatile by nature — a proxy in service has a different connection set every
|
||||
second — so the tests assert shape and the ports the configs claim, never a particular row.
|
||||
`AGENT_VERSION` is **1.2.0**, the version that ships all three sections; a host still running
|
||||
1.1.0 answers `GET /schema` with a block that names none of them, and its snapshot simply has no
|
||||
such keys.
|
||||
as the paths nginx is told to read); the socket tables and the resolver's answers lose only their
|
||||
identifiers, in line with the rule that scrubs a drive's serial: a globally routable address becomes
|
||||
a documentation one (RFC 5737 / RFC 3849), because a global IPv6 on Linux is EUI-64 and an inbound
|
||||
visitor's address is not ours to publish. Every port, state, column and width in those tables is
|
||||
verbatim, and those files are volatile by nature — a proxy in service has a different connection set
|
||||
every second — so the tests assert shape and the ports the configs claim, never a particular row.
|
||||
`AGENT_VERSION` is **1.3.0**, the version that adds the `resolver` section on top of the three
|
||||
phase 28 shipped; `GET /schema` names all four, each with its `available` flag, and `--no-proxy`
|
||||
and `--no-resolve` are listed there too. A host still running 1.2.0 answers with a snapshot that
|
||||
simply has no `resolver` key — which is why the version spread costs nothing: a section an agent
|
||||
did not collect is absence, and the gateway page counts the address-written routes it has and draws
|
||||
the name-written ones with a dash.
|
||||
|
||||
**The dashboard reads that section and never produces one.** `sources/gateway.py` indexes the
|
||||
entries by `(name, port)`, stamps each route's `out` with `resolved` / `via` / `resolved_ts` — the
|
||||
address the gateway dialled, the agent's own string for how it got it, and the moment that **entry**
|
||||
was answered — and lets the socket join reach a name-written route through it. Which is the whole
|
||||
of phase 32: the outbound endpoint of a `server` line, a `proxy_pass` or a `map` entry is now an
|
||||
address the box that opens the connection published, so the `gateway.out_conns@<domain>:<port>`
|
||||
gauges phase 30 task 02 stopped writing are written again, under the config's own spelling, with a
|
||||
figure that belongs to the domain instead of to whatever else was listening on its port. Four rules
|
||||
keep the join honest, each pinned in `tests/test_gateway_source.py`: a name nobody resolved keeps
|
||||
`resolved: null`, `connections: null`, `metric: null` and **no stored sample** (never `0`, never an
|
||||
empty list); matching is by normalised address, so `::ffff:10.1.2.64` in `/proc` and `10.1.2.64`
|
||||
from `getaddrinfo` are one endpoint while a zero-padded `010.1.2.64` — which is no one's address
|
||||
notation — matches nothing; two names resolving to one address, or two route lines reaching it, are
|
||||
**one** count held by every line that reaches it; and the module still never calls a resolver, a spy
|
||||
pin covering the parse, the gauge name and the capture alike. An `address: true` route claims none
|
||||
of the three provenance fields, because the config wrote that address and no lookup happened.
|
||||
|
||||
`deploy/install-agent.sh --fetch-bundle <url> --cert-name <name> --session <token>` is the same
|
||||
TLS install with the download done inside the registration run: it asks
|
||||
@@ -431,9 +496,12 @@ No `--cert-bundle`, no `--fetch-bundle`, no `--force`: the bundle already on dis
|
||||
render reads — the flag only replaces the file — so the render equals the installed unit, the re-run
|
||||
guard prints nothing, and `/etc/dashboards-agent.certs` and the token file keep their bytes. Then
|
||||
the same two checks the registration uses: the CA-only `--status` probe (with `--bind`) answers
|
||||
`{"ok": true, …, "version": "1.2.0"}` on `/health`, and the dashboard's own `https://` dial answers
|
||||
`{"ok": true, …, "version": "1.3.0"}` on `/health`, and the dashboard's own `https://` dial answers
|
||||
whatever the new version added — `agent_version` plus the `haproxy` / `nginx` / `sockets` sections
|
||||
for a 1.2.0 agent — beside the sections it always had.
|
||||
(1.2.0) and the `resolver` section (1.3.0) — beside the sections it always had. That is exactly how
|
||||
`gateway` moved 1.1.0 → 1.2.0 (phase 28) and 1.2.0 → **1.3.0** (phase 32, 2026-10-25: the same unit
|
||||
sha256 before and after, the bundle untouched, the guard silent, one file and one restart), and the
|
||||
`/gateway` page counted **33 of 33** routes afterwards where it had counted 2 before.
|
||||
|
||||
Two things the first full rollout learned, both cheap to hit and expensive to guess at:
|
||||
|
||||
@@ -887,8 +955,10 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
||||
`connections` graph (phase 30: one row per route the two config files declare, each naming its
|
||||
inbound port, its outbound endpoint, and the config's own selector for it — a haproxy backend
|
||||
name, an nginx `server_name`, a stream `map` key) with its **live count** — the socket table's
|
||||
ESTABLISHED rows, inbound by the socket's local port and outbound only where a remote
|
||||
`(address, port)` equals a target the config wrote as an address. Each row carries the `metric` it
|
||||
ESTABLISHED rows, inbound by the socket's local port and outbound where a remote
|
||||
`(address, port)` equals an endpoint the join can name — a target the config wrote as an address, or
|
||||
(phase 32 task 03) one of the addresses the gateway's own `resolver` section published for the name
|
||||
it wrote. Each row carries the `metric` it
|
||||
is stored under, and `shared: true` when more than one route reaches its endpoint (one count, shown
|
||||
twice, never split). `supported`/`sections`/`why` cover the degradation arms. `warnings` (phase 30
|
||||
task 03) is the list of the two config files **not connecting to themselves** — seven kinds, each
|
||||
@@ -903,16 +973,25 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
||||
or absent section warns about nothing — a host that cannot be read has not been found guilty of
|
||||
anything — and a `warnings: []` is therefore the truth, not a shrug. Non-zero, `/` gets **one**
|
||||
amber row (`N proxy config problems to address — <first message>`) and this page gets the list. **Nothing reconciles
|
||||
to `sockets.established` and that is the design**: a connection to a target the config wrote as a
|
||||
name (most of this fleet's fan-out) or on a port no config names is ignored — not counted, not
|
||||
totalled, not warned about — so a pipe count means "this route has N live connections" and nothing
|
||||
more. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
|
||||
to `sockets.established` and that is the design**: a connection to a target nothing can name (a
|
||||
name with no resolver entry behind it) or a socket on a port no config names is ignored — not
|
||||
counted, not totalled, not warned about — so a pipe count means "this route has N live connections"
|
||||
and nothing more. The payload also carries the agent's `resolver` block (`available`, `via`, and
|
||||
the projected `entries`: name, port, status, addresses, ts — nothing an operator has not reviewed)
|
||||
beside the `sections` pair that already says which halves answered, so a dash on one line is
|
||||
explainable from the document the page already holds. On the recorded
|
||||
ingress that section places every declared route, which makes the dash the exception rather than
|
||||
the normal look of this page, and its meaning is *this name has no record* — and on the live fleet
|
||||
it is the exception too: the gateway has run agent **1.3.0** since 2026-10-25, `resolver.available`
|
||||
answers `true` with one entry per declared target, and that host's page counted **33 of 33** routes
|
||||
with no dash at all. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
|
||||
a `404 {"configured": false}` answered before any client is built (both arms spy-pinned), a host
|
||||
that has never answered is the `unreachable` stub whose every count is `null`, and a 1.1.0 agent
|
||||
is `supported: false` plus its one `why` sentence — three arms, no red among them (invariant 3).
|
||||
`/history` serves the `gateway.in_conns@<port>` and `gateway.out_conns@<host>:<port>` gauges —
|
||||
the port in the outbound name because two services on one host are two facts, and only ever for an
|
||||
address endpoint — with `hosts=`
|
||||
endpoint whose address is known: one the config wrote, or (since phase 32 task 03) one the gateway
|
||||
resolved the name to — with `hosts=`
|
||||
defaulting to the ingress alone, unknown names answering as empty series, and no dial either way
|
||||
because history reads only SQLite.
|
||||
* `web/` — six static dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`, `/gateway`) plus
|
||||
@@ -1108,9 +1187,10 @@ more. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
|
||||
Above the map, one **filter box** finds a domain among 33 routes (`stackexpected`, `brain`, `4443`,
|
||||
`minecraft`) because the rows are in config order and the question is always about one name. It hides
|
||||
what the payload already sent and asks for **nothing**: no request, no re-derivation, no new route.
|
||||
It matches the strings the config wrote — the inbound word(s), the endpoint, the port, the haproxy
|
||||
`server` name, and a trunk's own domains and client addresses — case-insensitively, so a label CSS
|
||||
ellipsised still matches its whole text. A trunk that matches keeps its **whole fan-out**: typing
|
||||
It matches the strings the payload carries — the inbound word(s), the endpoint, the port, the haproxy
|
||||
`server` name, the **addresses the gateway resolved** for a name target, and a trunk's own domains
|
||||
and client addresses — case-insensitively, so a label CSS ellipsised still matches its whole text and
|
||||
typing `10.1.2.64` finds the route whose target that address belongs to. A trunk that matches keeps its **whole fan-out**: typing
|
||||
`4443` asks for the port, and every route hanging off it is the answer. `Escape` clears it, a chip
|
||||
beside the field counts `N of M routes`, and a query that matches nothing leaves both panels standing
|
||||
with `no route matches "…"` in each. What it never touches is the header's four tallies, the warnings
|
||||
@@ -1120,12 +1200,17 @@ more. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
|
||||
than inventing a zone no proxy declared.
|
||||
**Nothing on the page
|
||||
is coloured by a verdict** — a busy ingress is the job, so `severity.py` rates no `gateway.*` metric
|
||||
and no number gets a green/amber/red tint. A line to an endpoint the config wrote as a **name**
|
||||
prints `—`, never `0`: counting it would need a DNS lookup and the join never asks one, and the row's
|
||||
tooltip says which of the two facts it is looking at. Expect that on most of this ingress's routes —
|
||||
31 of its 33 are written as domains — and read it as *drawn, not counted* rather than as a fault; the
|
||||
footer says the same thing once, in plain terms, because a dash per row needs one explanation and not
|
||||
thirty-two. The header's four tallies are `ports`, `connections`, `live` and
|
||||
and no number gets a green/amber/red tint. A line prints `—`, never `0`, and its tooltip and its
|
||||
accessible name say **which** unknown it is: *the gateway could not resolve this name in this
|
||||
reading* when nothing is known for the endpoint, or *the socket table did not answer this reading*
|
||||
when an address is known and nothing was measured — two different things to go and fix. A resolved
|
||||
endpoint with nothing open is neither of them: it prints `0`, because that is a measurement. A
|
||||
counted line carries its provenance on hover (`… · address from libc, resolved 3m`), and a line the
|
||||
config wrote as an address claims no resolution at all. A dash on a domain-named route now means the
|
||||
gateway had **no record** for that name in this reading (`nxdomain`/`error`, or no `resolver` section
|
||||
at all on a pre-1.3.0 agent) rather than "this page cannot count names": the ingress here runs
|
||||
**1.3.0**, so all 33 of its declared routes carry a number. The
|
||||
footer says the same thing once, in plain terms, and points at the hover. The header's four tallies are `ports`, `connections`, `live` and
|
||||
`warnings`, every one of them the payload's own figure. A `warnings` count above zero raises a panel
|
||||
above the map with the config-integrity rows (`a backend no frontend routes to`, a `proxy_pass` with
|
||||
no `upstream` behind it, an SNI entry that reaches no address) in the overview's alert vocabulary;
|
||||
@@ -1142,7 +1227,8 @@ more. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
|
||||
documented here as a thing you can have: name-based liveness needs the agent to report resolved
|
||||
targets (a future agent version), and switching a route needs a second gated write path in front of
|
||||
the proxy configs, which is a change to this app's read-only promise rather than a feature. What runs
|
||||
today is **agent 1.2.0**, no name resolution, and no switching.
|
||||
today is **agent 1.3.0** — name resolution on the gateway, counts on every declared route, and no
|
||||
switching.
|
||||
The four event feeds (`/system`, `/disks`, `/llm`, `/ups`) are a **paged window** over the `event`
|
||||
table rather than a printout of it: the list is a bounded box that scrolls, 50 events to a
|
||||
page, with `1–50 of 200` between `‹ prev` and `next ›` (disabled at the ends) underneath.
|
||||
@@ -1326,7 +1412,7 @@ plus its included documents, and the two `/proc/net/tcp{,6}` captures — run th
|
||||
`host_agent` parsers, so the test compares the join against the agent's own document rather than
|
||||
against a copy of its field names. The assertions are the join's two rules and the limits around
|
||||
them: the graph carries exactly the ports, `server_name`s and targets those documents declare (all
|
||||
32 routes on this capture, as a **set**, so a missing or invented route fails by name); a socket is
|
||||
33 routes on this capture, as a **set**, so a missing or invented route fails by name); a socket is
|
||||
inbound by its local port and outbound only when its remote `(address, port)` equals a target the
|
||||
config wrote as an address; **every other socket is ignored** — `established`, `live_in` and
|
||||
`live_out` are all printed and deliberately never reconciled, because a connection to a name, or on
|
||||
@@ -1337,6 +1423,13 @@ stands for "not counted"; and **no join ever asks a resolver** — `socket.getad
|
||||
fails the test, because deciding "already an address, or a name?" is the only question that module
|
||||
may ask, and a DNS lookup in the poll loop would be rule 2 with a round trip attached.
|
||||
|
||||
The seventh capture, `gateway_resolver.json`, is in that suite's reach too: it is the agent's own
|
||||
`resolver` document from the real gateway, so `tests/test_agent.py` proves it has the section's shape
|
||||
(one entry per name `target_names()` derives, `via: "libc"`, the config's own `resolver` line, no
|
||||
globally routable address) and `tests/test_gateway_source.py` proves it reaches every route the
|
||||
configs write as a **name** and shares addresses with the recorded socket table — the capability the
|
||||
join needs, pinned before the join exists to use it.
|
||||
|
||||
`tests/rate_window_contract.py` is the same device for one number rather than a field table: the Δ
|
||||
window each source measures its rates over — `RATE_WINDOW_SECONDS` for the host agents,
|
||||
`LLM_INTERVAL_SECONDS` for the engines, one estimator per measuring cadence (phase 23). It holds
|
||||
|
||||
+523
-19
@@ -67,6 +67,28 @@ Design rules
|
||||
answering is `available: null` with one `sockets:` line and an otherwise intact snapshot,
|
||||
and `--no-proxy` removes it with the rest of the family, because a box with no proxy has
|
||||
nothing to watch from either side.
|
||||
* **Names become addresses where they are actually dialled.** The `resolver` section takes every
|
||||
`(name, port)` the two parsed configs write as a target — haproxy `server` lines, nginx
|
||||
`upstream` members, `proxy_pass` targets and `map` entry targets, and **nothing else, ever**
|
||||
(not a name from `/proc`, not a client address) — and asks libc what each one is:
|
||||
`getaddrinfo` over `SOCK_STREAM`, which is the resolution haproxy's own libc path performs *on
|
||||
this box*. That is the point — the dashboard resolving the same name through its own
|
||||
`resolv.conf` would be a different vantage answering a different question — so `via: "libc"`
|
||||
and the nginx config's own `resolver` line (`config_directive`) are both published and the
|
||||
dashboard resolves nothing. An address literal in a config is `status: "literal"` with no
|
||||
lookup (it is already an address, and dialling it to be told so is a round trip); a name that
|
||||
does not answer is `nxdomain`/`error` with `addresses: []`, never a stale guess and never a 0.
|
||||
The pass is SMART's shape for SMART's reason: at most `RESOLVE_MAX_NAMES` lookups inside
|
||||
`RESOLVE_PASS_BUDGET_SECONDS` of wall clock, cached `RESOLVE_CACHE_SECONDS`, the **first** pass
|
||||
inline and every later expiry refreshed on one single-flight background thread while the
|
||||
request is served the previous result — a black-holing resolver must never blow the dashboard's
|
||||
`HOST_TIMEOUT_SECONDS` and blink the *system* page over an add-on it does not draw. Each entry
|
||||
carries the time **it** was answered, so a cache-served entry cannot pretend to be current.
|
||||
`available` is the family's three-way answer (`true` a pass ran, `false` nothing to resolve
|
||||
because no proxy config was read — one `resolver:` line, saying **not found** when the files
|
||||
are simply absent — `null` a pass that could not run), and `--no-resolve` removes the key in
|
||||
silence the way `--no-smart` does; `--no-proxy` removes it too, because a box that collects no
|
||||
proxy has no declared names to look up.
|
||||
* **The state** is a small pid->cpu-time cache so `processes.top_cpu` can answer a
|
||||
question /proc cannot (CPU needs two samples), bounded and cleared when stale, plus
|
||||
the short-TTL SMART result cache above.
|
||||
@@ -158,6 +180,31 @@ SCHEMA (GET /snapshot)
|
||||
# --no-proxy already reports "did not look" by removing the key. No
|
||||
# config join: a socket does not know its domain, so the gateway page
|
||||
# joins these ports to `haproxy` and `nginx` itself.
|
||||
resolver {available: true|false|null, via: "libc", config_directive: str|null,
|
||||
ts: float|null, entries: [{name, port, status: ok|nxdomain|error|literal,
|
||||
addresses: [str], ts: float}], errors: [str]}
|
||||
# one libc lookup per (name, port) the two sections above write as a target,
|
||||
# performed here because this is the box the proxy dials from; additive
|
||||
# (phase 32) and absent entirely with --no-resolve or --no-proxy. `via` is
|
||||
# how the addresses were obtained (getaddrinfo — haproxy's libc path) and
|
||||
# `config_directive` is the nginx config's own `resolver` line kept verbatim
|
||||
# (`"10.1.0.1"` here), so the two vantages read side by side and a
|
||||
# divergence is the operator's finding rather than a second resolver here.
|
||||
# `ts` is the wall clock of the pass that produced the entries while each
|
||||
# entry carries the time *it* was answered, because an expired cache is
|
||||
# served as it was while a background thread refreshes it: never two passes
|
||||
# per snapshot, never a request that waits on DNS past the first one.
|
||||
# `addresses` is the unique answer sorted v4-ascending then v6-ascending so
|
||||
# two polls of a healthy gateway are byte-identical; `status: literal` means
|
||||
# the config already wrote an address and no lookup was made, and
|
||||
# `nxdomain`/`error` mean "no address right now" with `addresses: []` — the
|
||||
# page draws those routes with no count, never with a 0. `available` is
|
||||
# false only when there was nothing to resolve because no proxy config was
|
||||
# read (one `resolver:` line, saying **not found** when the files are simply
|
||||
# absent), and null for a pass that could not run; configs that were read
|
||||
# and name no target are true with `entries: []`. The lookups are bounded
|
||||
# (RESOLVE_MAX_NAMES, RESOLVE_PASS_BUDGET_SECONDS) and the excess says so in
|
||||
# one line, so no resolver however broken holds /snapshot past its timeout.
|
||||
errors [str] reads that failed; non-empty means the snapshot is partial
|
||||
|
||||
Deployment: see deploy/install-agent.sh (systemd unit, binds 127.0.0.1 unless you pass
|
||||
@@ -185,12 +232,12 @@ import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from collections.abc import Callable, Iterable, Mapping, Sequence
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from typing import Any, cast
|
||||
|
||||
AGENT_VERSION = "1.2.0"
|
||||
AGENT_VERSION = "1.3.0"
|
||||
SCHEMA_VERSION = 1
|
||||
PAGE_SIZE = 0 # resolved from sysconf at runtime
|
||||
|
||||
@@ -335,6 +382,46 @@ PROC_NET_MAX_LINES = 4096
|
||||
PROC_NET_STATE_LISTEN = "0A"
|
||||
PROC_NET_STATE_ESTABLISHED = "01"
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# name resolution (the additive `resolver` section)
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
#: How many declared targets one pass looks up. A gateway's config-declared targets are a few
|
||||
#: dozen with room to grow, and the bound is what stops a config that grew into the hundreds (or
|
||||
#: a map written by someone who liked wildcards) from turning a snapshot into a DNS storm. The
|
||||
#: excess is reported in one `errors` line naming the count — the nginx include-bound and SMART
|
||||
#: pass-budget shape: say what was not covered instead of serving a shorter list silently.
|
||||
RESOLVE_MAX_NAMES = 64
|
||||
|
||||
#: How long a resolved address set stays fresh. A LAN target moves when the operator moves it,
|
||||
#: not every poll, and this section exists to be *cheap*: the dashboard polls the agent every
|
||||
#: 10 s, and a lookup per name per poll would make the agent the noisiest DNS client on the
|
||||
#: network. Anything older than this is refreshed on a background thread, never on the request.
|
||||
RESOLVE_CACHE_SECONDS = 300.0
|
||||
|
||||
#: The wall-clock ceiling on one inline pass. `socket.getaddrinfo` takes no timeout argument, so
|
||||
#: the only protection against a black-holing resolver is stopping between lookups — and the
|
||||
#: reason that matters is the same one SMART learned: blowing `HOST_TIMEOUT_SECONDS` here turns
|
||||
#: the **system** page red over an add-on it never draws (AGENTS.md invariant 3/4). The first
|
||||
#: pass runs inline (a fresh agent must answer with real addresses), every later one runs on a
|
||||
#: background thread, so this bound is the worst cost of a single request.
|
||||
RESOLVE_PASS_BUDGET_SECONDS = 3.0
|
||||
|
||||
#: How the addresses were obtained, published rather than implied: `libc` means
|
||||
#: `getaddrinfo()`, which is the resolution haproxy's own `init-addr last,libc,none` performs on
|
||||
#: this box. It is not the `resolver` directive's answer (nginx queries that server directly),
|
||||
#: and the payload says both so an operator can see a divergence — this module does not hand-roll
|
||||
#: UDP DNS queries to name it a second time.
|
||||
RESOLVE_VIA = "libc"
|
||||
|
||||
#: The `getaddrinfo` return codes that mean "this name has no address right now" as opposed to
|
||||
#: "the lookup could not be performed". `EAI_NODATA` is what glibc calls a no-data answer and
|
||||
#: CPython does not export it (it is deprecated upstream), so the value is spelled out beside a
|
||||
#: `getattr` rather than guessed at: a temporary DNS failure (`EAI_AGAIN`) is deliberately **not**
|
||||
#: in this set, because "no answer" and "no server" are different operator problems.
|
||||
_EAI_NODATA = getattr(socket, "EAI_NODATA", -5) # CPython does not export it
|
||||
RESOLVE_NXDOMAIN_CODES = frozenset({socket.EAI_NONAME, _EAI_NODATA})
|
||||
|
||||
|
||||
# pci slot -> human model, filled on first look (lspci output cannot change at runtime).
|
||||
_GPU_MODEL_CACHE: dict[str, str | None] = {}
|
||||
@@ -404,7 +491,7 @@ class Collector:
|
||||
|
||||
def __init__(self, root: str | os.PathLike[str] = "/", *, top_processes: int = 5,
|
||||
net_all: bool = False, smartctl: str = "smartctl",
|
||||
smart: bool = True, proxy: bool = True,
|
||||
smart: bool = True, proxy: bool = True, resolve: bool = True,
|
||||
haproxy_config: str = HAPROXY_CONFIG_PATH,
|
||||
nginx_config: str = NGINX_CONFIG_PATH) -> None:
|
||||
self.root = Path(root)
|
||||
@@ -420,6 +507,22 @@ class Collector:
|
||||
self.proxy_enabled = proxy
|
||||
self.haproxy_config = haproxy_config
|
||||
self.nginx_config = nginx_config
|
||||
# Name resolution: its own flag, because it is a different kind of work from reading a
|
||||
# file — it leaves the host. It is meaningless without the proxy sections (they are the
|
||||
# only source of names, decision 3), so `--no-proxy` removes it as well as `--no-resolve`.
|
||||
self.resolve_enabled = resolve
|
||||
self.resolve_max_names = RESOLVE_MAX_NAMES
|
||||
self.resolve_cache_seconds = RESOLVE_CACHE_SECONDS
|
||||
self.resolve_pass_budget_seconds = RESOLVE_PASS_BUDGET_SECONDS
|
||||
self._resolver_lock = threading.Lock() # a pass may run while a handler serves
|
||||
# Entries keyed by `resolver_key(name, port)`; the timestamp is the monotonic age of the
|
||||
# pass that wrote them, `_resolver_pass_ts` the wall clock the section publishes (an entry
|
||||
# keeps the time *it* was answered, so a served-from-cache entry cannot look current).
|
||||
self._resolver_cache: dict[str, dict[str, Any]] = {}
|
||||
self._resolver_cache_ts: float | None = None
|
||||
self._resolver_pass_ts: float | None = None
|
||||
self._resolver_errors: list[str] = []
|
||||
self._resolver_refreshing = False # at most one background refresh at a time
|
||||
self.smart_timeout_seconds = SMARTCTL_TIMEOUT_SECONDS
|
||||
self.smart_budget_seconds = SMARTCTL_BUDGET_SECONDS
|
||||
self.smart_cache_seconds = SMARTCTL_CACHE_SECONDS
|
||||
@@ -917,8 +1020,19 @@ class Collector:
|
||||
|
||||
# -- proxy configs (nginx) ------------------------------------------------ #
|
||||
def nginx(self, errors: list[str]) -> dict[str, Any]:
|
||||
"""The `nginx` section on its own — `nginx_document()`'s first half."""
|
||||
return self.nginx_document(errors)[0]
|
||||
|
||||
def nginx_document(self, errors: list[str]) -> tuple[dict[str, Any], str | None]:
|
||||
"""Read and parse the host's nginx configs — never raises, never fails the snapshot.
|
||||
|
||||
Returns `(section, resolver directive)`. The second half is the only reason this method
|
||||
is not the section itself: the `resolver` section prints nginx's own `resolver` line next
|
||||
to `via: "libc"` so an operator can see the two vantages disagree, and taking it from the
|
||||
text this pass already read is what keeps that free (no second file read per snapshot).
|
||||
It deliberately does **not** go in the section: the `nginx` document stays byte-stable,
|
||||
which is what makes this phase's additivity control a strict top-level key comparison.
|
||||
|
||||
The same three-way answer as `haproxy()`, for the other half of the gateway's ingress:
|
||||
`true` the main config was read and understood, `false` there is no config at this path
|
||||
(this host runs no nginx — one `nginx: ` line saying the file is **not found**), `null`
|
||||
@@ -944,10 +1058,10 @@ class Collector:
|
||||
section["available"] = False
|
||||
errors.append(f"nginx: {config} not found — nginx is not installed here, or its "
|
||||
"config lives elsewhere (--nginx-config); --no-proxy opts out")
|
||||
return section
|
||||
return section, None
|
||||
except (OSError, ValueError) as exc:
|
||||
errors.append(f"nginx: {config} is there but could not be read: {exc}")
|
||||
return section
|
||||
return section, None
|
||||
|
||||
# The two extra return values are already spoken about in `errors`; the section itself
|
||||
# carries no counters, because "which files were read" is what `files` is for.
|
||||
@@ -959,11 +1073,11 @@ class Collector:
|
||||
f"parsed: {parsed['unparsed'][0]}")
|
||||
if parsed["error"] is not None:
|
||||
errors.append(f"nginx: {config} is there but could not be parsed: {parsed['error']}")
|
||||
return section
|
||||
return section, None
|
||||
if not parsed["recognised"]:
|
||||
errors.append(f"nginx: {config} holds no nginx context (http, stream, server, "
|
||||
"upstream, map) — not an nginx config")
|
||||
return section
|
||||
return section, None
|
||||
section["available"] = True
|
||||
# The main config first: `files` is the answer to "where do these servers come from",
|
||||
# and an operator comparing it with `nginx -t` should see the file nginx was started on.
|
||||
@@ -971,7 +1085,7 @@ class Collector:
|
||||
section["servers"] = parsed["servers"]
|
||||
section["upstreams"] = parsed["upstreams"]
|
||||
section["maps"] = parsed["maps"]
|
||||
return section
|
||||
return section, cast("str | None", parsed["resolver"])
|
||||
|
||||
def nginx_include_files(self, include_args: list[str],
|
||||
errors: list[str]) -> tuple[dict[str, str], int, list[str]]:
|
||||
@@ -1107,6 +1221,210 @@ class Collector:
|
||||
section["available"] = True if answered else None
|
||||
return section
|
||||
|
||||
# -- name resolution ------------------------------------------------------ #
|
||||
def resolver(self, haproxy: Mapping[str, Any], nginx: Mapping[str, Any],
|
||||
config_directive: str | None, errors: list[str]) -> dict[str, Any]:
|
||||
"""Resolve the names the two proxy configs declare. Never raises, never fails a snapshot.
|
||||
|
||||
The point is that a route written as `server mc1 minecraft.reeselink.com:25565` ends at an
|
||||
address, and the kernel's socket table speaks addresses: without this section the
|
||||
dashboard can only count the connections whose target the config wrote *as an address*,
|
||||
which on this ingress is 2 routes out of 33. The lookup happens here, on the box that
|
||||
opens the connections, because that is the vantage the proxy itself resolves from; the
|
||||
dashboard resolving the name through its own `resolv.conf` would answer a different
|
||||
question (and `sources/gateway.py` is spied to fail if it ever tries).
|
||||
|
||||
`available` is the family's three-way answer, with **one** `resolver:` line per non-`true`
|
||||
arm: `true` a pass ran, `false` there was nothing to resolve because neither proxy section
|
||||
was read (the line says *not found* when the files simply are not here, which is the
|
||||
wording the dashboard reads), `null` a pass that could not run. Configs that were read and
|
||||
name no target are `true` with `entries: []` — that is absence, not a fault.
|
||||
|
||||
The cache is SMART's idiom, and for the SMART reason: the first pass runs inline so a
|
||||
freshly started agent answers with real addresses, and every later expiry is refreshed on
|
||||
a single-flight background thread while the request is served the previous result. A
|
||||
resolver that has started black-holing must not cost the *system* poll its timeout over an
|
||||
add-on that page never draws (AGENTS.md invariant 3/4).
|
||||
"""
|
||||
targets = target_names(haproxy, nginx)
|
||||
if not targets and not _section_answered(haproxy) and not _section_answered(nginx):
|
||||
# Nothing was read, so there is nothing to resolve. The two config sections have
|
||||
# already said why in their own lines; this one says the resolver's own state, once.
|
||||
line = resolver_not_found_line(haproxy, nginx)
|
||||
errors.append(line)
|
||||
return {"available": False, "via": RESOLVE_VIA, "config_directive": config_directive,
|
||||
"ts": None, "entries": [], "errors": [line]}
|
||||
now = time.monotonic()
|
||||
with self._resolver_lock:
|
||||
stamp = self._resolver_cache_ts
|
||||
if stamp is not None:
|
||||
if now - stamp >= self.resolve_cache_seconds and not self._resolver_refreshing:
|
||||
self._resolver_refreshing = True
|
||||
threading.Thread(target=self._refresh_resolver, args=(list(targets),),
|
||||
name="resolver-refresh", daemon=True).start()
|
||||
# Served from cache either way — the fresh-cache case and the "a refresh is
|
||||
# already running" case — because waiting here is what this cache exists to avoid.
|
||||
return self._resolver_section(self._resolver_cache, self._resolver_pass_ts,
|
||||
self._resolver_errors, config_directive, errors)
|
||||
entries, pass_errors = self.resolver_pass(targets)
|
||||
return self._resolver_section(entries, self._resolver_pass_ts, pass_errors,
|
||||
config_directive, errors)
|
||||
|
||||
def _resolver_section(self, entries: Mapping[str, dict[str, Any]], pass_ts: float | None,
|
||||
pass_errors: Sequence[str], config_directive: str | None,
|
||||
errors: list[str]) -> dict[str, Any]:
|
||||
"""The section out of one pass's (or one cache's) result — the shape, in one place.
|
||||
|
||||
The lines go to **both** readers, because there are two: the snapshot's shared `errors`
|
||||
list, which is what `collectors.ADDON_ERROR_PREFIXES` filters to decide the *system* page
|
||||
is not degraded, and the section's own `errors`, which is the gateway page's sentence
|
||||
beside the entries they belong to. Same strings, one append each.
|
||||
"""
|
||||
section: dict[str, Any] = {"available": True, "via": RESOLVE_VIA,
|
||||
"config_directive": config_directive, "ts": pass_ts,
|
||||
"entries": [dict(entry) for entry in entries.values()],
|
||||
"errors": []}
|
||||
for line in pass_errors:
|
||||
errors.append(line)
|
||||
section["errors"].append(line)
|
||||
return section
|
||||
|
||||
def _refresh_resolver(self, targets: list[tuple[str, int | None]]) -> None:
|
||||
"""The expired-cache pass, off the request path (see `resolver()`).
|
||||
|
||||
A failure keeps the previous entries **and** re-stamps the cache: the snapshot keeps
|
||||
answering with the last truth it had, and the retry comes on the next expiry instead of
|
||||
every poll against a resolver that just failed. The line is replaced rather than stacked,
|
||||
because a refresh that fails every `resolve_cache_seconds` for a week is one fact, not
|
||||
2000 — the repeated-failure rule the poller applies to its own events.
|
||||
"""
|
||||
try:
|
||||
self.resolver_pass(targets)
|
||||
except Exception as exc: # noqa: BLE001 - the thread must always clear its own flag
|
||||
with self._resolver_lock:
|
||||
line = (f"resolver: background refresh failed: {type(exc).__name__}: {exc}")
|
||||
self._resolver_errors = [x for x in self._resolver_errors
|
||||
if not x.startswith("resolver: background refresh failed")]
|
||||
self._resolver_errors = [*self._resolver_errors, line]
|
||||
self._resolver_cache_ts = time.monotonic()
|
||||
finally:
|
||||
with self._resolver_lock:
|
||||
self._resolver_refreshing = False
|
||||
|
||||
def resolver_pass(self, targets: Sequence[tuple[str, int | None]]
|
||||
) -> tuple[dict[str, dict[str, Any]], list[str]]:
|
||||
"""One bounded libc lookup per declared target. Never raises out of this function.
|
||||
|
||||
Entries come back keyed by `resolver_key`, in target order, merged over the previous
|
||||
cache so a pass that stopped at the budget keeps the entries it did not reach (each with
|
||||
**its own** `ts`, so a stale address is visibly stale rather than freshly wrong). A name
|
||||
that never got looked up at all and has no previous answer simply has no entry: the line
|
||||
says how many were reached, and inventing an `error` status for a lookup that was never
|
||||
attempted would be rule 2 with a status field on it.
|
||||
"""
|
||||
errors: list[str] = []
|
||||
fresh: dict[str, dict[str, Any]] = {}
|
||||
deadline = time.monotonic() + self.resolve_pass_budget_seconds
|
||||
need_lookup = [pair for pair in targets if address_family(pair[0]) is None]
|
||||
over_bound = len(need_lookup) - self.resolve_max_names
|
||||
looked_up = 0
|
||||
stopped = False
|
||||
for name, port in targets:
|
||||
# Literals first, and never counted against either bound: reporting one is a fact and
|
||||
# dialling it would be a round trip to be told what the config already wrote, so a
|
||||
# truncated pass must not be the reason an address the config spells is missing.
|
||||
if address_family(name) is None:
|
||||
continue
|
||||
fresh[resolver_key(name, port)] = {"name": name, "port": port, "status": "literal",
|
||||
"addresses": [name], "ts": time.time()}
|
||||
for name, port in need_lookup:
|
||||
if looked_up >= self.resolve_max_names:
|
||||
continue # one line for the whole excess, after the loop
|
||||
fresh[resolver_key(name, port)] = self._resolve_target(name, port)
|
||||
looked_up += 1
|
||||
if time.monotonic() >= deadline:
|
||||
stopped = True
|
||||
break
|
||||
if stopped:
|
||||
errors.append(f"resolver: pass budget {self.resolve_pass_budget_seconds:.1f}s "
|
||||
f"exhausted, {looked_up} of {len(need_lookup)} targets looked up")
|
||||
if over_bound > 0:
|
||||
errors.append(f"resolver: {over_bound} of {len(need_lookup)} targets are beyond the "
|
||||
f"{self.resolve_max_names}-name bound and were not looked up — the "
|
||||
"section covers what it could resolve")
|
||||
entries = self._store_resolver_cache(targets, fresh, errors)
|
||||
return entries, errors
|
||||
|
||||
def _store_resolver_cache(self, targets: Sequence[tuple[str, int | None]],
|
||||
fresh: Mapping[str, dict[str, Any]], errors: Sequence[str]
|
||||
) -> dict[str, dict[str, Any]]:
|
||||
"""Merge this pass into the cache in target order, and stamp it. Returns the entries.
|
||||
|
||||
Order is not cosmetics: `entries` is a list a page renders, so the same config must draw
|
||||
in the same order on every poll. Names the config dropped leave the cache with it — they
|
||||
are no longer declared targets, not targets that stopped resolving.
|
||||
"""
|
||||
merged: dict[str, dict[str, Any]] = {}
|
||||
with self._resolver_lock:
|
||||
previous = self._resolver_cache
|
||||
for name, port in targets:
|
||||
key = resolver_key(name, port)
|
||||
entry = fresh.get(key) or previous.get(key)
|
||||
if entry is not None:
|
||||
merged[key] = entry
|
||||
self._resolver_cache = merged
|
||||
self._resolver_cache_ts = time.monotonic()
|
||||
self._resolver_pass_ts = time.time()
|
||||
# The pass's own lines replace the previous ones: a cache replay has to say the same
|
||||
# thing the pass that produced it said (SMART's cache does the same), and a line from
|
||||
# an older pass — a budget hit that has since stopped happening — would be a stale
|
||||
# complaint about a reading that is now complete.
|
||||
self._resolver_errors = list(errors)
|
||||
return dict(merged)
|
||||
|
||||
def _resolve_target(self, name: str, port: int | None) -> dict[str, Any]:
|
||||
"""One `getaddrinfo`, and the entry that says what it answered.
|
||||
|
||||
Both families are kept (`getaddrinfo` answers v4 and v6 when the host has both) because
|
||||
the gateway connects over whichever one routing picks and the socket table speaks either.
|
||||
A `gaierror` naming no-name/no-data is `nxdomain` — the name is answered and has no
|
||||
address — and anything else is `error`, because "the server did not answer" and "the name
|
||||
has no record" are different things to fix. Both carry `addresses: []`, never a stale
|
||||
guess and never a `0`. The `ts` is taken here rather than passed in, so an entry says when
|
||||
**it** was answered even in a pass that then stopped at the budget.
|
||||
|
||||
The keyword is `type=`, and that is load-bearing in a way only a real libc can teach:
|
||||
`proto` is the **protocol number** (`IPPROTO_TCP` = 6) while `SOCK_STREAM` = 1 is a socket
|
||||
*type*, so `proto=SOCK_STREAM` asks libc for a stream socket over ICMP and it answers
|
||||
`EAI_SERVICE` for **every** name — a resolver that reports `error` on a healthy gateway,
|
||||
with every route on the page uncounted, in a way no stubbed test can see because a stub
|
||||
takes any keyword. What we want is libc's answer to "where do I connect a TCP socket",
|
||||
which is exactly haproxy's `init-addr last,libc,none` question. Pinned by
|
||||
`tests/test_agent.py` with no stub in sight, against an address libc parses by itself.
|
||||
"""
|
||||
try:
|
||||
infos = socket.getaddrinfo(name, port, type=socket.SOCK_STREAM)
|
||||
except socket.gaierror as exc:
|
||||
code = exc.errno if isinstance(exc.errno, int) else 0
|
||||
return {"name": name, "port": port,
|
||||
"status": "nxdomain" if code in RESOLVE_NXDOMAIN_CODES else "error",
|
||||
"addresses": [], "ts": time.time()}
|
||||
except OSError as exc: # a resolver that fails as an I/O error, not as a DNS one
|
||||
return {"name": name, "port": port, "status": "error", "addresses": [],
|
||||
"detail": str(exc), "ts": time.time()}
|
||||
except Exception as exc: # noqa: BLE001 - the promise is that this function never raises
|
||||
# `getaddrinfo` is the one call in this agent that leaves the host, and whatever it
|
||||
# does — a platform-specific `ValueError` for a name the C library cannot take, a
|
||||
# stub in a test — the answer is one entry that says "error" beside the targets that
|
||||
# answered, never a half-snapshot and never the *system* page going red over an
|
||||
# add-on it does not draw (AGENTS.md invariant 4).
|
||||
return {"name": name, "port": port, "status": "error", "addresses": [],
|
||||
"detail": f"{type(exc).__name__}: {exc}", "ts": time.time()}
|
||||
addresses = sort_resolver_addresses(str(info[4][0]) for info in infos)
|
||||
# An empty answer list is libc saying the name resolved to nothing: no-data, not a fault.
|
||||
return {"name": name, "port": port, "status": "ok" if addresses else "nxdomain",
|
||||
"addresses": addresses, "ts": time.time()}
|
||||
|
||||
# -- network ------------------------------------------------------------- #
|
||||
def network(self) -> dict[str, Any]:
|
||||
interfaces: list[dict[str, Any]] = []
|
||||
@@ -1479,11 +1797,24 @@ class Collector:
|
||||
document["haproxy"] = self._safe(lambda: self.haproxy(errors), "haproxy", errors) \
|
||||
or {"available": None, "config": self.haproxy_config,
|
||||
"frontends": [], "backends": []}
|
||||
document["nginx"] = self._safe(lambda: self.nginx(errors), "nginx", errors) \
|
||||
or {"available": None, "config": self.nginx_config, "files": [],
|
||||
"servers": [], "upstreams": [], "maps": []}
|
||||
nginx_pair = self._safe(lambda: self.nginx_document(errors), "nginx", errors)
|
||||
nginx_section, nginx_resolver = nginx_pair or (
|
||||
{"available": None, "config": self.nginx_config, "files": [],
|
||||
"servers": [], "upstreams": [], "maps": []}, None)
|
||||
document["nginx"] = nginx_section
|
||||
document["sockets"] = self._safe(lambda: self.sockets(errors), "sockets", errors) \
|
||||
or {"available": None, "listens": [], "established": []}
|
||||
# The resolver rides with the family and is removed by both flags: `--no-resolve` is
|
||||
# the operator turning off the one add-on that leaves the host, and `--no-proxy` means
|
||||
# there are no declared names to resolve in the first place (decision 3). Absent, not
|
||||
# empty — "not collected" stays a different fact from "collected and empty".
|
||||
if self.resolve_enabled:
|
||||
document["resolver"] = self._safe(
|
||||
lambda: self.resolver(cast("Mapping[str, Any]", document["haproxy"]),
|
||||
cast("Mapping[str, Any]", nginx_section),
|
||||
nginx_resolver, errors), "resolver", errors) \
|
||||
or {"available": None, "via": RESOLVE_VIA, "config_directive": None,
|
||||
"ts": None, "entries": [], "errors": []}
|
||||
document["errors"] = errors
|
||||
return document
|
||||
|
||||
@@ -2038,13 +2369,15 @@ def parse_nginx_config(main_text: str,
|
||||
`{servers: [{listen: [{addr, port}], server_names: [str], proxy_passes: [{location, scheme,
|
||||
target, variable}]}], upstreams: [{name, servers: [{host, port}]}],
|
||||
maps: [{variable, entries: [{key, target}]}], files: [str], includes: [str], unparsed:
|
||||
[str], recognised: bool, error: str|null}`.
|
||||
[str], recognised: bool, error: str|null, resolver: str|null}`.
|
||||
|
||||
Three of those keys are the ones `Collector.nginx` does not serve, the way
|
||||
Four of those keys are the ones `Collector.nginx` does not serve, the way
|
||||
`parse_haproxy_config`'s `recognised` is not: `error` is why the main text could not be
|
||||
parsed, `unparsed` names an included document that could not, and `includes` is the raw
|
||||
parsed, `unparsed` names an included document that could not, `includes` is the raw
|
||||
argument list (a caller that has no filesystem — every unit test — still gets to see what a
|
||||
config *would* have read). `files` is the included documents that were merged, in order;
|
||||
config *would* have read), and `resolver` is the config's own `resolver` directive's
|
||||
arguments, which the `resolver` section serves instead (`Collector.nginx_document`).
|
||||
`files` is the included documents that were merged, in order;
|
||||
the collector puts the main config in front of it to make the section's `files`.
|
||||
|
||||
A document that fails to parse is dropped and named in `unparsed`; it never drops the others
|
||||
@@ -2053,13 +2386,18 @@ def parse_nginx_config(main_text: str,
|
||||
`error` set, not an exception on a snapshot path.
|
||||
"""
|
||||
out: dict[str, Any] = {"servers": [], "upstreams": [], "maps": [], "files": [],
|
||||
"includes": [], "unparsed": [], "recognised": False, "error": None}
|
||||
"includes": [], "unparsed": [], "recognised": False, "error": None,
|
||||
"resolver": None}
|
||||
try:
|
||||
main = _nginx_statements(main_text)
|
||||
except ValueError as exc:
|
||||
out["error"] = str(exc)
|
||||
return out
|
||||
out["recognised"] = any(statement[1] in NGINX_CONTEXTS for statement in main)
|
||||
# The config's own `resolver` line, for the `resolver` section's `config_directive`. It is
|
||||
# found here rather than served here on purpose: the `nginx` section stays byte-stable, so
|
||||
# this phase's additivity control is a strict top-level key comparison with no exception list.
|
||||
out["resolver"] = _find_resolver_statement(main)
|
||||
_collect_nginx(main, out, root=True)
|
||||
documents = included or {}
|
||||
merged: list[str] = []
|
||||
@@ -2075,6 +2413,8 @@ def parse_nginx_config(main_text: str,
|
||||
out["unparsed"].append(path)
|
||||
continue
|
||||
merged.append(path)
|
||||
if out["resolver"] is None:
|
||||
out["resolver"] = _find_resolver_statement(statements)
|
||||
# root=False: an include line inside an included file is a second level, and this
|
||||
# pass is one level deep by design (see NGINX_INCLUDE_MAX_FILES).
|
||||
_collect_nginx(statements, out, root=False)
|
||||
@@ -2197,6 +2537,165 @@ def socket_truncation_line(paths: list[str]) -> str:
|
||||
"read takes — the sockets past the bound are not in this section")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# name resolution (the additive `resolver` section)
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
def address_family(value: str) -> str | None:
|
||||
"""`v4`, `v6`, or None — is this text already an address?
|
||||
|
||||
`socket.inet_pton` and nothing else: it is the same test the `sockets` section's decoder
|
||||
uses, it needs no second module (the agent's import footprint is pinned by
|
||||
`tests/test_agent.py`), and it is the rule that lets an IP literal in a config be reported
|
||||
as what it is instead of dialled. An interface scope (`fe80::1%eth0`) is not a pure address,
|
||||
so it comes back None and travels through the resolver like any other name — libc decides
|
||||
what a scoped literal means, and it is not this file's job to invent an answer.
|
||||
"""
|
||||
for family, label in ((socket.AF_INET, "v4"), (socket.AF_INET6, "v6")):
|
||||
try:
|
||||
socket.inet_pton(family, value)
|
||||
except (OSError, ValueError):
|
||||
continue
|
||||
return label
|
||||
return None
|
||||
|
||||
|
||||
def resolver_key(name: str, port: int | None) -> str:
|
||||
"""One declared target, as this module's cache key.
|
||||
|
||||
The port belongs in the key because a name on two ports is two targets: this fleet routes
|
||||
`minecraft.reeselink.com` to 25565 and to 25566, two services, and one entry for both would
|
||||
be one answer to a question nobody asked.
|
||||
"""
|
||||
return f"{name}:{port}"
|
||||
|
||||
|
||||
def _resolver_sort_key(value: str) -> tuple[int, bytes]:
|
||||
"""v4 ascending then v6 ascending, with anything undecodable last and stable.
|
||||
|
||||
The order matters for the payload rather than for the lookup: `getaddrinfo` returns whatever
|
||||
order this host's resolver and `/etc/gai.conf` feel like today, so an unsorted set makes two
|
||||
polls of a healthy gateway differ for no reason the operator could act on.
|
||||
"""
|
||||
for family, rank in ((socket.AF_INET, 0), (socket.AF_INET6, 1)):
|
||||
try:
|
||||
return (rank, socket.inet_pton(family, value))
|
||||
except (OSError, ValueError):
|
||||
continue
|
||||
return (2, value.encode("utf-8", errors="replace"))
|
||||
|
||||
|
||||
def sort_resolver_addresses(values: Iterable[str]) -> list[str]:
|
||||
"""Unique addresses in the payload's order: family first, then the address itself."""
|
||||
return sorted(set(values), key=_resolver_sort_key)
|
||||
|
||||
|
||||
def target_names(haproxy: Mapping[str, Any] | None,
|
||||
nginx: Mapping[str, Any] | None) -> list[tuple[str, int | None]]:
|
||||
"""Every `(name, port)` the two **parsed** proxy configs write as a target, no duplicates.
|
||||
|
||||
This is the whole allow-list for the resolver (decision 3 of the phase): haproxy's
|
||||
`backend`/`server` targets first (config order), then nginx's `upstream` members, its
|
||||
server-block `proxy_pass` targets, and its `map` entry targets, deduplicated keeping
|
||||
first-seen order. Nothing else is ever in here — not a name from `/proc/net/tcp`, not a
|
||||
client address, not anything a request supplied — because resolving a name nobody configured
|
||||
is a round trip that produces a fact about the internet instead of a fact about this
|
||||
gateway, and `tests/test_agent.py` pins that the lookups are exactly this list.
|
||||
|
||||
A `proxy_pass $var` contributes nothing (the map decides, and the map's own entries are
|
||||
here); an entry whose target is not an address at all is `target: null` upstream and is
|
||||
skipped here for the same reason the `nginx` section refused to name it.
|
||||
"""
|
||||
targets: list[tuple[str, int | None]] = []
|
||||
seen: set[tuple[str, int | None]] = set()
|
||||
|
||||
def add(host: Any, port: Any) -> None:
|
||||
if not isinstance(host, str) or not host:
|
||||
return
|
||||
pair = (host, port if isinstance(port, int) and not isinstance(port, bool) else None)
|
||||
if pair in seen:
|
||||
return
|
||||
seen.add(pair)
|
||||
targets.append(pair)
|
||||
|
||||
for block in _mapping_list(haproxy, "backends"):
|
||||
for server in _mapping_list(block, "servers"):
|
||||
add(server.get("target"), server.get("port"))
|
||||
for upstream in _mapping_list(nginx, "upstreams"):
|
||||
for member in _mapping_list(upstream, "servers"):
|
||||
add(member.get("host"), member.get("port"))
|
||||
for server in _mapping_list(nginx, "servers"):
|
||||
for hop in _mapping_list(server, "proxy_passes"):
|
||||
target = hop.get("target")
|
||||
if isinstance(target, Mapping):
|
||||
add(target.get("host"), target.get("port"))
|
||||
for block in _mapping_list(nginx, "maps"):
|
||||
for entry in _mapping_list(block, "entries"):
|
||||
target = entry.get("target")
|
||||
if isinstance(target, Mapping):
|
||||
add(target.get("host"), target.get("port"))
|
||||
return targets
|
||||
|
||||
|
||||
def _mapping_list(section: Any, key: str) -> list[Mapping[str, Any]]:
|
||||
"""One list of mappings out of a section that may be absent, null, or the wrong shape.
|
||||
|
||||
The section is built by this file, so anything else means a caller handed this function
|
||||
something it did not build — and the honest answer to that is an empty list, not a
|
||||
`TypeError` on a snapshot path.
|
||||
"""
|
||||
if not isinstance(section, Mapping):
|
||||
return []
|
||||
rows = section.get(key)
|
||||
if not isinstance(rows, list):
|
||||
return []
|
||||
return [row for row in rows if isinstance(row, Mapping)]
|
||||
|
||||
|
||||
def _section_answered(section: Any) -> bool:
|
||||
"""Did this proxy section report that it read its config? `available` is the section's own answer."""
|
||||
return isinstance(section, Mapping) and section.get("available") is True
|
||||
|
||||
|
||||
def resolver_not_found_line(haproxy: Any, nginx: Any) -> str:
|
||||
"""The one `resolver:` line for the arm where there was nothing to resolve.
|
||||
|
||||
It says *not found* when a config is simply absent — the pinned cross-module wording the
|
||||
dashboard reads to answer "is this proxy installed here?" (the `haproxy:`, `nginx:` and
|
||||
`smartctl:` lines share it) — and *unreadable* when the file is there and this process could
|
||||
not read it, because those are two different operator actions. One line either way: the two
|
||||
sections have already reported their own reasons, and a third restatement per snapshot is the
|
||||
ink the empty-state rules exist to avoid.
|
||||
"""
|
||||
reasons = []
|
||||
for name, section in (("haproxy", haproxy), ("nginx", nginx)):
|
||||
available = section.get("available") if isinstance(section, Mapping) else None
|
||||
reasons.append(f"{name} not found" if available is False else f"{name} unreadable")
|
||||
return (f"resolver: nothing to resolve — {' and '.join(reasons)}; --no-resolve opts out")
|
||||
|
||||
|
||||
def _find_resolver_statement(statements: _NginxStatements) -> str | None:
|
||||
"""Depth-first search for one `resolver` directive, in document order.
|
||||
|
||||
A `map` body is not searched: inside one, a bare word is a **key**, so a map with a
|
||||
`resolver` key (a header name, a hostname someone routed on) would otherwise be reported as
|
||||
this box's DNS server.
|
||||
"""
|
||||
for statement in statements:
|
||||
kind, name, args = statement[0], statement[1], statement[2]
|
||||
if kind == "d":
|
||||
if name == "resolver" and args:
|
||||
return " ".join(str(arg) for arg in args)
|
||||
continue
|
||||
if name == "map":
|
||||
continue
|
||||
found = _find_resolver_statement(cast("_NginxStatements", statement[3]))
|
||||
if found is not None:
|
||||
return found
|
||||
return None
|
||||
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# TLS bundle: what the dashboard signs, what the installer places
|
||||
# --------------------------------------------------------------------------- #
|
||||
@@ -2663,8 +3162,12 @@ def main(argv: list[str] | None = None) -> int:
|
||||
help="skip the smartctl pass entirely (disks report smart: null, "
|
||||
"no error entries)")
|
||||
parser.add_argument("--no-proxy", action="store_true",
|
||||
help="skip the proxy add-on entirely (no `haproxy`, `nginx` or "
|
||||
"`sockets` key, no error entries)")
|
||||
help="skip the proxy add-on entirely (no `haproxy`, `nginx`, `sockets` "
|
||||
"or `resolver` key, no error entries)")
|
||||
parser.add_argument("--no-resolve", action="store_true",
|
||||
help="skip the resolver pass (no `resolver` key, no error entries): the "
|
||||
"two config sections and the socket table are still read, only the "
|
||||
"name lookups are not")
|
||||
parser.add_argument("--haproxy-config", default=HAPROXY_CONFIG_PATH,
|
||||
help="haproxy config to read; a missing file is the documented "
|
||||
"degradation (available=false, one error entry), not a failure")
|
||||
@@ -2679,7 +3182,8 @@ def main(argv: list[str] | None = None) -> int:
|
||||
|
||||
collector = Collector(args.root, top_processes=args.top, net_all=args.net_all,
|
||||
smartctl=args.smartctl, smart=not args.no_smart,
|
||||
proxy=not args.no_proxy, haproxy_config=args.haproxy_config,
|
||||
proxy=not args.no_proxy, resolve=not args.no_resolve,
|
||||
haproxy_config=args.haproxy_config,
|
||||
nginx_config=args.nginx_config)
|
||||
if args.once:
|
||||
print(json.dumps({"ok": True, **collector.snapshot()}, indent=2, default=str))
|
||||
|
||||
+1
-1
@@ -26,7 +26,7 @@ dependencies = [
|
||||
dev = [
|
||||
"pytest>=9.1.1",
|
||||
"pytest-cov>=7.1.0",
|
||||
"pytest-asyncio>=1.4.0",
|
||||
"pytest-asyncio>=1.4.1",
|
||||
"ruff>=0.17.1",
|
||||
"pyright>=1.1.414",
|
||||
"debugpy>=1.8.22",
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# Re-record the gateway's proxy configs and kernel socket tables into tests/fixtures/.
|
||||
# Re-record the gateway's proxy configs, kernel socket tables and libc resolutions into
|
||||
# tests/fixtures/.
|
||||
#
|
||||
# ./scripts/capture-gateway-config.sh # dials the `gateway-ops` ssh alias
|
||||
# GATEWAY_SSH=lab-gw ./scripts/capture-gateway-config.sh
|
||||
@@ -12,13 +13,24 @@
|
||||
# gateway_nginx_stream_d_<n>.conf cat /etc/nginx/stream.d/*.conf
|
||||
# gateway_proc_net_tcp.txt cat /proc/net/tcp
|
||||
# gateway_proc_net_tcp6.txt cat /proc/net/tcp6
|
||||
# gateway_resolver.json getent ahostsv4/ahostsv6, once per declared target
|
||||
#
|
||||
# These documents are the ground truth behind `agent/host_agent.py`'s `haproxy`, `nginx` and
|
||||
# `sockets` sections (phase 28) the way `tests/fixtures/nut_<ups>.vars` pins the UPS wire and
|
||||
# `smartctl_driveripper_<dev>.txt` pins the drives. The include globs are read **out of the
|
||||
# captured main config**, so a gateway that moves its vhosts to a third directory grows a third
|
||||
# fixture on the next run instead of keeping a fixture set that quietly stopped being the tree
|
||||
# the agent merges.
|
||||
# These documents are the ground truth behind `agent/host_agent.py`'s `haproxy`, `nginx`,
|
||||
# `sockets` and `resolver` sections (phases 28 and 32) the way `tests/fixtures/nut_<ups>.vars`
|
||||
# pins the UPS wire and `smartctl_driveripper_<dev>.txt` pins the drives. The include globs are
|
||||
# read **out of the captured main config**, so a gateway that moves its vhosts to a third
|
||||
# directory grows a third fixture on the next run instead of keeping a fixture set that quietly
|
||||
# stopped being the tree the agent merges.
|
||||
#
|
||||
# The resolver document is the one fixture whose *name list* comes from the repo rather than from
|
||||
# the remote: the targets are `agent.host_agent.target_names()` over the configs captured above,
|
||||
# so the capture can never ask for more names than the agent will resolve, and a fixture naming a
|
||||
# target the parser no longer derives is a fixture that fails a test. Each name is then read on
|
||||
# the gateway itself — `getent ahostsv4 <name>` and `getent ahostsv6 <name>`, the same libc
|
||||
# resolution `via: "libc"` claims and the one haproxy's `init-addr last,libc,none` performs — so
|
||||
# the addresses in the fixture are the answers the *routing* box gives, not this laptop's. A
|
||||
# `config_directive` read out of the already captured nginx text costs no round trip, so the
|
||||
# fixture's provenance line and the agent's own answer come from the same bytes.
|
||||
#
|
||||
# Four rules, three of them `scripts/capture-fixtures.sh`'s:
|
||||
#
|
||||
@@ -27,7 +39,12 @@
|
||||
# The header is comment-shaped because `#` is a comment in haproxy's config and in nginx's,
|
||||
# and a `/proc` line whose 4th field is not a kernel state is skipped by the socket parser —
|
||||
# provenance that costs no parser anything, which `tests/test_agent.py` pins by parsing the
|
||||
# fixtures exactly as they sit on disk.
|
||||
# fixtures exactly as they sit on disk. `gateway_resolver.json` is the one fixture that cannot
|
||||
# carry it: it is JSON the tests `json.load`, and a `#` line is not JSON, so its provenance is
|
||||
# this block plus the `via`/`config_directive`/`ts` keys inside it (the same reason
|
||||
# `slots_gufo.json` and `agent_<host>.json` carry none). Its `ts` is a fixed constant, not the
|
||||
# capture's clock: the value is a *shape* (an entry carries the time it was answered) and a
|
||||
# live timestamp would make every re-run diff for a difference no test can read.
|
||||
# * **The configs are not rewritten at all.** A domain is not an identifier to hide here: a
|
||||
# `server_name` and the `proxy_pass` target beside it *are* the ingress map the gateway page
|
||||
# exists to draw, and a scrubbed fixture is a fixture that tests a config this gateway does
|
||||
@@ -55,7 +72,20 @@
|
||||
# * **A bad read never replaces a good fixture.** Empty output, an ssh failure or a permission
|
||||
# denial prints one line naming the file and the reason, leaves whatever is on disk alone,
|
||||
# and exits 1 — the `slots_gufo.json` rule, so a laptop with no route to the gateway cannot
|
||||
# turn six verbatim captures into six empty files.
|
||||
# turn seven verbatim captures into seven empty files. The resolver document keeps the rule with
|
||||
# one refinement: **one** name answering nothing is a fact and is recorded as
|
||||
# `status: "nxdomain"`, while a *broken vantage* is not — an ssh transport failure, a missing
|
||||
# `getent`, or not one declared name answering an address leaves the fixture untouched and
|
||||
# exits 1, because a document of nothing-but-failures would turn every live route on the page
|
||||
# into a permanent dash and every test of the join into a test of nothing.
|
||||
#
|
||||
# Two documents, one scrub. The socket tables and the resolver document each rewrite their
|
||||
# globally routable addresses through the **same** `Replacer` (one rule, two shapes, below), in
|
||||
# first-seen order per document. That order is per-document, so if a name ever resolves to a
|
||||
# globally routable address its replacement in `gateway_resolver.json` need not be the number the
|
||||
# socket tables gave that same address — which would silently unjoin `tests/test_gateway_source.py`'s
|
||||
# captured routes. `test_the_captured_resolver_document_needed_no_scrub_at_all` is the pin on that
|
||||
# hazard: it holds today because every declared target here answers a private address.
|
||||
#
|
||||
# No sudo, deliberately: haproxy's config is 0644, nginx's tree is 0644 and `/proc/net/tcp{,6}`
|
||||
# is world-readable, which is exactly what `dashboards-agent` reads as itself. A capture that
|
||||
@@ -80,12 +110,15 @@ usage() {
|
||||
cat <<'TXT'
|
||||
Usage: ./scripts/capture-gateway-config.sh [-h]
|
||||
|
||||
Re-records the gateway's proxy configs and kernel socket tables into tests/fixtures/:
|
||||
/etc/haproxy/haproxy.cfg, /etc/nginx/nginx.conf plus the documents its own glob includes
|
||||
name, and /proc/net/tcp{,6}. Every read goes over the `$GATEWAY_SSH` ssh alias (default
|
||||
`gateway-ops`) of a world-readable path, and a read that fails leaves the existing fixture
|
||||
alone and exits 1. The configs land unedited; in the socket tables a globally routable
|
||||
address becomes a documentation one and nothing else moves. See the block comment.
|
||||
Re-records the gateway's proxy configs, kernel socket tables and libc resolutions into
|
||||
tests/fixtures/: /etc/haproxy/haproxy.cfg, /etc/nginx/nginx.conf plus the documents its own
|
||||
glob includes name, /proc/net/tcp{,6}, and one `getent ahostsv4`/`getent ahostsv6` read on
|
||||
the gateway per target name `agent.host_agent.target_names()` derives from those configs
|
||||
(written as `gateway_resolver.json`, the agent's own `resolver` section shape). Every read
|
||||
goes over the `$GATEWAY_SSH` ssh alias (default `gateway-ops`) of a world-readable path or
|
||||
command, and a read that fails leaves the existing fixture alone and exits 1. The configs
|
||||
land unedited; in the socket tables and the resolver document a globally routable address
|
||||
becomes a documentation one and nothing else moves. See the block comment.
|
||||
TXT
|
||||
}
|
||||
|
||||
@@ -93,26 +126,60 @@ case "${1:-}" in
|
||||
-h|--help) usage; exit 0 ;;
|
||||
esac
|
||||
|
||||
# The socket tables' one edit, and the only place in this script that touches a captured byte.
|
||||
# Kept as a here-doc in the house style (capture-fixtures.sh embeds two); it is pinned by
|
||||
# running this script against a fake remote in tests/test_deploy_scripts.py.
|
||||
SOCKET_IDENTIFIERS_OUT=$(cat <<'PY'
|
||||
"""Move the globally-routable addresses in a /proc/net/tcp{,6} table to documentation space.
|
||||
# The interpreter for the two steps that import the repo's own parser.
|
||||
# `uv run python` is the same interpreter the tests use, which is the point: a capture whose name
|
||||
# list came from a different parser than the one the suite pins is a fixture that agrees with
|
||||
# nothing. A copy of this script with no project beside it (a test sandbox) has nothing for uv to
|
||||
# resolve, so it falls back to `python3` — `agent/host_agent.py` is stdlib-only, and PYTHONPATH
|
||||
# below is what makes it importable either way.
|
||||
export PYTHONPATH="$ROOT${PYTHONPATH:+:$PYTHONPATH}"
|
||||
run_py() {
|
||||
if command -v uv >/dev/null 2>&1 && [ -f "$ROOT/pyproject.toml" ]; then
|
||||
(cd "$ROOT" && uv run python "$@")
|
||||
else
|
||||
python3 "$@"
|
||||
fi
|
||||
}
|
||||
|
||||
Every column, every port and every byte of spacing is the kernel's. What changes is an address
|
||||
`ipaddress` calls global, which becomes a documentation address in first-seen order (RFC 5737
|
||||
for v4, RFC 3849 for v6) re-encoded through the kernel's own per-32-bit-word little-endian
|
||||
spelling — so the replacement is exactly as wide as the original and an IPv4-mapped v6 row stays
|
||||
an IPv4-mapped v6 row. AGENTS.md invariant 6 is the reason: this repo's capture rule is that
|
||||
identifiers never go in tracked files, and a global IPv6 on Linux is EUI-64 (it encodes the
|
||||
NIC's MAC) while a visitor's public address belongs to the visitor. Loopback, private,
|
||||
link-local, multicast, wildcard and already-documentation addresses are returned untouched,
|
||||
which is what makes re-running this over its own output change nothing.
|
||||
# The identifier rule (AGENTS.md invariant 6) and the resolver document, in one module with three
|
||||
# subcommands, and the only place in this script that touches a captured byte. One `Replacer` for
|
||||
# both documents is the point: two copies of "globally routable becomes documentation space, in
|
||||
# first-seen order" is two rules. Kept as a here-doc in the house style (capture-fixtures.sh
|
||||
# embeds two); it is pinned by running this script against a fake remote in
|
||||
# tests/test_deploy_scripts.py.
|
||||
SOCKET_IDENTIFIERS_OUT=$(cat <<'PY'
|
||||
"""The gateway capture's identifier rule and resolver document.
|
||||
|
||||
sockets <table> a /proc/net/tcp{,6} table on stdout, scrubbed
|
||||
plan <fixtures dir> <plan file> every target the captured configs declare
|
||||
resolver <plan> <raw dir> <fixture> <ts> the resolver document, or a refusal
|
||||
|
||||
**The scrub** (sockets, resolver). Every column, every port and every byte of spacing is the
|
||||
kernel's or `getent`'s. What changes is an address `ipaddress` calls global, which becomes a
|
||||
documentation address in first-seen order (RFC 5737 for v4, RFC 3849 for v6); in a socket table
|
||||
it is re-encoded through the kernel's own per-32-bit-word little-endian spelling, so the
|
||||
replacement is exactly as wide as the original and an IPv4-mapped v6 row stays an IPv4-mapped v6
|
||||
row. AGENTS.md invariant 6 is the reason: this repo's capture rule is that identifiers never go in
|
||||
tracked files, and a global IPv6 on Linux is EUI-64 (it encodes the NIC's MAC) while a visitor's
|
||||
public address belongs to the visitor — and a name that resolves to one is the same exposure by
|
||||
another route. Loopback, private, link-local, multicast, wildcard and already-documentation
|
||||
addresses are returned untouched, which is what makes re-running this over its own output change
|
||||
nothing.
|
||||
|
||||
**The plan** is derived from the repo's own parser and never by re-implementing one —
|
||||
`from agent.host_agent import target_names` — so the capture can never out-ask the agent. The
|
||||
name list, the order, the literal/name split and the `resolver` directive all come out of the
|
||||
agent's own reading of the config fixtures this run captured.
|
||||
"""
|
||||
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
import tempfile
|
||||
from typing import Any
|
||||
|
||||
DOC_V4 = int(ipaddress.IPv4Address("203.0.113.0"))
|
||||
DOC_V6 = int(ipaddress.IPv6Address("2001:db8::"))
|
||||
@@ -184,10 +251,37 @@ class Replacer:
|
||||
out.append(text[last:])
|
||||
return "".join(out)
|
||||
|
||||
def address(self, value: str) -> str:
|
||||
"""One dotted/colon address in its own spelling: scrub it, keep the spelling.
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
The resolver document's shape, where the socket table's is hex words. Same rule and the
|
||||
same mapping: a globally routable address becomes a documentation one and everything else
|
||||
— private, loopback, link-local, wildcard, already-documentation — is returned as it came.
|
||||
A `::ffff:`-mapped address keeps its mapped spelling, because the spelling is a fact about
|
||||
the answer, not an identifier.
|
||||
"""
|
||||
try:
|
||||
candidate = ipaddress.ip_address(value)
|
||||
except ValueError:
|
||||
return value
|
||||
mapped = getattr(candidate, "ipv4_mapped", None)
|
||||
subject = mapped if mapped is not None else candidate
|
||||
if not subject.is_global:
|
||||
return value
|
||||
key = str(subject)
|
||||
if key not in self.mapping:
|
||||
self.mapping[key] = self.documentation(subject.version)
|
||||
replacement = self.mapping[key]
|
||||
self.replacements += 1
|
||||
if mapped is not None:
|
||||
return f"::ffff:{replacement}"
|
||||
return replacement
|
||||
|
||||
|
||||
def sockets(argv: list[str]) -> int:
|
||||
"""`sockets <table>` — the table on stdout with its two address columns scrubbed."""
|
||||
replacer = Replacer()
|
||||
with open(argv[1], encoding="utf-8", errors="replace") as handle:
|
||||
with open(argv[2], encoding="utf-8", errors="replace") as handle:
|
||||
for line in handle:
|
||||
sys.stdout.write(replacer.line(line))
|
||||
if replacer.replacements:
|
||||
@@ -197,6 +291,179 @@ def main(argv: list[str]) -> int:
|
||||
return 0
|
||||
|
||||
|
||||
def plan(argv: list[str]) -> int:
|
||||
"""`plan <fixtures dir> <plan file>` — every declared target, in the agent's own order.
|
||||
|
||||
Reads the config fixtures this run just captured, from a tree shaped the way the agent's
|
||||
`--root` shapes it, so the plan is the answer to "what would the agent on this box ask?" and
|
||||
not a second opinion about what a config declares. The first line is the nginx `resolver`
|
||||
directive read out of the same bytes the agent reads them from; the rest is one tab-separated
|
||||
line per target: `target`, `name` or `literal`, the name, the port (empty when the config
|
||||
wrote none).
|
||||
"""
|
||||
from agent.host_agent import Collector, address_family, target_names
|
||||
|
||||
fixtures, out = pathlib.Path(argv[2]), pathlib.Path(argv[3])
|
||||
tree = pathlib.Path(tempfile.mkdtemp(prefix="gateway-plan-"))
|
||||
for rel, fixture in (("etc/haproxy/haproxy.cfg", "gateway_haproxy.cfg"),
|
||||
("etc/nginx/nginx.conf", "gateway_nginx_main.conf")):
|
||||
if (fixtures / fixture).is_file():
|
||||
body = "".join(line for line in (fixtures / fixture).read_text().splitlines(keepends=True)
|
||||
if not line.startswith("#"))
|
||||
target = tree / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(body)
|
||||
else:
|
||||
print(f"plan: {fixture} is not on disk — capture the configs before the resolver",
|
||||
file=sys.stderr)
|
||||
return 1
|
||||
for prefix, rel in (("gateway_nginx_conf_d_", "etc/nginx/conf.d"),
|
||||
("gateway_nginx_stream_d_", "etc/nginx/stream.d")):
|
||||
for found in sorted(fixtures.glob(f"{prefix}*.conf")):
|
||||
target = tree / rel / f"{found.name[len(prefix):]}"
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text("".join(line for line in found.read_text().splitlines(keepends=True)
|
||||
if not line.startswith("#")))
|
||||
|
||||
collector = Collector(str(tree), smartctl="/nonexistent/smartctl")
|
||||
errors: list[str] = []
|
||||
haproxy = collector.haproxy(errors)
|
||||
nginx, directive = collector.nginx_document(errors)
|
||||
targets = target_names(haproxy, nginx)
|
||||
if not targets:
|
||||
print("plan: the captured configs declare no target, so there is nothing to resolve — "
|
||||
"is the config capture intact?", file=sys.stderr)
|
||||
return 1
|
||||
# The name is the one part of the plan that goes back out over ssh, inside a remote command
|
||||
# string. A config target is written by the operator, so this is not a hostile input today —
|
||||
# but a fixture step that interpolated arbitrary config text into a shell command would be a
|
||||
# remote-command hole waiting for somebody to write a semicolon in a `server_name`, and the
|
||||
# honest answer to a name the script cannot safely ask about is to refuse the run, not to ask
|
||||
# it anyway. Names, dots, dashes, underscores and scoped literals only; no whitespace, quote,
|
||||
# semicolon, dollar or backtick.
|
||||
unsafe = [name for name, _port in targets
|
||||
if not re.fullmatch(r"[A-Za-z0-9._:.-]{1,253}", name)]
|
||||
if unsafe:
|
||||
print(f"plan: {len(unsafe)} target name(s) the script cannot ask about safely, starting "
|
||||
f"with {unsafe[0]!r} — nothing was read and nothing was written", file=sys.stderr)
|
||||
return 1
|
||||
with out.open("w", encoding="utf-8") as handle:
|
||||
handle.write(f"directive\t{directive if directive is not None else ''}\n")
|
||||
for name, port in targets:
|
||||
# The agent's own test, imported rather than restated: a target that is already an
|
||||
# address is `status: "literal"` and costs no read on the remote, exactly as the
|
||||
# agent's pass skips its own lookup.
|
||||
kind = "literal" if address_family(name) else "name"
|
||||
handle.write(f"target\t{kind}\t{name}\t{'' if port is None else port}\n")
|
||||
return 0
|
||||
|
||||
|
||||
def resolver(argv: list[str]) -> int:
|
||||
"""`resolver <plan file> <raw dir> <fixture> <ts>` — the document, or a refusal.
|
||||
|
||||
Written to the fixture path only when at least one declared name answered an address; every
|
||||
name failing means the vantage is broken (a resolver down, a firewall between, a `getent` that
|
||||
answers nothing), and a fixture of failures would make every live route on the page a
|
||||
permanent dash and every test of the join a test of nothing. One name failing among successes
|
||||
is a fact, and is kept as `status: "nxdomain"`.
|
||||
"""
|
||||
# The shape's single source of truth: the same `via` and the same address order the agent
|
||||
# publishes, imported rather than remembered, so the fixture is the agent's document.
|
||||
from agent.host_agent import RESOLVE_VIA, sort_resolver_addresses
|
||||
|
||||
plan_path = pathlib.Path(argv[2])
|
||||
raw = argv[3]
|
||||
out = pathlib.Path(argv[4])
|
||||
ts = float(argv[5])
|
||||
replacer = Replacer()
|
||||
directive: str | None = None
|
||||
entries: list[dict[str, Any]] = []
|
||||
names = 0
|
||||
answered = 0
|
||||
index = 0
|
||||
with open(plan_path, encoding="utf-8") as handle:
|
||||
for line in handle:
|
||||
fields = line.rstrip("\n").split("\t")
|
||||
if fields[0] == "directive":
|
||||
directive = fields[1] if len(fields) > 1 and fields[1] else None
|
||||
continue
|
||||
if fields[0] != "target":
|
||||
continue
|
||||
kind, name = fields[1], fields[2]
|
||||
port: int | None = int(fields[3]) if len(fields) > 3 and fields[3] else None
|
||||
if kind == "literal":
|
||||
found, is_literal = [name], True
|
||||
else:
|
||||
names += 1
|
||||
is_literal = False
|
||||
# `getent ahostsv6` answers `::ffff:a.b.c.d` for a name with no AAAA record — the
|
||||
# spelling glibc's AF_INET6 view synthesises, and the one `getaddrinfo` the agent
|
||||
# performs never publishes it. Keeping it would claim an address the agent does
|
||||
# not, so a mapped answer whose v4 form is in the same set is dropped. A mapped
|
||||
# answer with no plain one beside it survives, because then it is the answer.
|
||||
found = [field[0] for path in (f"{index}.v4", f"{index}.v6")
|
||||
for field in (line.split() for line in read_raw(raw, path).splitlines())
|
||||
if field]
|
||||
plain = {f"::ffff:{address}" for address in found
|
||||
if not address.startswith("::ffff:")}
|
||||
found = [address for address in found if address not in plain]
|
||||
index += 1
|
||||
# Scrub first, then sort: `addresses` is the published list, so the agent's ordering
|
||||
# rule belongs to what the fixture prints and not to what the resolver answered.
|
||||
addresses = sort_resolver_addresses(replacer.address(a) for a in found)
|
||||
if is_literal:
|
||||
status = "literal"
|
||||
else:
|
||||
status = "ok" if addresses else "nxdomain"
|
||||
if addresses:
|
||||
answered += 1
|
||||
entries.append({"name": name, "port": port, "status": status,
|
||||
"addresses": addresses, "ts": ts})
|
||||
|
||||
if names and not answered:
|
||||
print(f"resolver: none of the {names} declared name(s) answered an address — the vantage "
|
||||
"is broken, not the fixture; nothing was written", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
document = {"available": True, "via": RESOLVE_VIA, "config_directive": directive,
|
||||
"ts": ts, "entries": entries, "errors": []}
|
||||
tmp = out.with_name(f".{out.name}.tmp")
|
||||
tmp.write_text(json.dumps(document, indent=2) + "\n", encoding="utf-8")
|
||||
tmp.replace(out)
|
||||
tallies: dict[str, int] = {}
|
||||
for row in entries:
|
||||
tallies[row["status"]] = tallies.get(row["status"], 0) + 1
|
||||
summary = ", ".join(f"{count} {status}" for status, count in sorted(tallies.items()))
|
||||
if replacer.replacements:
|
||||
summary += (f"; {replacer.replacements} address(es) moved to documentation space "
|
||||
f"({len(replacer.mapping)} distinct)")
|
||||
print(f"{len(entries)} target(s): {summary}")
|
||||
return 0
|
||||
|
||||
|
||||
def read_raw(directory: str, name: str) -> str:
|
||||
"""One `getent` reply, or empty when the name never got that far.
|
||||
|
||||
A missing file is not an error here: the transport failure that would have left a hole is
|
||||
fatal in the shell and stops the run before this step, so a hole means a `literal` target,
|
||||
which is read out of the plan instead.
|
||||
"""
|
||||
path = os.path.join(directory, name)
|
||||
if not os.path.isfile(path):
|
||||
return ""
|
||||
with open(path, encoding="utf-8", errors="replace") as handle:
|
||||
return handle.read()
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
commands = {"sockets": sockets, "plan": plan, "resolver": resolver}
|
||||
mode = argv[1] if len(argv) > 1 else "sockets"
|
||||
if mode not in commands:
|
||||
print(f"this module does sockets, plan or resolver, not '{mode}'", file=sys.stderr)
|
||||
return 2
|
||||
return commands[mode](argv)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(sys.argv))
|
||||
PY
|
||||
@@ -250,7 +517,7 @@ capture() {
|
||||
{
|
||||
header "$kind" "$path"
|
||||
case "$kind" in
|
||||
sockets) python3 -c "$SOCKET_IDENTIFIERS_OUT" "$tmp" ;;
|
||||
sockets) run_py -c "$SOCKET_IDENTIFIERS_OUT" sockets "$tmp" ;;
|
||||
*) cat "$tmp" ;;
|
||||
esac
|
||||
} >"$FIX/$fixture"
|
||||
@@ -343,6 +610,96 @@ fi
|
||||
capture gateway_proc_net_tcp.txt sockets /proc/net/tcp
|
||||
capture gateway_proc_net_tcp6.txt sockets /proc/net/tcp6
|
||||
|
||||
# --- the resolver document ---------------------------------------------------
|
||||
# The names the two configs declare, read on the box that dials them. This is the only fixture
|
||||
# whose *question* comes from the repo: the plan is `target_names()` over the configs captured
|
||||
# above, so the capture asks exactly what the agent will ask and never one name more.
|
||||
#
|
||||
# One world-readable, unprivileged read per name — `getent ahostsv4 <name>` then
|
||||
# `getent ahostsv6 <name>` — because that is libc resolution on the routing box, which is what
|
||||
# `via: "libc"` claims and what haproxy's `init-addr last,libc,none` performs. No sudo, no `dig`,
|
||||
# no `host`, no copy tool: a capture that needed a tool the gateway does not ship would record
|
||||
# something the agent cannot see.
|
||||
#
|
||||
# The exit statuses are read as three different sentences. `0` answered; `2` is getent's "no such
|
||||
# record", which is a fact about the name and becomes `status: "nxdomain"`; `255` is ssh itself
|
||||
# failing and `127` is a missing `getent`, either of which means the vantage is broken rather than
|
||||
# the name, so the run stops and the fixture on disk stays exactly as it was.
|
||||
capture_resolver() {
|
||||
local scratch rc_plan=0 rc_doc=0
|
||||
scratch="$(mktemp -d "${TMPDIR:-/tmp}/gateway-resolver.XXXXXX")" || {
|
||||
echo " ✗ gateway_resolver.json — no scratch directory; the fixture on disk is left alone"
|
||||
failed=1
|
||||
return 1
|
||||
}
|
||||
|
||||
if ! run_py -c "$SOCKET_IDENTIFIERS_OUT" plan "$FIX" "$scratch/plan" 2>"$scratch/plan.err"; then
|
||||
# Last line, not first: an exception's own sentence is the last one, and this is the line the
|
||||
# operator reads to decide what to fix.
|
||||
echo " ✗ gateway_resolver.json — $(tail -n 1 "$scratch/plan.err" 2>/dev/null)"
|
||||
echo " (the name list comes from agent.host_agent.target_names(); the fixture on disk is "
|
||||
echo " left alone)"
|
||||
rm -rf "$scratch"
|
||||
failed=1
|
||||
return 1
|
||||
fi
|
||||
|
||||
local kind name port index=0 rc4=0 rc6=0 broken=""
|
||||
while IFS=$'\t' read -r line f1 f2 f3; do
|
||||
[ "$line" = target ] || continue # the directive line needs no read: it is in the text
|
||||
kind="$f1"; name="$f2"; port="$f3"
|
||||
[ "$kind" = name ] || continue # a literal is already an address; the agent asks nothing
|
||||
rc4=0; rc6=0
|
||||
ssh "${SSH_OPTS[@]}" "$GATEWAY_SSH" "getent ahostsv4 $name" >"$scratch/$index.v4" \
|
||||
2>"$scratch/$index.err" || rc4=$?
|
||||
ssh "${SSH_OPTS[@]}" "$GATEWAY_SSH" "getent ahostsv6 $name" >"$scratch/$index.v6" \
|
||||
2>"$scratch/$index.err6" || rc6=$?
|
||||
# `2` on its own is getent's "no such record", which the document records as nxdomain. These
|
||||
# two are not: one means the box never answered, the other means it has no getent to ask.
|
||||
if [ "$rc4" -eq 255 ] || [ "$rc6" -eq 255 ]; then
|
||||
broken="$GATEWAY_SSH did not answer (ssh exited 255)"
|
||||
break
|
||||
fi
|
||||
if [ "$rc4" -eq 127 ] || [ "$rc6" -eq 127 ]; then
|
||||
broken="getent is not available on $GATEWAY_SSH"
|
||||
break
|
||||
fi
|
||||
index=$((index + 1))
|
||||
done <"$scratch/plan"
|
||||
|
||||
if [ -n "$broken" ]; then
|
||||
echo " ✗ gateway_resolver.json — $broken; a broken vantage is not a fixture, and the one on "
|
||||
echo " disk is left alone"
|
||||
rm -rf "$scratch"
|
||||
failed=1
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Fixed, not `date`: the fixture's `ts` is a shape (an entry carries the time it was answered)
|
||||
# and a live clock would make every re-run diff for a difference no test can read. 2026-10-25
|
||||
# 00:00:00Z, the date this document was first recorded.
|
||||
local summary
|
||||
if ! summary="$(run_py -c "$SOCKET_IDENTIFIERS_OUT" resolver "$scratch/plan" "$scratch" \
|
||||
"$FIX/gateway_resolver.json" 1792886400 2>"$scratch/doc.err")"; then
|
||||
echo " ✗ gateway_resolver.json — $(tail -n 1 "$scratch/doc.err" 2>/dev/null)"
|
||||
rm -rf "$scratch"
|
||||
failed=1
|
||||
return 1
|
||||
fi
|
||||
echo " ✓ gateway_resolver.json ($summary, on $GATEWAY_SSH)"
|
||||
rm -rf "$scratch"
|
||||
}
|
||||
|
||||
# The resolver rides on the configs: no config capture means no name list, and a document derived
|
||||
# from a half-captured tree would be a fixture of the wrong question. Say so rather than being
|
||||
# silent about the one fixture this run did not refresh.
|
||||
if [ "$failed" -eq 0 ]; then
|
||||
capture_resolver
|
||||
else
|
||||
echo " - gateway_resolver.json — not attempted: the configs it derives its name list from did "
|
||||
echo " not all capture"
|
||||
fi
|
||||
|
||||
if [ "$failed" -ne 0 ]; then
|
||||
echo
|
||||
echo "capture FAILED — every fixture named above is unchanged on disk. Is $GATEWAY_SSH an ssh"
|
||||
|
||||
+8
-11
@@ -241,18 +241,15 @@ def _endpoint_values(connections: Any) -> list[Any]:
|
||||
Several lines can reach one address (`:443 → 10.1.1.9:443` and `:8443 → 10.1.1.9:443`), and
|
||||
`sources/gateway.py` repeats that endpoint's whole count on each of them with `shared: true`
|
||||
rather than splitting a number nobody measured. Summing the **lines** would book the same
|
||||
sockets twice and print a router busier than it is, so the total is taken over the endpoints
|
||||
the lines name — the same reason the outbound gauges are keyed by `(address, port)`.
|
||||
sockets twice and print a router busier than it is.
|
||||
|
||||
The grouping is `sources/gateway.distinct_endpoint_counts` and not a local `(host, port)` dict,
|
||||
because since phase 32 task 03 the two spellings of one endpoint can be two different **names**:
|
||||
two `server_name`s the gateway resolved to one address are one kernel endpoint whose sockets
|
||||
must be booked once, and a total keyed on the config's text would count them once per name. One
|
||||
reading of "one endpoint" — the join's — is what keeps `live_out` and the pipes agreeing.
|
||||
"""
|
||||
values: dict[tuple[Any, Any], Any] = {}
|
||||
for row in connections if isinstance(connections, Sequence) else ():
|
||||
if not isinstance(row, Mapping):
|
||||
continue
|
||||
out = row.get("out")
|
||||
if not isinstance(out, Mapping):
|
||||
continue
|
||||
values.setdefault((out.get("host"), out.get("port")), row.get("connections"))
|
||||
return list(values.values())
|
||||
return gateway_source.distinct_endpoint_counts(connections)
|
||||
|
||||
|
||||
def _gateway_summary(payload: Mapping[str, Any] | None, settings: Settings) -> dict[str, Any]:
|
||||
|
||||
@@ -38,11 +38,16 @@ class UpstreamError(Exception):
|
||||
|
||||
|
||||
#: Agent add-on sections that report their own absence on purpose. Their `errors` lines are
|
||||
#: facts about a source this page does not draw — `smart:` belongs to `/disks`, `haproxy:` and
|
||||
#: `nginx:` to the gateway page (phase 29) — so they never make the **system** payload
|
||||
#: facts about a source this page does not draw — `smart:` belongs to `/disks`, `haproxy:`,
|
||||
#: `nginx:` and `resolver:` to the gateway page (phases 28/32) — so they never make the
|
||||
#: **system** payload
|
||||
#: `degraded`, which is the overview's alert input. Without this a fleet-wide agent upgrade
|
||||
#: would put a red "agent degraded" banner and an overview WARN on every host that has no
|
||||
#: haproxy or nginx installed, which is invariant 3's failure mode with an upgrade's clothes on.
|
||||
#: The `resolver:` arms are the same case with a DNS-shaped name: a host with no proxy config has
|
||||
#: nothing to resolve, and a name the resolver could not answer (`resolver: pass budget …`,
|
||||
#: a budget line, or the section's own absence line) is an add-on being honest about one route,
|
||||
#: not the box being broken.
|
||||
#: The page that owns the add-on still reads the lines: they stay verbatim in `agent.errors`.
|
||||
#:
|
||||
#: **A new agent add-on section adds its prefix here in the same commit.** The list the agent's
|
||||
@@ -50,7 +55,7 @@ class UpstreamError(Exception):
|
||||
#: (`tests/test_agent.py::test_the_addon_error_prefixes_are_the_same_list_the_dashboard_filters`),
|
||||
#: so a section that ships its absence-report without a prefix here fails instead of turning
|
||||
#: every host amber.
|
||||
ADDON_ERROR_PREFIXES: tuple[str, ...] = ("smart", "haproxy", "nginx", "sockets")
|
||||
ADDON_ERROR_PREFIXES: tuple[str, ...] = ("smart", "haproxy", "nginx", "sockets", "resolver")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
@@ -20,9 +20,13 @@ What this module owns, and what it deliberately does not:
|
||||
* **No verdict.** `severity.py` rates no connection count and this payload carries no
|
||||
`severity` dict on one — the `net.*` precedent: 400 connections on a reverse proxy is the
|
||||
job, and 0 on an idle one is healthy. Pages colour nothing here.
|
||||
* **No DNS, ever.** The agent never resolves a name and neither does this module
|
||||
(`tests/test_gateway_source.py` spies `socket.getaddrinfo` and pins that it is not called
|
||||
once). That single fact decides the whole outbound join, below.
|
||||
* **No DNS, ever.** The *agent* resolves; this module does not. Agent 1.3.0's `resolver` section
|
||||
is one `getaddrinfo` per declared target, run **on the host that opens those connections**
|
||||
(haproxy's own `init-addr last,libc,none`), and this projection only reads the answer out of the
|
||||
snapshot. The dashboard asking the same name through its own `resolv.conf` would be a second
|
||||
vantage answering a different question, so `tests/test_gateway_source.py` spies
|
||||
`socket.getaddrinfo` and pins that `parse()` never calls it — including on the path that
|
||||
consumes a resolver document.
|
||||
|
||||
The join: two arms, and the ignore rule
|
||||
---------------------------------------
|
||||
@@ -34,25 +38,31 @@ actually support, and deleted the rest. What survives, and what is deliberately
|
||||
remote address is the client. `listeners[port]` takes the count and up to `MAX_CLIENTS` distinct
|
||||
client addresses.
|
||||
|
||||
**Outbound** is sound in exactly one case: the socket's remote `(address, port)` **equals a
|
||||
target the config wrote as an address**. That is a fact `/proc` states directly, so the endpoint
|
||||
carries it.
|
||||
**Outbound** is sound in exactly two spellings of one fact: the socket's remote `(address, port)`
|
||||
**equals a target the config wrote as an address**, or it equals an address **the gateway itself
|
||||
published for the name it wrote** — agent 1.3.0's `resolver` section, which is the resolution that
|
||||
opened the socket. Either way the far end is named in the same currency `/proc` writes it in, and
|
||||
the row says which of the two it was (`out.address`, `out.resolved`, `out.via`) beside the count.
|
||||
The comparison is made on **parsed addresses** (`_address_key`), never on strings: `/proc`'s
|
||||
`inet_ntop` output, an IPv4-mapped `::ffff:10.1.2.64` and a resolver's `10.1.2.64` are one address,
|
||||
and a route that resolves yet fails to count because of a spelling is the bug that rule exists to
|
||||
keep out. The payload still publishes both spellings verbatim.
|
||||
|
||||
**Everything else is ignored — not counted, not totalled, not warned about.** A config writes
|
||||
`reeseapps-brain.reeselink.com:443`; `/proc` reports `10.1.1.106:443`. Nothing in this stack
|
||||
resolves a name (the `socket.getaddrinfo` spy pins that), so a name target can never be matched
|
||||
to a socket, and a port several targets share — this fleet's `stream.conf` map, sixteen `:443`
|
||||
entries — credits none of them. The arm that used to bridge that gap credited the *port* and
|
||||
labelled it `by_port`, and swept every leftover into `unmatched.outbound` /
|
||||
`unmatched.groups` / `unmatched.other`. On the live router that residual was mostly this app
|
||||
talking to itself (`9120`, the agent's own port, and `2222`): a residual that is mostly other
|
||||
software is not a signal, and a card of it teaches the operator to ignore the panel. So
|
||||
`unmatched`, `by_port` and `partial` are gone, and a socket outside the two config files draws
|
||||
nothing at all.
|
||||
`reeseapps-brain.reeselink.com:443`; `/proc` reports `10.1.1.106:443`. Until phase 32 shipped the
|
||||
capture of that name nothing in this stack could join the two, so a port several targets share —
|
||||
this fleet's `stream.conf` map, sixteen `:443` entries — credited none of them. The arm that used
|
||||
to bridge that gap credited the *port* and labelled it `by_port`, and swept every leftover into
|
||||
`unmatched.outbound` / `unmatched.groups` / `unmatched.other`. On the live router that residual was
|
||||
mostly this app talking to itself (`9120`, the agent's own port, and `2222`): a residual that is
|
||||
mostly other software is not a signal, and a card of it teaches the operator to ignore the panel.
|
||||
So `unmatched`, `by_port` and `partial` are gone for good — the resolver document replaced the
|
||||
guessing, not a wider guess — and a socket outside the two config files draws nothing at all.
|
||||
|
||||
The consequence for a target written as a **name** is its honest answer: `connections` stays
|
||||
`null` — `n/a`, never `0`. An address endpoint carries the number, `0` included when the sockets
|
||||
answered and it genuinely has none.
|
||||
The consequence for a name the gateway **did not** publish an address for (`nxdomain`, a resolver
|
||||
that would not answer, an agent older than 1.3.0, `--no-resolve`) is its honest answer:
|
||||
`connections` stays `null` — `n/a`, never `0`, and **no sample is stored** for it. A countable
|
||||
endpoint carries the number, `0` included when the sockets answered and it genuinely has none.
|
||||
|
||||
**A shared endpoint repeats its count and never splits it.** Several connections can reach one
|
||||
address (`:443 → backend:443` and `:8443 → backend:443`); each carries the endpoint's whole
|
||||
@@ -162,19 +172,22 @@ sockets. One name for both is one `INSERT OR REPLACE` overwriting the other in s
|
||||
the aggregation failure rule 2 exists to catch. The port is also what makes the name stable
|
||||
when the same host later gains or loses a second service.
|
||||
|
||||
Only a connection whose endpoint the join could count publishes a gauge, which since task 02
|
||||
means **the name-keyed `gateway.out_conns@<domain>:<port>` series stop being written**: the
|
||||
number they carried was the port's, not the target's, and storing an unattributable figure
|
||||
under a target's name is rule 2 with a series name on it. The rows already stored age out
|
||||
under the unchanged retention policy, so nothing here needs a schema or prune change
|
||||
(invariant 9).
|
||||
Only a connection whose endpoint the join could count publishes a gauge: an address the config
|
||||
wrote, **or** a name the gateway resolved. Phase 30 task 02 stopped writing the name-keyed
|
||||
`gateway.out_conns@<domain>:<port>` series because the only figure they could have carried was the
|
||||
*port's* — the port-only join's count filed under a target's name, which is rule 2 with a series
|
||||
name on it. Phase 32 task 03 resumes writing them on a different footing: the number is the
|
||||
endpoint's, and the endpoint is the one this name resolved to, so the name is honest. It stays the
|
||||
**config's own spelling** (`:25565` and `:25566` remain two gauges, and the series phase 29 would
|
||||
have written keeps growing under the same name), and a name nobody resolved still publishes
|
||||
nothing at all.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
import time
|
||||
from collections.abc import Mapping, MutableMapping, Sequence
|
||||
from collections.abc import Collection, Mapping, MutableMapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
from ..poller import Result
|
||||
@@ -280,6 +293,29 @@ def _mappings(value: Any) -> list[Mapping[str, Any]]:
|
||||
return [item for item in value if isinstance(item, Mapping)]
|
||||
|
||||
|
||||
def _addresses(value: Any) -> list[str]:
|
||||
"""The unique non-empty strings of `value`, first-seen order: an agent's address list.
|
||||
|
||||
Order is the agent's, because that is the order the page prints and a sorted list would
|
||||
reorder a fan-out on every poll where libc answered differently.
|
||||
"""
|
||||
if not isinstance(value, Sequence) or isinstance(value, str | bytes):
|
||||
return []
|
||||
found: list[str] = []
|
||||
for item in value:
|
||||
text = _text(item)
|
||||
if text is not None and text not in found:
|
||||
found.append(text)
|
||||
return found
|
||||
|
||||
|
||||
def _stamp(value: Any) -> float | None:
|
||||
"""A number usable as a timestamp, else ``None``. A bool is not a number here."""
|
||||
if isinstance(value, bool) or not isinstance(value, int | float):
|
||||
return None
|
||||
return float(value)
|
||||
|
||||
|
||||
def _section(snapshot: Mapping[str, Any], key: str) -> Mapping[str, Any] | None:
|
||||
"""One of the three sections, or ``None`` when the agent did not publish it."""
|
||||
value = snapshot.get(key)
|
||||
@@ -306,10 +342,14 @@ def _available(section: Mapping[str, Any] | None) -> bool | None:
|
||||
def supported(snapshot: Mapping[str, Any]) -> bool:
|
||||
"""Does this agent publish the proxy/socket sections at all?
|
||||
|
||||
Presence of **any** one of the three counts, so a hypothetical agent that publishes
|
||||
`sockets` but not the configs is "supported, with a half missing" (which `sections` and
|
||||
`why` then name) rather than "predates support". A 1.1.0 agent and a `--no-proxy` agent
|
||||
both publish none, which is the single arm this answers `False` for.
|
||||
The three phase-28 sections and nothing else: presence of **any** one of them counts, so a
|
||||
hypothetical agent that publishes `sockets` but not the configs is "supported, with a half
|
||||
missing" (which `sections` and `why` then name) rather than "predates support". A 1.1.0 agent
|
||||
and a `--no-proxy` agent both publish none, which is the single arm this answers `False` for.
|
||||
|
||||
Phase 32's `resolver` section is **not** in that test on purpose. A 1.3.0 agent answers `true`
|
||||
because it still publishes the three, and a 1.3.0 agent run with `--no-resolve` answers `true`
|
||||
too: the resolver is what makes a *name* countable, never what makes this page supported.
|
||||
"""
|
||||
return any(key in snapshot for key in SECTIONS)
|
||||
|
||||
@@ -535,18 +575,19 @@ def _connection(source: str, in_port: int, host: str, out_port: int | None, name
|
||||
kind: str, selectors: Sequence[str]) -> dict[str, Any]:
|
||||
"""One line of the graph: an inbound port, an outbound endpoint, and the config's word for it.
|
||||
|
||||
Three of these fields are set by the join later, and each says so by being `None` now:
|
||||
Three of these fields are set later, and each says so by being `None` here:
|
||||
|
||||
* `connections` — a count is a socket fact and this function reads config text. A line whose
|
||||
endpoint the config wrote as a **name** keeps the `None` forever: `/proc` has addresses,
|
||||
nothing here resolves, so nobody can ever count it.
|
||||
* `connections` — a count is a socket fact and this function reads config text.
|
||||
* `shared` — `true` when another line reaches the same endpoint and therefore carries the same
|
||||
number, so a reader never mistakes two lines for twice the traffic.
|
||||
* `metric` — the gauge this line publishes, and **`None` for exactly the lines `connections`
|
||||
can never have**: a series that can only ever be empty is not a series, and asking
|
||||
`/api/gateway/history` for one draws a flat line at nothing. Where it is set, the name
|
||||
carries the port as well as the address, because `:25565` and `:25566` on one host are two
|
||||
services and one name would make the second insert overwrite the first.
|
||||
* `metric` — the gauge this line publishes.
|
||||
|
||||
`apply_resolution` then widens all three on one question: can the kernel name this line's far
|
||||
end? An address the config wrote can, and so can a name the gateway resolved on the box that
|
||||
dials it. A name nobody resolved never can, and keeps `connections: null`, `metric: null` and
|
||||
**no stored sample**. Where a metric exists its name carries the port as well as the host,
|
||||
because `:25565` and `:25566` on one host are two services and one name would make the second
|
||||
insert overwrite the first.
|
||||
"""
|
||||
suffix = f"{host}:{out_port}" if out_port is not None else host
|
||||
address = _is_ip_literal(host)
|
||||
@@ -868,33 +909,281 @@ def _local_port(socket: Mapping[str, Any]) -> int | None:
|
||||
return _port(local.get("port"))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# the resolver section: read, never re-derived
|
||||
#
|
||||
# Agent 1.3.0 looks each declared target up with `getaddrinfo` **on the host that dials it** and
|
||||
# publishes the answer, its `via` string and its own timestamp. This module reads that document and
|
||||
# asks libc nothing: the dashboard's `resolv.conf` is a different vantage, and the fleet's names
|
||||
# resolve back through this same router. `tests/test_gateway_source.py` spies `socket.getaddrinfo`
|
||||
# across the whole `parse()` path — including a snapshot whose resolver section is full — so the
|
||||
# day somebody "helpfully" resolves a name here, the test fails instead of a page.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
def resolution_index(snapshot: Mapping[str, Any]
|
||||
) -> dict[tuple[str, int | None], Mapping[str, Any]]:
|
||||
"""`(name, port) -> entry` for every resolver entry that published at least one address.
|
||||
|
||||
Three refusals, each the section's own answer rather than this module's:
|
||||
|
||||
* **no `resolver` section at all** (agent 1.2.0, or 1.3.0 with `--no-resolve`), or one whose
|
||||
`available` is not `True` → `{}`, which is byte-for-byte the phase-30 world: no name is
|
||||
joinable and every name line keeps its dash. An entry published beside `available: null` is a
|
||||
pass that could not run, and a stale address is not a fact about this poll — the `warnings()`
|
||||
rule, for the same reason: a section that did not answer is not judged;
|
||||
* an entry with **no addresses** (`nxdomain`, `error`) contributes nothing — an empty answer is
|
||||
not a set containing a `None` address, and joining on one would match a socket nobody named;
|
||||
* a repeated `(name, port)` keeps the **first** entry, the merge rule the connection graph
|
||||
already applies to a selector.
|
||||
|
||||
The port is read with `_port`, so an entry naming no port is keyed `None` and can only ever be
|
||||
found by a connection whose endpoint names no port either — a pair that is never joinable.
|
||||
"""
|
||||
section = _section(snapshot, "resolver")
|
||||
if section is None or _available(section) is not True:
|
||||
return {}
|
||||
index: dict[tuple[str, int | None], Mapping[str, Any]] = {}
|
||||
for entry in _mappings(section.get("entries")):
|
||||
name = _text(entry.get("name"))
|
||||
if name is None or not _addresses(entry.get("addresses")):
|
||||
continue
|
||||
index.setdefault((name, _port(entry.get("port"))), entry)
|
||||
return index
|
||||
|
||||
|
||||
def resolver_entries(section: Mapping[str, Any] | None
|
||||
) -> list[dict[str, Any]]:
|
||||
"""The section's entries as the payload publishes them: name, port, status, addresses, ts.
|
||||
|
||||
Every well-formed entry of a section that **answered**, `nxdomain` and `error` included — the
|
||||
index filters within this list, and a page that says *this name has no record* needs the entry
|
||||
that says so. A section that did not answer (`available` false or null) publishes none: an entry
|
||||
beside a failed pass is not a thing to render as an answer. Projected to the five keys rather
|
||||
than passed through, so a payload key a page indexes cannot appear by an agent growing a field
|
||||
nobody reviewed.
|
||||
"""
|
||||
if section is None or _available(section) is not True:
|
||||
return []
|
||||
rows: list[dict[str, Any]] = []
|
||||
for entry in _mappings(section.get("entries")):
|
||||
name = _text(entry.get("name"))
|
||||
if name is None:
|
||||
continue
|
||||
rows.append({"name": name, "port": _port(entry.get("port")),
|
||||
"status": _text(entry.get("status")),
|
||||
"addresses": _addresses(entry.get("addresses")),
|
||||
"ts": _stamp(entry.get("ts"))})
|
||||
return rows
|
||||
|
||||
|
||||
def apply_resolution(connections: Sequence[dict[str, Any]],
|
||||
index: Mapping[tuple[str, int | None], Mapping[str, Any]],
|
||||
via: str | None,
|
||||
ts: float | None) -> None:
|
||||
"""Stamp each connection's `out` with what the gateway resolved, and widen what is countable.
|
||||
|
||||
Three provenance fields, stamped on **every** row so a page never has to test for a key that
|
||||
may or may not be there:
|
||||
|
||||
* `resolved` — the address list published for this `(name, port)`, or `null`. **Never `[]`**:
|
||||
an entry with no addresses and no entry at all are one fact to a reader, and `is null` says
|
||||
it in one test instead of a length check that has to know which empty it found. It is stamped
|
||||
on the rows written as a **name** only: `address: true` already says the config wrote the
|
||||
address, and copying the agent's `literal` echo of that same string into `resolved` would put
|
||||
a `via: libc` lookup claim on a resolution that never happened. `endpoint_of` reads the half
|
||||
that applies, exactly as the page's `address` test always has.
|
||||
* `via` — the agent's own string (`"libc"`), copied verbatim, and `null` where nothing was
|
||||
resolved. This module adds no vantage and invents no word for one.
|
||||
* `resolved_ts` — that **entry's** timestamp, falling back to the section's (`ts`) when the
|
||||
entry carries none: a name refreshed five minutes ago and one looked up on this poll are two
|
||||
claims, and the page's tooltip should print the one that is true.
|
||||
|
||||
Then the one behaviour change, the widened joinability. Whether a line gets a gauge is
|
||||
`endpoint_of`'s answer — "does this row have at least one key the kernel can be compared with" —
|
||||
so there is exactly one definition of joinable in this module and a malformed resolver address
|
||||
(a name where an address should be) places nothing rather than naming a series no sample can ever
|
||||
reach:
|
||||
|
||||
```python
|
||||
# A line is countable when the kernel can name its far end: either the config wrote an
|
||||
# address, or the gateway's own libc resolution — the one that opens this connection —
|
||||
# published one. Phase 30 retired name-keyed gauges because they could only ever be
|
||||
# empty; this is the fact that made that untrue. The NAME keeps the config's spelling so
|
||||
# the stored series is the same one phase 29/30 would have written.
|
||||
```
|
||||
|
||||
A line the resolver could not place keeps `metric: null` and `connections: null`: no sample, no
|
||||
zero, and the dash the footer explains. Nothing here counts a socket — that is `join_sockets`,
|
||||
which runs after this and reads the keys this stamped onto `out.resolved`.
|
||||
"""
|
||||
for connection in connections:
|
||||
out = connection.get("out")
|
||||
if not isinstance(out, MutableMapping):
|
||||
continue
|
||||
host = _text(out.get("host"))
|
||||
port = _port(out.get("port"))
|
||||
if out.get("address") is True:
|
||||
# nothing to resolve, so nothing was: the three fields stay null on an address row
|
||||
out["resolved"] = out["via"] = out["resolved_ts"] = None
|
||||
continue
|
||||
entry = index.get((host, port)) if host is not None else None
|
||||
addresses = _addresses(entry.get("addresses")) if entry is not None else []
|
||||
out["resolved"] = list(addresses) if addresses else None
|
||||
out["via"] = via if addresses else None
|
||||
stamped = _stamp(entry.get("ts")) if (entry is not None and addresses) else None
|
||||
out["resolved_ts"] = stamped if stamped is not None else (ts if addresses else None)
|
||||
# Joinable is `endpoint_of`'s answer, not a second opinion here: an entry whose addresses
|
||||
# are not addresses (a malformed agent answer) places nothing, and a gauge name for a line
|
||||
# that can never publish a sample would be a metric no history can ever answer.
|
||||
if connection.get("metric") is None and endpoint_of(connection):
|
||||
connection["metric"] = f"{SOURCE}.out_conns@{out.get('host')}:{port}"
|
||||
|
||||
|
||||
def _address_key(value: Any) -> str | None:
|
||||
"""One address in the single spelling the **match** is made in, or ``None`` for a non-address.
|
||||
|
||||
`ipaddress` parses text and never dials, which is why it answers this and a resolver may not
|
||||
(`_is_ip_literal` uses it for the same reason). It canonicalises an IPv6 (`2001:0DB8::0001` →
|
||||
`2001:db8::1`) and `ipv4_mapped` folds the `::ffff:10.1.2.64` form `inet_ntop` writes for an
|
||||
IPv4 peer seen through the IPv6 table back onto the plain `10.1.2.64` a config or a resolver
|
||||
wrote — so one address in two tables is one key.
|
||||
|
||||
What is deliberately **not** folded is a spelling `ipaddress` refuses, a zero-padded IPv4 octet
|
||||
(`010.1.2.64`) above all: `inet_aton` reads a leading `0` as *octal*, the two readings are two
|
||||
addresses, and picking one here would book a socket on a route because of a guess. Such a target
|
||||
is not an address to this module, so it is a **name**, and the agent's resolver section is what
|
||||
turns it into one the kernel agrees with.
|
||||
|
||||
Only the match key is normalised. `out.host` and `out.resolved` keep publishing exactly what the
|
||||
config and the agent wrote, because the page prints the config's text, not this one.
|
||||
"""
|
||||
text = _text(value)
|
||||
if text is None:
|
||||
return None
|
||||
try:
|
||||
address = ipaddress.ip_address(text)
|
||||
except ValueError:
|
||||
return None
|
||||
if isinstance(address, ipaddress.IPv6Address) and address.ipv4_mapped is not None:
|
||||
return str(address.ipv4_mapped)
|
||||
return str(address)
|
||||
|
||||
|
||||
def endpoint_of(connection: Mapping[str, Any]) -> list[tuple[str, int]]:
|
||||
"""Every `(address, port)` key the kernel can name for this line's far end.
|
||||
|
||||
Two sources and never a mixture: the address **the config wrote** (one key), or every address
|
||||
the agent's resolver published for the name it wrote (one key each, because a multi-homed name
|
||||
is dialled at any of them and the sockets are the same route either way).
|
||||
|
||||
An empty list is what `None` was before task 03: nothing here can be compared with `/proc`, so
|
||||
the line is drawn and never counted, publishes no gauge, and carries no `0`. A target that names
|
||||
**no port** is in that arm too — the join needs the pair, and the port of a socket is not
|
||||
something this module may supply.
|
||||
|
||||
The keys are normalised (`_address_key`) and this function is the only place that normalisation
|
||||
lives, for the endpoints map and for the socket lookup both. The payload above it stays verbatim.
|
||||
"""
|
||||
out = connection.get("out")
|
||||
if not isinstance(out, Mapping):
|
||||
return []
|
||||
port = _port(out.get("port"))
|
||||
if port is None:
|
||||
return []
|
||||
if out.get("address") is True:
|
||||
key = _address_key(out.get("host"))
|
||||
return [(key, port)] if key is not None else []
|
||||
keys: list[tuple[str, int]] = []
|
||||
for address in _addresses(out.get("resolved")):
|
||||
key = _address_key(address)
|
||||
if key is not None and (key, port) not in keys:
|
||||
keys.append((key, port))
|
||||
return keys
|
||||
|
||||
|
||||
def _endpoint_groups(key_sets: Sequence[Collection[tuple[str, int]]]) -> list[int]:
|
||||
"""One group number per connection: lines reaching **one** endpoint share one counter row.
|
||||
|
||||
Two lines share an endpoint when their key sets *overlap*, which covers three shapes and not
|
||||
merely the obvious one: one `(address, port)` written by two vhosts (phase 30's `shared`), two
|
||||
**names** the resolver placed on one address, and a multi-homed name sharing one address with a
|
||||
single-homed one. `-1` is "no endpoint": a name nobody resolved, or a target naming no port.
|
||||
|
||||
Numbers are handed out in first-seen order so the grouping is deterministic, and the arithmetic
|
||||
it protects is one line: a socket increments exactly one counter, so repeating a number down a
|
||||
list of routes never repeats the traffic.
|
||||
"""
|
||||
parent: dict[tuple[str, int], tuple[str, int]] = {}
|
||||
|
||||
def find(key: tuple[str, int]) -> tuple[str, int]:
|
||||
root = key
|
||||
while parent[root] != root:
|
||||
root = parent[root]
|
||||
while parent[key] != root: # path compression: the chains stay flat
|
||||
parent[key], key = root, parent[key]
|
||||
return root
|
||||
|
||||
for keys in key_sets:
|
||||
if not keys:
|
||||
continue
|
||||
anchor = min(keys)
|
||||
parent.setdefault(anchor, anchor)
|
||||
root = find(anchor)
|
||||
for key in keys:
|
||||
parent.setdefault(key, key)
|
||||
other = find(key)
|
||||
if other != root:
|
||||
parent[other] = root
|
||||
|
||||
numbers: dict[tuple[str, int], int] = {}
|
||||
groups: list[int] = []
|
||||
for keys in key_sets:
|
||||
if not keys:
|
||||
groups.append(-1)
|
||||
continue
|
||||
root = find(min(keys))
|
||||
if root not in numbers:
|
||||
numbers[root] = len(numbers)
|
||||
groups.append(numbers[root])
|
||||
return groups
|
||||
|
||||
|
||||
def distinct_endpoint_counts(connections: Any) -> list[Any]:
|
||||
"""One count per **distinct** endpoint, first-seen order — the only figures a total may add.
|
||||
|
||||
`shared: true` says several lines carry one number because they reach one kernel endpoint, so a
|
||||
total over the **lines** books the same sockets twice and prints a router busier than it is.
|
||||
This is the reader `/api/summary`'s `live_out` goes through, and it lives here rather than there
|
||||
because the answer depends on the same normalised `(address, port)` identity the join uses: since
|
||||
task 03 two lines can reach one address while the two config files spell it as two different
|
||||
**names**, and a total keyed on the config's text would count that endpoint once per name.
|
||||
|
||||
A line with no endpoint contributes nothing at all: an unknown is not a zero, and the caller
|
||||
turns "nothing measured" into `null` (`api._live_total`).
|
||||
"""
|
||||
rows = _mappings(connections)
|
||||
groups = _endpoint_groups([set(endpoint_of(row)) for row in rows])
|
||||
values: dict[int, Any] = {}
|
||||
for row, group in zip(rows, groups, strict=True):
|
||||
if group < 0:
|
||||
continue
|
||||
values.setdefault(group, row.get("connections"))
|
||||
return list(values.values())
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# the join: two arms, and the ignore rule
|
||||
#
|
||||
# `outbound[]`, `by_port`, `unmatched`, `unmatched.groups`, `partial` and `MAX_GROUP_HOSTS`
|
||||
# were deleted here in phase 30 task 02. They existed to answer "which configured target does
|
||||
# this socket belong to?" for a target the config wrote as a **name**, and the honest answer is
|
||||
# that nothing in this stack can know (no DNS, ever). What was left over went into a card of
|
||||
# `unmatched` counts that, on the live router, was mostly this app's own agent port and SSH —
|
||||
# noise dressed as a signal. A socket the two configs do not describe is now ignored.
|
||||
# this socket belong to?" for a target the config wrote as a **name** while nothing in this stack
|
||||
# could know what that name meant, and the leftover went into a card of `unmatched` counts that, on
|
||||
# the live router, was mostly this app's own agent port and SSH — noise dressed as a signal. Phase
|
||||
# 32 did not bring the guessing back: it brought the document the **gateway** produced about its own
|
||||
# connections, which `apply_resolution` reads. A socket the two configs do not describe is still
|
||||
# ignored, and a name the gateway did not resolve is still counted nowhere.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
def endpoint_of(connection: Mapping[str, Any]) -> tuple[str, int] | None:
|
||||
"""The joinable endpoint key `(address, port)` of a connection, or ``None``.
|
||||
|
||||
``None`` means "a name, or an address that names no port": there is nothing in `/proc` to
|
||||
compare it with, so the connection is not an endpoint the join can count.
|
||||
"""
|
||||
out = connection.get("out")
|
||||
if not isinstance(out, Mapping) or out.get("address") is not True:
|
||||
return None
|
||||
host = out.get("host")
|
||||
port = out.get("port")
|
||||
if not isinstance(host, str) or not isinstance(port, int) or isinstance(port, bool):
|
||||
return None
|
||||
return host, port
|
||||
|
||||
|
||||
def join_sockets(listeners: Mapping[int, dict[str, Any]],
|
||||
endpoints: MutableMapping[tuple[str, int], dict[str, Any]],
|
||||
established: Sequence[Mapping[str, Any]]) -> None:
|
||||
@@ -902,8 +1191,11 @@ def join_sockets(listeners: Mapping[int, dict[str, Any]],
|
||||
|
||||
* a socket whose **local port** is a configured listener is inbound: that listener's
|
||||
`connections` goes up by one and its `clients` gains the remote address;
|
||||
* otherwise a socket whose **remote `(address, port)`** is an endpoint the config wrote as
|
||||
an address is outbound: that endpoint's `connections` goes up by one;
|
||||
* otherwise a socket whose **remote `(address, port)`** is an endpoint the config wrote as an
|
||||
address, or one the gateway's resolver published for the name it wrote, is outbound: that
|
||||
endpoint's `connections` goes up by one. The lookup is `_address_key(remote)`, so the
|
||||
`::ffff:`-mapped spelling `/proc` uses and the plain one a resolver returned are the same
|
||||
key — string equality here is the bug class where a route resolves and still never counts;
|
||||
* anything else is **not counted at all**, by decision — see the module docstring.
|
||||
|
||||
The identity that follows, and that `tests/test_gateway_source.py` pins: every socket is
|
||||
@@ -925,8 +1217,9 @@ def join_sockets(listeners: Mapping[int, dict[str, Any]],
|
||||
if remote_addr not in seen:
|
||||
seen.append(remote_addr)
|
||||
continue
|
||||
if remote_addr is not None and isinstance(remote_port, int):
|
||||
endpoint = endpoints.get((remote_addr, remote_port))
|
||||
key = _address_key(remote_addr)
|
||||
if key is not None and remote_port is not None:
|
||||
endpoint = endpoints.get((key, remote_port))
|
||||
if endpoint is not None:
|
||||
endpoint["connections"] = int(endpoint["connections"]) + 1
|
||||
for port, seen in clients.items():
|
||||
@@ -954,6 +1247,11 @@ def _skeleton(host: str) -> dict[str, Any]:
|
||||
"connections": [],
|
||||
"warnings": [],
|
||||
"sockets": {"listens": None, "established": None},
|
||||
# Phase 32's fourth section, in the one place the key set is declared so `parse` and
|
||||
# `stub` cannot drift. `available: null` is "this agent said nothing about resolution"
|
||||
# — a 1.2.0 agent, `--no-resolve`, or a host that has not answered — and is deliberately
|
||||
# not `false`, which is the agent's own word for "nothing to resolve".
|
||||
"resolver": {"available": None, "via": None, "entries": []},
|
||||
}
|
||||
|
||||
|
||||
@@ -1012,6 +1310,12 @@ def parse(snapshot: Mapping[str, Any], host: str) -> dict[str, Any]:
|
||||
version = _text(snapshot.get("agent_version"))
|
||||
if version:
|
||||
payload["agent_version"] = version
|
||||
resolver = _section(snapshot, "resolver")
|
||||
resolver_via = _text(resolver.get("via")) if resolver is not None else None
|
||||
resolver_ts = _stamp(resolver.get("ts")) if resolver is not None else None
|
||||
payload["resolver"] = {"available": _available(resolver),
|
||||
"via": resolver_via,
|
||||
"entries": resolver_entries(resolver)}
|
||||
if not supports:
|
||||
# The 1.1.0 / --no-proxy arm: every key the page indexes is there and empty, every
|
||||
# count is null, and `why` is the one sentence. Nothing here is red — an agent that
|
||||
@@ -1035,6 +1339,13 @@ def parse(snapshot: Mapping[str, Any], host: str) -> dict[str, Any]:
|
||||
# consulted and is the same in the `sockets`-silent arm. Its per-line `connections` is filled
|
||||
# in below, and stays `None` for a target the config wrote as a name.
|
||||
payload["connections"] = config_connections(haproxy, nginx)
|
||||
# Phase 32's fourth section, applied between the graph and the join: the addresses the gateway
|
||||
# resolved **on the box that dials them**. Nothing here asks a resolver — this reads the
|
||||
# document the agent published — and a snapshot without one (1.2.0, `--no-resolve`) yields an
|
||||
# empty index, which leaves every name-written line exactly as phase 30 drew it: no address,
|
||||
# no count, no gauge, no sample.
|
||||
apply_resolution(payload["connections"], resolution_index(snapshot),
|
||||
resolver_via, resolver_ts)
|
||||
# Config integrity is config text too, for the same reason: a broken route is just as true when
|
||||
# the socket table never answered. Each half is judged only if its own section did answer, so
|
||||
# this is `[]` — not a guess — for a host with no haproxy, no nginx, or an unreadable file.
|
||||
@@ -1068,20 +1379,27 @@ def parse(snapshot: Mapping[str, Any], host: str) -> dict[str, Any]:
|
||||
payload["listeners"] = list(listeners.values())
|
||||
|
||||
# --- the endpoints the socket table can be joined to -----------------------
|
||||
# Only an endpoint the config wrote as an **address** is joinable: `/proc` spells addresses,
|
||||
# this stack resolves nothing, and a name is therefore a route with no measurable end. One
|
||||
# entry per `(address, port)` — several lines can reach one endpoint, and `routes` is what
|
||||
# makes `shared` true on every one of them afterwards.
|
||||
endpoints: dict[tuple[str, int], dict[str, Any]] = {}
|
||||
for connection in payload["connections"]:
|
||||
key = endpoint_of(connection)
|
||||
if key is None:
|
||||
# Joinable means the **kernel can name the far end**: an address the config wrote, or one the
|
||||
# gateway's own resolver published for the name it wrote. A name nobody resolved has no key and
|
||||
# therefore no count, no gauge and no sample. Several lines can reach one endpoint — two vhosts
|
||||
# naming one address, or two names resolving to it — and `_endpoint_groups` is what makes them
|
||||
# one counter row whose `routes` is what turns `shared` on afterwards.
|
||||
key_sets: list[set[tuple[str, int]]] = [set(endpoint_of(connection))
|
||||
for connection in payload["connections"]]
|
||||
groups = _endpoint_groups(key_sets)
|
||||
rows: list[dict[str, Any]] = []
|
||||
for group in groups:
|
||||
if group < 0:
|
||||
continue
|
||||
seen = endpoints.get(key)
|
||||
if seen is None:
|
||||
endpoints[key] = {"connections": 0, "routes": 1}
|
||||
else:
|
||||
seen["routes"] = int(seen["routes"]) + 1
|
||||
while len(rows) <= group:
|
||||
rows.append({"connections": 0, "routes": 0})
|
||||
rows[group]["routes"] = int(rows[group]["routes"]) + 1
|
||||
endpoints: dict[tuple[str, int], dict[str, Any]] = {}
|
||||
for keys, group in zip(key_sets, groups, strict=True):
|
||||
if group < 0:
|
||||
continue
|
||||
for key in keys:
|
||||
endpoints[key] = rows[group]
|
||||
|
||||
if not counting:
|
||||
# The configs are known and the live half is not: every line is drawn, every count stays
|
||||
@@ -1090,11 +1408,10 @@ def parse(snapshot: Mapping[str, Any], host: str) -> dict[str, Any]:
|
||||
|
||||
# --- the join: inbound by local port, outbound by exact address, nothing else --
|
||||
join_sockets(listeners, endpoints, established)
|
||||
for connection in payload["connections"]:
|
||||
key = endpoint_of(connection)
|
||||
if key is None:
|
||||
continue # a name, or an address with no port: never counted, so never a number
|
||||
endpoint = endpoints[key]
|
||||
for connection, group in zip(payload["connections"], groups, strict=True):
|
||||
if group < 0:
|
||||
continue # a name nobody resolved, or an address with no port: never counted
|
||||
endpoint = rows[group]
|
||||
connection["connections"] = int(endpoint["connections"])
|
||||
connection["shared"] = int(endpoint["routes"]) > 1
|
||||
payload["sockets"] = {"listens": len(listening), "established": len(established)}
|
||||
@@ -1107,15 +1424,20 @@ def samples(payload: Mapping[str, Any], ts: float | None = None) -> list[Sample]
|
||||
Three refusals, and each is a different fact:
|
||||
|
||||
* A `null` count publishes **nothing**. "Unknown" is not a zero, and a chart of zeros for a dead
|
||||
socket table — or of confident zeros for every route whose target the config wrote as a
|
||||
**name** — is the failure mode rule 2 names.
|
||||
socket table — or of confident zeros for a route whose target the config wrote as a **name**
|
||||
the gateway never resolved — is the failure mode rule 2 names. `0` is published, and only for
|
||||
an endpoint whose address is known while no socket is open to it.
|
||||
* **One sample per metric name.** Several route lines can reach one endpoint and each of them
|
||||
carries that endpoint's whole count (`shared: true`); the gauge is keyed by the endpoint, so
|
||||
emitting per line would store the same number twice under one name for one reading.
|
||||
* A gauge never gets a bool.
|
||||
|
||||
That is also why the name-keyed `gateway.out_conns@<domain>:<port>` series stopped being
|
||||
written in phase 30 task 02: the figure they carried belonged to the port, not the domain.
|
||||
Which lines have a metric at all is `apply_resolution`'s answer, not this function's: since
|
||||
phase 32 task 03 that is an endpoint the config wrote as an address **or** a name the gateway
|
||||
published addresses for, and the name-keyed `gateway.out_conns@<domain>:<port>` series phase 30
|
||||
task 02 stopped writing are written again because there is now a figure that belongs to the
|
||||
domain rather than to whatever else was listening on its port.
|
||||
|
||||
The timestamp is the poll's (when the dashboard read the count), carried on `ts`; nothing here
|
||||
is a rate, so nothing here needs a window.
|
||||
"""
|
||||
|
||||
@@ -133,7 +133,10 @@
|
||||
hanging off it on the right. This is the structure of the file, not a summary of it.
|
||||
A leaf reads inbound first: line 1 is the config's word for who reaches the endpoint
|
||||
(`server_name`, `map` key, `default`) and line 2 is the counterpart it flows to, so do
|
||||
not swap them back — the question this page answers is which domain goes where. -->
|
||||
not swap them back — the question this page answers is which domain goes where.
|
||||
`out.resolved` is the agent's own libc answer for a target the config wrote as a name,
|
||||
and it is why a name line can carry a number at all: without it the line is drawn and
|
||||
never counted. -->
|
||||
<div class="gate-map" data-k="map"></div>
|
||||
<div class="pipe-empty faint" data-k="empty" hidden></div>
|
||||
<!-- The third sentence a panel can need, and the only one the operator causes: this file
|
||||
@@ -193,9 +196,8 @@
|
||||
<p class="faint">a line is drawn because the config says the route exists, and it animates only
|
||||
while at least one connection is open on it — a resting line is not a dead one, and nothing
|
||||
here is coloured by a verdict: a busy ingress is the job.</p>
|
||||
<p class="faint">a dash means the endpoint is written in the config as a <b>name</b>; the page
|
||||
matches sockets by <b>address</b> and resolves nothing, so this route is drawn but not
|
||||
counted.</p>
|
||||
<p class="faint">a dash means the gateway could not resolve that name in this reading — hover a
|
||||
counted line for the address it matched and when that answer was taken.</p>
|
||||
<p class="faint">the switch shows what the config says; toggling a route needs a config write
|
||||
path and is not wired up yet.</p>
|
||||
<p class="faint">the filter box hides the routes whose own labels do not contain what you typed —
|
||||
|
||||
@@ -102,9 +102,58 @@ const CLIENTS_SHOWN = 2;
|
||||
/** The unknown as the leaf prints it: a dash, not a repeated column of identical words. */
|
||||
const NO_COUNT = '—';
|
||||
|
||||
/** Why a leaf has no number — the same sentence every such line gives, on `title`. */
|
||||
const NO_COUNT_WHY = 'not attributed: the config names this endpoint as a name, '
|
||||
+ 'and nothing here resolves names';
|
||||
/**
|
||||
* The two causes of a missing count, in the payload's own facts (phase 32 task 04).
|
||||
*
|
||||
* A `null` count used to mean one thing and say one thing. Since the gateway resolves the names it
|
||||
* reads it means one of **two**, and they are two different operator actions:
|
||||
*
|
||||
* * nothing is known for the name — `out.resolved` is `null` and the config wrote no address, so
|
||||
* there is no endpoint to match a socket against (`nxdomain`, `error`, or an agent whose
|
||||
* `resolver` section did not answer);
|
||||
* * an address **is** known and nothing was measured — the kernel's table went quiet, which is the
|
||||
* same fact the trunk prints as `n/a` beside its own `the socket table did not answer`.
|
||||
*
|
||||
* `countCause` picks between them from data and never guesses, and both surfaces quote the fragment
|
||||
* it returns, so the tooltip and the `aria-label` cannot drift. `0` is none of this: it is a
|
||||
* measurement, and it prints as one.
|
||||
*/
|
||||
const WHY_UNRESOLVED = 'the gateway could not resolve this name in this reading';
|
||||
const WHY_NO_SOCKETS = 'the socket table did not answer this reading';
|
||||
|
||||
/** Which of the two causes this line has — an address is "known" when the config wrote one or the
|
||||
* gateway published one, which is exactly the test `sources/gateway.py` joins on. Chosen from data:
|
||||
* a line is never told it has the other cause, which would send the operator to the wrong box. */
|
||||
function countCause(conn) {
|
||||
const out = conn.out || {};
|
||||
const placed = out.address === true || (out.resolved || []).length > 0;
|
||||
return placed ? WHY_NO_SOCKETS : WHY_UNRESOLVED;
|
||||
}
|
||||
|
||||
/** The dash's tooltip: the fact that no number is printed, plus the cause — one composition so the
|
||||
* row and the `aria-label` cannot drift from each other's sentence. */
|
||||
function countWhy(conn) {
|
||||
return `no number: ${countCause(conn)}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* The tooltip of a **counted** line: its number, and where the address behind it came from.
|
||||
*
|
||||
* The provenance is `out.via` (the agent's own word — `libc` — copied, never paraphrased) and the
|
||||
* age of `out.resolved_ts` through the shared age helpers. A line the config wrote as an address has
|
||||
* neither and says nothing extra: `out.address` is already the whole story, and inventing a lookup
|
||||
* for it would be rule 2. A missing `resolved_ts` costs the age fragment and nothing else, so the
|
||||
* sentence never prints `n/a` or `NaN` where an age belongs.
|
||||
*/
|
||||
function countTitle(conn, shown) {
|
||||
const out = conn.out || {};
|
||||
const head = `${shown} open connection(s) on this endpoint`;
|
||||
const via = typeof out.via === 'string' && out.via ? out.via : null;
|
||||
if (out.resolved == null || via == null) return head;
|
||||
const age = out.resolved_ts == null ? null : duration(Dash.ageSeconds(out.resolved_ts));
|
||||
return age == null ? `${head} · address from ${via}`
|
||||
: `${head} · address from ${via}, resolved ${age}`;
|
||||
}
|
||||
|
||||
/** The sentence a trunk with nothing hanging off it gets instead of an empty leaf list. */
|
||||
const NO_TARGET = 'no target named in the config';
|
||||
@@ -166,7 +215,8 @@ function groupsFor(snapshot, key) {
|
||||
*/
|
||||
function haystack(conn) {
|
||||
const out = conn.out || {};
|
||||
return [String((conn.in || {}).port ?? ''), out.host, out.name, out.kind, ...(out.selectors || [])]
|
||||
return [String((conn.in || {}).port ?? ''), out.host, out.name, out.kind,
|
||||
...(out.selectors || []), ...(out.resolved || [])]
|
||||
.filter((s) => typeof s === 'string' && s).join(' ').toLowerCase();
|
||||
}
|
||||
|
||||
@@ -243,7 +293,8 @@ function counterpartText(out) {
|
||||
*
|
||||
* Both of the row's lines are ellipsised by CSS at a phone width (AGENTS.md gap 13), so this is the
|
||||
* version that never truncates: the whole endpoint, **every** inbound selector, the inbound port, and
|
||||
* whether the count is a measurement or a `—`.
|
||||
* whether the count is a measurement or a `—` — and when it is a `—`, **which** of the two unknowns
|
||||
* it is, because "the name has no address" and "nobody measured" are two fixes (phase 32 task 04).
|
||||
*/
|
||||
function leafAria(conn, on) {
|
||||
const out = conn.out || {};
|
||||
@@ -255,7 +306,7 @@ function leafAria(conn, on) {
|
||||
// The state word is what the green *means*. A reader who cannot see the colour gets the same
|
||||
// answer it gives, and it is an existence sentence, never a verdict: "in the config", not "good".
|
||||
bits.push(on ? 'in the config' : 'not in the config');
|
||||
bits.push(conn.connections == null ? 'connections not attributed'
|
||||
bits.push(conn.connections == null ? `connections not attributed: ${countCause(conn)}`
|
||||
: `${conn.connections} live connections`);
|
||||
return bits.join(', ');
|
||||
}
|
||||
@@ -439,12 +490,13 @@ function renderLeaf(li, conn) {
|
||||
counterpart.hidden = inbound == null;
|
||||
if (inbound != null) setTxt(counterpart, counterpartText(conn.out));
|
||||
|
||||
// A `—` is not a measurement of zero and the tooltip says which of the two it is, so the column
|
||||
// of dashes down a `:443` fan-out reads as "these targets are names" rather than "idle".
|
||||
// A `—` is not a measurement of zero and the tooltip says which of the two unknowns it is, so a
|
||||
// column of them down a `:443` fan-out reads as "no address for these targets" rather than "idle" —
|
||||
// and a `0`, which is a measurement, keeps its own sentence plus the address it was measured to.
|
||||
const shown = countText(conn.connections);
|
||||
const count = li.querySelector('.pipe-count');
|
||||
setTxt(count, shown == null ? NO_COUNT : shown);
|
||||
count.title = shown == null ? NO_COUNT_WHY : `${shown} open connection(s) on this endpoint`;
|
||||
count.title = shown == null ? countWhy(conn) : countTitle(conn, shown);
|
||||
|
||||
const svg = li.querySelector('.pipe');
|
||||
const live = typeof conn.connections === 'number' && isFinite(conn.connections)
|
||||
|
||||
+19
-3
@@ -257,7 +257,7 @@ GATEWAY_TREE = {
|
||||
"proc/net/tcp6": "gateway_proc_net_tcp6.txt",
|
||||
}
|
||||
|
||||
GATEWAY_MODES = ("full", "predates", "no_haproxy", "no_nginx", "sockets_silent")
|
||||
GATEWAY_MODES = ("full", "resolved", "predates", "no_haproxy", "no_nginx", "sockets_silent")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
@@ -269,9 +269,19 @@ def gateway_snapshot(host_agent_module: Any, tmp_path: Path) -> Any:
|
||||
`sources/gateway.py` are the bytes agent 1.2.0 serves. A field rename on either side of the
|
||||
wire therefore fails a source test instead of drawing an empty pipe on the page (rule 7).
|
||||
|
||||
Modes are the degradation arms the page has to render:
|
||||
One more axis is a **version**, not a degradation:
|
||||
|
||||
* `full` — all three sections `available: true`, live sockets from the capture
|
||||
* `resolved` — the same tree as `full` plus agent 1.3.0's fourth section, the recorded
|
||||
`gateway_resolver.json` attached verbatim (the document `capture-gateway-config.sh` reads
|
||||
back with `getent ahostsv4/6` on the box that dials). This is what the deployed ingress
|
||||
serves after phase 32 task 05, and it is the arm in which a name-written route becomes
|
||||
countable.
|
||||
|
||||
Modes are otherwise the degradation arms the page has to render:
|
||||
|
||||
* `full` — all three sections `available: true`, live sockets from the capture, and **no**
|
||||
`resolver` key at all: a 1.2.0 agent, or a 1.3.0 one run with `--no-resolve`. Phase 30's
|
||||
payload, verbatim, which is what the phase-32 additivity pins compare against
|
||||
* `predates` — none of the three keys at all (an agent older than 1.2.0, or `--no-proxy`)
|
||||
* `no_haproxy` / `no_nginx` — that one config file is absent: `available: false` plus the
|
||||
agent's own **not found** line, the other half of the page untouched
|
||||
@@ -308,6 +318,12 @@ def gateway_snapshot(host_agent_module: Any, tmp_path: Path) -> Any:
|
||||
snapshot["haproxy"] = collector.haproxy(errors)
|
||||
snapshot["nginx"] = collector.nginx(errors)
|
||||
snapshot["sockets"] = collector.sockets(errors)
|
||||
if mode == "resolved":
|
||||
# The recorded capture, not a re-derivation: the names are the ones `target_names()`
|
||||
# derives from the config fixtures above and the answers are libc's on the router, so
|
||||
# the join is tested against the document the agent will actually serve (rule 7). The
|
||||
# dashboard never asks a resolver, and nothing here does either.
|
||||
snapshot["resolver"] = json.loads((FIXTURES / "gateway_resolver.json").read_text())
|
||||
return snapshot
|
||||
|
||||
return build
|
||||
|
||||
@@ -16,6 +16,7 @@ map $ssl_preread_server_name $targetBackend {
|
||||
brain.reeseapps.com reeseapps-brain-of-reese.reeselink.com:443;
|
||||
brain-of-code.reeseapps.com reeseapps-brain-of-code.reeselink.com:443;
|
||||
immich.reeseapps.com reeseapps-immich.reeselink.com:443;
|
||||
whiteboard.reeseapps.com reeseapps-whiteboard.reeselink.com:443;
|
||||
|
||||
stackexpected.com stackexpected.reeselink.com:443;
|
||||
gitea.stackexpected.com stackexpected-gitea.reeselink.com:443;
|
||||
|
||||
+72
-45
@@ -11,48 +11,75 @@
|
||||
# a diff here on re-capture is expected. The tests assert shape, the two states the agent
|
||||
# keeps, and the ports the captured configs claim to bind — never a particular row.
|
||||
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
|
||||
0: 00000000:20FB 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9978 1 0000000000000000 100 0 0 10 0
|
||||
1: 00000000:08AE 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7797 1 0000000000000000 100 0 0 10 0
|
||||
2: 00000000:0050 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9974 1 0000000000000000 100 0 0 10 0
|
||||
3: 00000000:01BB 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9972 1 0000000000000000 100 0 0 10 0
|
||||
4: 00000000:115B 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9976 1 0000000000000000 100 0 0 10 0
|
||||
5: 1B01010A:23A0 00000000:0000 0A 00000000:00000000 00:00000000 00000000 994 0 9479 1 0000000000000000 100 0 0 10 0
|
||||
6: 1B01010A:BE2A B001010A:01BB 06 00000000:00000000 03:000013F5 00000000 0 0 0 3 0000000000000000
|
||||
7: 0100007F:115B 0100007F:818C 01 00000000:00000000 00:00000000 00000000 101 0 62860 1 0000000000000000 20 4 30 10 -1
|
||||
8: 1B01010A:BE2E B001010A:01BB 06 00000000:00000000 03:00001407 00000000 0 0 0 3 0000000000000000
|
||||
9: 1B01010A:23A0 0204A8C0:C626 06 00000000:00000000 03:00000820 00000000 0 0 0 3 0000000000000000
|
||||
10: 0100007F:115B 0100007F:C096 01 00000000:00000000 00:00000000 00000000 101 0 49558 1 0000000000000000 20 4 26 10 -1
|
||||
11: 0100007F:115B 0100007F:B502 06 00000000:00000000 03:00000841 00000000 0 0 0 3 0000000000000000
|
||||
12: 1B01010A:8290 A401010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 11850 1 0000000000000000 20 4 28 10 -1
|
||||
13: 1B01010A:01BB 017100CB:90FA 01 00000000:00000000 00:00000000 00000000 101 0 21476 1 0000000000000000 35 4 31 10 -1
|
||||
14: 1B01010A:23A0 0204A8C0:992E 06 00000000:00000000 03:00000C0A 00000000 0 0 0 3 0000000000000000
|
||||
15: 1B01010A:CB4C B001010A:01BB 06 00000000:00000000 03:00000ECB 00000000 0 0 0 3 0000000000000000
|
||||
16: 1B01010A:9C42 2700010A:01BB 06 00000000:00000000 03:00001259 00000000 0 0 0 3 0000000000000000
|
||||
17: 1B01010A:01BB 0100010A:B2AC 01 00000000:00000000 00:00000000 00000000 101 0 77631 1 0000000000000000 20 4 31 10 -1
|
||||
18: 1B01010A:01BB E600020A:BBE8 06 00000000:00000000 03:00000992 00000000 0 0 0 3 0000000000000000
|
||||
19: 0100007F:C99E 0100007F:115B 06 00000000:00000000 03:000016BE 00000000 0 0 0 3 0000000000000000
|
||||
20: 1B01010A:C1FE 2700010A:01BB 06 00000000:00000000 03:00000662 00000000 0 0 0 3 0000000000000000
|
||||
21: 1B01010A:01BB E600020A:B5C4 06 00000000:00000000 03:000005EE 00000000 0 0 0 3 0000000000000000
|
||||
22: 1B01010A:840E B001010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 21477 1 0000000000000000 20 4 30 10 -1
|
||||
23: 1B01010A:D22A 4E01010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 77632 1 0000000000000000 21 4 30 10 -1
|
||||
24: 1B01010A:01BB 017100CB:956A 01 00000000:00000000 00:00000000 00000000 101 0 62858 1 0000000000000000 29 4 30 10 -1
|
||||
25: 0100007F:115B 0100007F:B4F6 06 00000000:00000000 03:000005ED 00000000 0 0 0 3 0000000000000000
|
||||
26: 1B01010A:23A0 0204A8C0:8EE6 06 00000000:00000000 03:00001408 00000000 0 0 0 3 0000000000000000
|
||||
27: 1B01010A:01BB EC01010A:94DE 01 00000000:00000000 00:00000000 00000000 101 0 11849 1 0000000000000000 20 4 31 10 -1
|
||||
28: 0100007F:818C 0100007F:115B 01 00000000:00000000 00:00000000 00000000 101 0 62859 1 0000000000000000 21 4 24 10 -1
|
||||
29: 1B01010A:23A0 0204A8C0:D5DE 06 00000000:00000000 03:0000020F 00000000 0 0 0 3 0000000000000000
|
||||
30: 0100007F:115B 0100007F:B4F8 06 00000000:00000000 03:000005F6 00000000 0 0 0 3 0000000000000000
|
||||
31: 1B01010A:A828 7600010A:63DD 06 00000000:00000000 03:00000F7F 00000000 0 0 0 3 0000000000000000
|
||||
32: 0100007F:C096 0100007F:115B 01 00000000:00000000 00:00000000 00000000 101 0 49557 1 0000000000000000 20 4 24 10 -1
|
||||
33: 1B01010A:D444 FE00010A:2AF8 01 00000000:00000000 00:00000000 00000000 101 0 49559 1 0000000000000000 20 4 30 10 -1
|
||||
34: 1B01010A:9032 B001010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 11954 1 0000000000000000 20 4 30 10 -1
|
||||
35: 1B01010A:977E 2700010A:01BB 06 00000000:00000000 03:00000C72 00000000 0 0 0 3 0000000000000000
|
||||
36: 0100007F:115B 0100007F:B4EA 06 00000000:00000000 03:000005F0 00000000 0 0 0 3 0000000000000000
|
||||
37: 1B01010A:08AE 0204A8C0:8290 01 00000048:00000000 01:00000017 00000000 0 0 86478 4 0000000000000000 24 8 15 10 -1
|
||||
38: 1B01010A:9ED0 2700010A:01BB 06 00000000:00000000 03:0000170F 00000000 0 0 0 3 0000000000000000
|
||||
39: 1B01010A:0050 027100CB:843E 01 0000018C:00000000 01:00000E57 00000007 101 0 84306 2 0000000000000000 4864 4 31 1 7
|
||||
40: 1B01010A:CB5C B001010A:01BB 06 00000000:00000000 03:00000ED6 00000000 0 0 0 3 0000000000000000
|
||||
41: 1B01010A:9776 2700010A:01BB 06 00000000:00000000 03:00000875 00000000 0 0 0 3 0000000000000000
|
||||
42: 1B01010A:C59C FE00010A:2AF8 01 00000000:00000000 00:00000000 00000000 101 0 62861 1 0000000000000000 21 4 30 10 -1
|
||||
43: 1B01010A:9154 2700010A:01BB 06 00000000:00000000 03:00001572 00000000 0 0 0 3 0000000000000000
|
||||
44: 1B01010A:23A0 0204A8C0:9C42 06 00000000:00000000 03:00000FF3 00000000 0 0 0 3 0000000000000000
|
||||
0: 1B01010A:23A0 00000000:0000 0A 00000000:00000000 00:00000000 00000000 994 0 9291 1 0000000000000000 100 0 0 10 0
|
||||
1: 00000000:20FB 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9804 1 0000000000000000 100 0 0 10 0
|
||||
2: 00000000:08AE 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 8256 1 0000000000000000 100 0 0 10 0
|
||||
3: 00000000:0050 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9800 1 0000000000000000 100 0 0 10 0
|
||||
4: 00000000:01BB 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9798 1 0000000000000000 100 0 0 10 0
|
||||
5: 00000000:115B 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9802 1 0000000000000000 100 0 0 10 0
|
||||
6: 1B01010A:E27A 7600010A:63DD 06 00000000:00000000 03:00000D25 00000000 0 0 0 3 0000000000000000
|
||||
7: 1B01010A:23A0 0204A8C0:A726 06 00000000:00000000 03:00000082 00000000 0 0 0 3 0000000000000000
|
||||
8: 1B01010A:90A0 A401010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 788517 1 0000000000000000 21 4 31 10 -1
|
||||
9: 1B01010A:C76A 2700010A:01BB 06 00000000:00000000 03:00001004 00000000 0 0 0 3 0000000000000000
|
||||
10: 1B01010A:23A0 0204A8C0:A9EA 06 00000000:00000000 03:0000141C 00000000 0 0 0 3 0000000000000000
|
||||
11: 1B01010A:AC88 2700010A:01BB 06 00000000:00000000 03:00001745 00000000 0 0 0 3 0000000000000000
|
||||
12: 1B01010A:01BB 0204A8C0:D28C 01 00000000:00000000 00:00000000 00000000 101 0 470456 1 0000000000000000 23 4 31 2 2
|
||||
13: 1B01010A:AFCE 2700010A:01BB 06 00000000:00000000 03:00000CA6 00000000 0 0 0 3 0000000000000000
|
||||
14: 1B01010A:EA9C 6A01010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 894482 1 0000000000000000 20 4 30 10 -1
|
||||
15: 1B01010A:01BB 0204A8C0:B050 01 00000000:00000000 00:00000000 00000000 101 0 901901 1 0000000000000000 20 4 31 10 -1
|
||||
16: 1B01010A:C778 2700010A:01BB 06 00000000:00000000 03:00001077 00000000 0 0 0 3 0000000000000000
|
||||
17: 1B01010A:01BB E600020A:C768 06 00000000:00000000 03:00000FBA 00000000 0 0 0 3 0000000000000000
|
||||
18: 1B01010A:881A 2700010A:01BB 06 00000000:00000000 03:0000140B 00000000 0 0 0 3 0000000000000000
|
||||
19: 1B01010A:C750 2700010A:01BB 06 00000000:00000000 03:00000E46 00000000 0 0 0 3 0000000000000000
|
||||
20: 1B01010A:AFBE 2700010A:01BB 06 00000000:00000000 03:00000B6B 00000000 0 0 0 3 0000000000000000
|
||||
21: 1B01010A:C74A 2700010A:01BB 06 00000000:00000000 03:00000D8B 00000000 0 0 0 3 0000000000000000
|
||||
22: 1B01010A:E6AE B001010A:01BB 06 00000000:00000000 03:00000664 00000000 0 0 0 3 0000000000000000
|
||||
23: 1B01010A:EDD0 2700010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 903407 1 0000000000000000 20 4 32 10 -1
|
||||
24: 1B01010A:C916 FE00010A:2AF8 01 00000000:00000000 00:00000000 00000000 101 0 470459 1 0000000000000000 22 4 30 10 -1
|
||||
25: 1B01010A:08AE 0204A8C0:9D44 01 00000000:00000000 02:000AFC6F 00000000 0 0 903449 4 0000000000000000 20 4 17 10 -1
|
||||
26: 1B01010A:D958 4E01010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 890783 1 0000000000000000 21 4 30 10 -1
|
||||
27: 1B01010A:87F6 2700010A:01BB 06 00000000:00000000 03:00001217 00000000 0 0 0 3 0000000000000000
|
||||
28: 0100007F:BC28 0100007F:115B 01 00000000:00000000 00:00000000 00000000 101 0 879324 1 0000000000000000 20 4 24 10 -1
|
||||
29: 1B01010A:EDC0 2700010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 901902 1 0000000000000000 20 4 33 10 -1
|
||||
30: 1B01010A:01BB 0204A8C0:94D8 01 00000000:00000000 00:00000000 00000000 101 0 887225 1 0000000000000000 20 4 26 7 7
|
||||
31: 1B01010A:880A 2700010A:01BB 06 00000000:00000000 03:000016C8 00000000 0 0 0 3 0000000000000000
|
||||
32: 1B01010A:01BB 0204A8C0:B05E 01 00000000:00000000 00:00000000 00000000 101 0 902458 1 0000000000000000 20 4 14 10 -1
|
||||
33: 1B01010A:CEC4 B001010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 684979 1 0000000000000000 20 4 30 10 -1
|
||||
34: 1B01010A:CB90 DF00010A:1451 02 00000001:00000000 01:00000022 00000000 99 0 903275 2 0000000000000000 100 0 1 10 -1
|
||||
35: 1B01010A:AC7A 2700010A:01BB 06 00000000:00000000 03:000016F1 00000000 0 0 0 3 0000000000000000
|
||||
36: 1B01010A:23A0 0204A8C0:994E 06 00000000:00000000 03:0000046B 00000000 0 0 0 3 0000000000000000
|
||||
37: 0100007F:C45A 0100007F:115B 06 00000000:00000000 03:00000C8A 00000000 0 0 0 3 0000000000000000
|
||||
38: 0100007F:115B 0100007F:C454 06 00000000:00000000 03:00000D86 00000000 0 0 0 3 0000000000000000
|
||||
39: 1B01010A:8140 6A01010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 887226 1 0000000000000000 20 4 32 10 -1
|
||||
40: 1B01010A:01BB 0204A8C0:C044 06 00000000:00000000 03:00000839 00000000 0 0 0 3 0000000000000000
|
||||
41: 1B01010A:9A6C 7600010A:63DE 06 00000000:00000000 03:00000CC8 00000000 0 0 0 3 0000000000000000
|
||||
42: 1B01010A:8828 2700010A:01BB 06 00000000:00000000 03:00001552 00000000 0 0 0 3 0000000000000000
|
||||
43: 1B01010A:C184 FE00010A:2AF8 01 00000000:00000000 00:00000000 00000000 101 0 880000 1 0000000000000000 20 4 30 10 -1
|
||||
44: 1B01010A:01BB 0204A8C0:B052 01 00000000:00000000 00:00000000 00000000 101 0 903348 1 0000000000000000 21 4 31 10 -1
|
||||
45: 1B01010A:EDD8 2700010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 902459 1 0000000000000000 20 4 33 43 -1
|
||||
46: 1B01010A:01BB 0100010A:AD60 01 00000000:00000000 00:00000000 00000000 101 0 890782 1 0000000000000000 20 4 31 4 4
|
||||
47: 1B01010A:01BB E600020A:C770 06 00000000:00000000 03:00000EC1 00000000 0 0 0 3 0000000000000000
|
||||
48: 1B01010A:23A0 0204A8C0:B316 06 00000000:00000000 03:00001035 00000000 0 0 0 3 0000000000000000
|
||||
49: 0100007F:115B 0100007F:BC28 01 00000000:00000000 00:00000000 00000000 101 0 879999 1 0000000000000000 20 4 26 10 -1
|
||||
50: 1B01010A:01BB E600020A:A480 06 00000000:00000000 03:00000C14 00000000 0 0 0 3 0000000000000000
|
||||
51: 1B01010A:8804 2700010A:01BB 06 00000000:00000000 03:0000126B 00000000 0 0 0 3 0000000000000000
|
||||
52: 1B01010A:23A0 0204A8C0:B31E 06 00000000:00000000 03:00001036 00000000 0 0 0 3 0000000000000000
|
||||
53: 0100007F:115B 0100007F:C436 06 00000000:00000000 03:00000C14 00000000 0 0 0 3 0000000000000000
|
||||
54: 1B01010A:D16A B001010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 464535 1 0000000000000000 20 4 30 10 -1
|
||||
55: 1B01010A:E490 6A01010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 853018 1 0000000000000000 20 4 32 10 -1
|
||||
56: 1B01010A:C756 2700010A:01BB 06 00000000:00000000 03:00000E62 00000000 0 0 0 3 0000000000000000
|
||||
57: 1B01010A:01BB 0204A8C0:9FA2 01 00000000:00000000 00:00000000 00000000 101 0 853017 1 0000000000000000 20 4 30 7 7
|
||||
58: 0100007F:115B 0100007F:BA26 01 00000000:00000000 00:00000000 00000000 101 0 470458 1 0000000000000000 21 4 31 10 -1
|
||||
59: 1B01010A:01BB 0303A8C0:8B88 01 00000000:00000000 00:00000000 00000000 101 0 684978 1 0000000000000000 30 4 30 3 2
|
||||
60: 1B01010A:A132 2700010A:01BB 06 00000000:00000000 03:00000BF8 00000000 0 0 0 3 0000000000000000
|
||||
61: 0100007F:115B 0100007F:C446 06 00000000:00000000 03:00000C1D 00000000 0 0 0 3 0000000000000000
|
||||
62: 1B01010A:01BB 0204A8C0:BAFA 01 00000000:00000000 00:00000000 00000000 101 0 894481 1 0000000000000000 20 4 31 7 7
|
||||
63: 0100007F:115B 0100007F:C456 06 00000000:00000000 03:00000DA6 00000000 0 0 0 3 0000000000000000
|
||||
64: 1B01010A:B862 B001010A:01BB 01 00000000:00000000 00:00000000 00000000 101 0 705498 1 0000000000000000 20 4 31 10 -1
|
||||
65: 1B01010A:23A0 0204A8C0:8618 06 00000000:00000000 03:00000C3A 00000000 0 0 0 3 0000000000000000
|
||||
66: 0100007F:BA26 0100007F:115B 01 00000000:00000000 00:00000000 00000000 101 0 470457 1 0000000000000000 21 4 22 10 -1
|
||||
67: 1B01010A:23A0 0204A8C0:AD66 06 00000000:00000000 03:00000852 00000000 0 0 0 3 0000000000000000
|
||||
68: 1B01010A:23A0 0204A8C0:9952 06 00000000:00000000 03:0000046B 00000000 0 0 0 3 0000000000000000
|
||||
69: 1B01010A:DC6A 2700010A:01BB 06 00000000:00000000 03:000009C9 00000000 0 0 0 3 0000000000000000
|
||||
70: 1B01010A:01BB EC01010A:953E 01 00000000:00000000 00:00000000 00000000 101 0 788516 1 0000000000000000 20 4 31 10 -1
|
||||
71: 1B01010A:E6AC B001010A:01BB 06 00000000:00000000 03:0000065D 00000000 0 0 0 3 0000000000000000
|
||||
|
||||
+14
-13
@@ -11,16 +11,17 @@
|
||||
# a diff here on re-capture is expected. The tests assert shape, the two states the agent
|
||||
# keeps, and the ports the captured configs claim to bind — never a particular row.
|
||||
sl local_address remote_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
|
||||
0: 00000000000000000000000000000000:20FB 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9979 1 0000000000000000 100 0 0 10 0
|
||||
1: 00000000000000000000000000000000:08AE 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7799 1 0000000000000000 100 0 0 10 0
|
||||
2: 00000000000000000000000000000000:1451 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9181 1 0000000000000000 100 0 0 10 0
|
||||
3: 00000000000000000000000000000000:0050 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9975 1 0000000000000000 100 0 0 10 0
|
||||
4: 00000000000000000000000000000000:0016 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9179 1 0000000000000000 100 0 0 10 0
|
||||
5: 00000000000000000000000000000000:01BB 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9973 1 0000000000000000 100 0 0 10 0
|
||||
6: 00000000000000000000000000000000:115B 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9977 1 0000000000000000 100 0 0 10 0
|
||||
7: 00000000000000000000000000000000:63DE 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9184 1 0000000000000000 100 0 0 10 0
|
||||
8: 00000000000000000000000000000000:63DD 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9183 1 0000000000000000 100 0 0 10 0
|
||||
9: 00000000000000000000000000000000:07D0 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9180 1 0000000000000000 100 0 0 10 0
|
||||
10: 00000000000000000000000000000000:1F90 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9182 1 0000000000000000 100 0 0 10 0
|
||||
11: B80D0120000000000000000001000000:01BB 2377D3FDF349E0970000000003000000:C0AC 01 00000000:00000000 00:00000000 00000000 101 0 49556 1 0000000000000000 80 4 26 10 -1
|
||||
12: B80D0120000000000000000001000000:01BB 2377D3FDF349E0970000000003000000:A120 01 00000000:00000000 00:00000000 00000000 101 0 11953 1 0000000000000000 59 4 30 7 7
|
||||
0: 00000000000000000000000000000000:07D0 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9105 1 0000000000000000 100 0 0 10 0
|
||||
1: 00000000000000000000000000000000:63DD 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9108 1 0000000000000000 100 0 0 10 0
|
||||
2: 00000000000000000000000000000000:63DE 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9109 1 0000000000000000 100 0 0 10 0
|
||||
3: 00000000000000000000000000000000:1F90 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9107 1 0000000000000000 100 0 0 10 0
|
||||
4: 00000000000000000000000000000000:20FB 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9805 1 0000000000000000 100 0 0 10 0
|
||||
5: 00000000000000000000000000000000:08AE 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 8264 1 0000000000000000 100 0 0 10 0
|
||||
6: 00000000000000000000000000000000:0050 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9801 1 0000000000000000 100 0 0 10 0
|
||||
7: 00000000000000000000000000000000:1451 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9106 1 0000000000000000 100 0 0 10 0
|
||||
8: 00000000000000000000000000000000:0016 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 9104 1 0000000000000000 100 0 0 10 0
|
||||
9: 00000000000000000000000000000000:01BB 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9799 1 0000000000000000 100 0 0 10 0
|
||||
10: 00000000000000000000000000000000:115B 00000000000000000000000000000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 9803 1 0000000000000000 100 0 0 10 0
|
||||
11: B80D0120000000000000000001000000:01BB B80D0120000000000000000002000000:D76A 01 00000000:00000000 00:00000000 00000000 101 0 705497 1 0000000000000000 20 4 30 10 -1
|
||||
12: B80D0120000000000000000001000000:01BB B80D0120000000000000000003000000:9FBA 01 00000000:00000000 00:00000000 00000000 101 0 464534 1 0000000000000000 57 4 30 3 2
|
||||
13: B80D0120000000000000000001000000:01BB B80D0120000000000000000003000000:E0C6 01 00000000:00000000 00:00000000 00000000 101 0 879323 1 0000000000000000 42 4 26 10 -1
|
||||
|
||||
Vendored
+297
@@ -0,0 +1,297 @@
|
||||
{
|
||||
"available": true,
|
||||
"via": "libc",
|
||||
"config_directive": "10.1.0.1",
|
||||
"ts": 1792886400.0,
|
||||
"entries": [
|
||||
{
|
||||
"name": "reeseapps-gitea.reeselink.com",
|
||||
"port": 2222,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.1.164"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "stackexpected-gitea.reeselink.com",
|
||||
"port": 2222,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.1.78"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "web.reeselink.com",
|
||||
"port": 5201,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.0.223"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "unifi-external.reeselink.com",
|
||||
"port": 8080,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.0.222"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "minecraft.reeselink.com",
|
||||
"port": 25565,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.0.118"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "minecraft.reeselink.com",
|
||||
"port": 25566,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.0.118"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "nextcloud.reeselink.com",
|
||||
"port": 11000,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.0.254"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "nextcloud.reeselink.com",
|
||||
"port": 11001,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.0.254"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "gitea.reeselink.com",
|
||||
"port": 3000,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.0.137"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "jellyfin.reeselink.com",
|
||||
"port": 8096,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.0.75"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "127.0.0.1",
|
||||
"port": 8081,
|
||||
"status": "literal",
|
||||
"addresses": [
|
||||
"127.0.0.1"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "bricktracker.reeselink.com",
|
||||
"port": 3333,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.0.66"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "keycloak.reeselink.com",
|
||||
"port": 8080,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.1.75"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "emmaleaf.reeselink.com",
|
||||
"port": 80,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.0.138"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "127.0.0.1",
|
||||
"port": 4443,
|
||||
"status": "literal",
|
||||
"addresses": [
|
||||
"127.0.0.1"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "reeseapps-homepage.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.1.33"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "reeseapps-ntfy.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.1.176"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "reeseapps-gitea.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.1.164"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "reeseapps-litellm.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.0.39"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "reeseapps-openwebui.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.2.39"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "reeseapps-keycloak.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.3.148"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "reeseapps-forgejo.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.3.189"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "reeseapps-brain-of-reese.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.2.64"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "reeseapps-brain-of-code.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.0.106"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "reeseapps-immich.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.3.159"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "reeseapps-whiteboard.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.2.79"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "stackexpected.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.3.64"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "stackexpected-gitea.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.1.78"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "stackexpected-keycloak.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.0.42"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "stackexpected-nextcloud.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.1.106"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "stackexpected-hedgedoc.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.3.142"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
},
|
||||
{
|
||||
"name": "wfc.reeselink.com",
|
||||
"port": 443,
|
||||
"status": "ok",
|
||||
"addresses": [
|
||||
"10.1.1.41"
|
||||
],
|
||||
"ts": 1792886400.0
|
||||
}
|
||||
],
|
||||
"errors": []
|
||||
}
|
||||
+712
-15
@@ -12,6 +12,7 @@ Four things are asserted here:
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import http.client
|
||||
import importlib.util
|
||||
import ipaddress
|
||||
@@ -331,13 +332,14 @@ def write_fake_disk(root: Path, name: str, model: str | None = None) -> None:
|
||||
write_file(root, f"sys/block/{name}/device/model", f"{model}\n")
|
||||
|
||||
|
||||
#: The add-on sections report their own absence in `errors` (`smart:`, `haproxy:`), because
|
||||
#: "this box has no smartmontools / no haproxy" is a fact the page that owns the add-on has to
|
||||
#: be able to print. Tests that assert a snapshot is otherwise clean filter the same prefixes
|
||||
#: the dashboard does — `collectors.ADDON_ERROR_PREFIXES`, pinned equal to this list by
|
||||
#: `test_the_addon_error_prefixes_are_the_same_list_the_dashboard_filters`, so neither side can
|
||||
#: The add-on sections report their own absence in `errors` (`smart:`, `haproxy:`, `resolver:`),
|
||||
#: because
|
||||
#: "this box has no smartmontools / no haproxy / nothing to resolve" is a fact the page that owns
|
||||
#: the add-on has to be able to print. Tests that assert a snapshot is otherwise clean filter the
|
||||
#: same prefixes the dashboard does — `collectors.ADDON_ERROR_PREFIXES`, pinned equal to this list
|
||||
#: by `test_the_addon_error_prefixes_are_the_same_list_the_dashboard_filters`, so neither side can
|
||||
#: quietly grow a fourth opinion.
|
||||
ADDON_ERROR_PREFIXES = ("smart", "haproxy", "nginx", "sockets")
|
||||
ADDON_ERROR_PREFIXES = ("smart", "haproxy", "nginx", "sockets", "resolver")
|
||||
|
||||
|
||||
def test_the_addon_error_prefixes_are_the_same_list_the_dashboard_filters() -> None:
|
||||
@@ -357,6 +359,36 @@ def core_errors(snapshot: dict[str, Any]) -> list[str]:
|
||||
if not str(line).lstrip().lower().startswith(ADDON_ERROR_PREFIXES)]
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def no_real_dns(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Not one test in this file resolves a name across the network (phase 32's hermetic mandate).
|
||||
|
||||
The patch is module-wide on purpose — `host_agent.socket` *is* the `socket` module — and the
|
||||
stub answers a hostname with a deterministic pair of **documentation-space** addresses
|
||||
(RFC 5737 / RFC 3849, the same substitution `scripts/capture-gateway-config.sh` makes to a
|
||||
socket table, so nothing a resolver test prints could carry a real endpoint either). Every IP
|
||||
literal and `localhost` goes to the real function: this file's HTTP and TLS tests connect to
|
||||
numeric loopback, and a stub that answered those too would break them for a reason nobody is
|
||||
testing. A test that wants its own answer — `nxdomain`, a slow resolver, a call ledger —
|
||||
patches the same attribute in its own body, which runs after this fixture.
|
||||
"""
|
||||
real = socket.getaddrinfo
|
||||
|
||||
def stub(name: Any, port: Any = None, *args: Any, **kwargs: Any) -> Any:
|
||||
text = str(name)
|
||||
if host_agent.address_family(text) is not None or text in ("", "localhost"):
|
||||
return real(name, port, *args, **kwargs)
|
||||
digest = hashlib.sha256(text.encode("utf-8")).hexdigest()
|
||||
number = int(digest[:4], 16)
|
||||
host_port = port if isinstance(port, int) else 0
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 0, "",
|
||||
(f"203.0.113.{number % 256}", host_port)),
|
||||
(socket.AF_INET6, socket.SOCK_STREAM, 0, "",
|
||||
(f"2001:db8::{digest[4:8]}", host_port, 0, 0))]
|
||||
|
||||
monkeypatch.setattr(host_agent.socket, "getaddrinfo", stub)
|
||||
|
||||
|
||||
def fake_calls(tmp_path: Path) -> list[str]:
|
||||
called = tmp_path / "smart-out" / "_called"
|
||||
return called.read_text().split() if called.exists() else []
|
||||
@@ -1585,11 +1617,16 @@ def test_a_corrupt_nginx_config_leaves_the_rest_of_the_snapshot_intact(tmp_path:
|
||||
def test_an_nginx_failure_inside_the_collector_still_yields_a_section(tmp_path: Path,
|
||||
monkeypatch) -> None:
|
||||
"""Whatever raises in here, the document keeps the key with `available: null` — a snapshot
|
||||
that loses the key would read as a pre-1.2.0 agent, not as a failed add-on."""
|
||||
that loses the key would read as a pre-1.2.0 agent, not as a failed add-on.
|
||||
|
||||
`nginx_document` is the seam `snapshot()` calls (it is the pair of the section and the
|
||||
`resolver` directive); `Collector.nginx` is its one-line wrapper, so patching the document is
|
||||
patching the real path a snapshot takes.
|
||||
"""
|
||||
def explode(*args: object) -> dict[str, Any]:
|
||||
raise RuntimeError("boom")
|
||||
|
||||
monkeypatch.setattr(host_agent.Collector, "nginx", explode)
|
||||
monkeypatch.setattr(host_agent.Collector, "nginx_document", explode)
|
||||
snap = host_agent.Collector(str(tmp_path / "root")).snapshot()
|
||||
assert snap["nginx"] == {"available": None, "config": host_agent.NGINX_CONFIG_PATH,
|
||||
"files": [], "servers": [], "upstreams": [], "maps": []}
|
||||
@@ -2767,15 +2804,13 @@ def test_the_tls_fields_are_documented_in_the_schema() -> None:
|
||||
assert "--cert-bundle" in (host_agent.__doc__ or "")
|
||||
|
||||
|
||||
def test_1_2_0_is_the_version_that_ships_the_three_proxy_sections() -> None:
|
||||
"""The version bump is the fleet's way of saying "this file has the gateway sections".
|
||||
def test_the_schema_names_the_three_proxy_sections() -> None:
|
||||
"""Phase 28's three sections are all in the SCHEMA block, each with its three-way `available`.
|
||||
|
||||
Phase 28 added three sections and one flag to the payload, so an install that predates them
|
||||
is distinguishable by one string: `--status`, `/health` and this test all read the same
|
||||
constant. The SCHEMA block is `GET /schema`'s body, so it names all three, each with the
|
||||
three-way `available` the dashboard reads, and `--no-proxy`, the flag that removes the family.
|
||||
The SCHEMA block is `GET /schema`'s body, so a deployment can read what an agent publishes
|
||||
from the agent itself. The version constant is pinned by the newest phase's test below — one
|
||||
place names the number, so a bump moves one assertion and not a file's worth of history.
|
||||
"""
|
||||
assert host_agent.AGENT_VERSION == "1.2.0"
|
||||
doc = host_agent.__doc__ or ""
|
||||
for section in ("haproxy", "nginx", "sockets"):
|
||||
assert schema_documents(section), f"{section} is not documented in the SCHEMA block"
|
||||
@@ -2784,6 +2819,24 @@ def test_1_2_0_is_the_version_that_ships_the_three_proxy_sections() -> None:
|
||||
assert "--no-proxy" in doc and "--haproxy-config" in doc and "--nginx-config" in doc
|
||||
|
||||
|
||||
def test_the_agent_version_is_1_3_0_and_the_schema_names_the_resolver_section() -> None:
|
||||
"""1.3.0 is the version that resolves the names the configs declare.
|
||||
|
||||
The bump is how the fleet tells an agent that publishes `resolver` from one that does not —
|
||||
`--status`, `/health` and this test all read the same constant, and `sources/gateway.py`'s
|
||||
"the agent is too old" arm keys on the sections the version belongs to. The SCHEMA block is
|
||||
`GET /schema`'s body, so it names the section, its three-way `available`, and `--no-resolve`,
|
||||
the flag that removes it in silence.
|
||||
"""
|
||||
assert host_agent.AGENT_VERSION == "1.3.0"
|
||||
doc = host_agent.__doc__ or ""
|
||||
assert schema_documents("resolver"), "resolver is not documented in the SCHEMA block"
|
||||
block = doc.split(" resolver ", 1)[-1]
|
||||
assert "available" in block, "the resolver block does not document its three-way answer"
|
||||
assert "--no-resolve" in doc, "the flag that removes the section is undocumented"
|
||||
assert "libc" in block, "how the addresses were obtained is not published in the schema"
|
||||
|
||||
|
||||
# ---- the recorded gateway: the captures themselves (phase 28 task 04) -------------------- #
|
||||
|
||||
#: Every document `scripts/capture-gateway-config.sh` records, at the tree path the agent reads
|
||||
@@ -2983,6 +3036,151 @@ def test_the_recorded_configs_and_the_recorded_sockets_agree(tmp_path: Path) ->
|
||||
"the agent's own listener should be in the socket table and claimed by neither config"
|
||||
|
||||
|
||||
#: Every top-level key a **1.2.0** snapshot published, recorded here so this phase's central claim
|
||||
#: outlives the script it was measured with (`data/logs/verify-32-additive.py`, which imports the
|
||||
#: released file out of git and compares the two documents value for value). `data/` is gitignored;
|
||||
#: this table is not, and it is what makes "purely additive" a permanent pin.
|
||||
SNAPSHOT_KEYS_BEFORE_1_3_0 = frozenset({
|
||||
"agent_version", "cpu", "cpu_temperature_celsius", "disk_io", "duration_ms", "errors",
|
||||
"filesystems", "gpus", "haproxy", "host", "memory", "network", "nginx", "processes",
|
||||
"schema", "sockets", "ts", "temperatures",
|
||||
})
|
||||
|
||||
|
||||
def test_the_resolver_section_is_the_only_key_1_3_0_adds(tmp_path: Path) -> None:
|
||||
"""1.3.0 = 1.2.0 + `resolver`, and `--no-resolve` is exactly the old document.
|
||||
|
||||
The phase's safety property: an agent upgrade must not move a number on a page that never
|
||||
asked about name resolution, and a host that opts out must not be left with an empty shell of a
|
||||
key. Compared as key **sets** against the recorded pre-1.3.0 shape rather than against another
|
||||
module, so the pin is hermetic and permanent; the value-for-value half of the claim runs in the
|
||||
script named above, which loads the released file from git and calls both against one `--root`.
|
||||
"""
|
||||
root = gateway_tree(tmp_path / "root")
|
||||
with_resolver = host_agent.Collector(str(root), smartctl="/nonexistent/smartctl").snapshot()
|
||||
assert set(with_resolver) == SNAPSHOT_KEYS_BEFORE_1_3_0 | {"resolver"}, (
|
||||
f"1.3.0 changed the document beyond adding one section: "
|
||||
f"{sorted(set(with_resolver) ^ (SNAPSHOT_KEYS_BEFORE_1_3_0 | {'resolver'}))}")
|
||||
assert with_resolver["resolver"]["available"] is True
|
||||
|
||||
opted_out = host_agent.Collector(str(root), smartctl="/nonexistent/smartctl",
|
||||
resolve=False).snapshot()
|
||||
assert set(opted_out) == SNAPSHOT_KEYS_BEFORE_1_3_0, "--no-resolve leaves a key behind"
|
||||
assert not [line for line in opted_out["errors"]
|
||||
if str(line).lstrip().lower().startswith("resolver")], (
|
||||
"opting out is silence, the `--no-smart` precedent")
|
||||
|
||||
no_proxy = host_agent.Collector(str(root), smartctl="/nonexistent/smartctl",
|
||||
proxy=False).snapshot()
|
||||
assert set(no_proxy) == SNAPSHOT_KEYS_BEFORE_1_3_0 - {"haproxy", "nginx", "sockets"}, (
|
||||
"the resolver belongs to the proxy family and is removed with it")
|
||||
|
||||
|
||||
def test_the_recorded_resolver_document_is_the_agents_own_section(tmp_path: Path) -> None:
|
||||
"""`gateway_resolver.json` is a section of the payload, not notes about one.
|
||||
|
||||
Nothing here is a hand-written expectation: the entry set and the `resolver` directive are
|
||||
recomputed from the config fixtures in this directory with the agent's own `target_names()` and
|
||||
`nginx_document()`, so the two captures cannot drift apart. A route added to the gateway shows
|
||||
up in both on the next run; a fixture naming a target the parser no longer derives fails here
|
||||
instead of teaching `sources/gateway.py` to join a route the config does not declare.
|
||||
|
||||
The per-entry assertions are the ones the page will rely on: a status out of the four the
|
||||
section publishes, an `ok` with no address, an `nxdomain` with one, or a list the agent's own
|
||||
ordering rule would reshuffle, all mean the fixture is not what a poll would serve.
|
||||
"""
|
||||
document = json.loads((FIXTURES / "gateway_resolver.json").read_text())
|
||||
assert set(document) == {"available", "via", "config_directive", "ts", "entries", "errors"}
|
||||
assert document["available"] is True, "the recorded gateway reads, so the section is available"
|
||||
assert document["via"] == host_agent.RESOLVE_VIA
|
||||
assert document["errors"] == []
|
||||
|
||||
collector = gateway_collector(gateway_tree(tmp_path / "root"))
|
||||
errors: list[str] = []
|
||||
haproxy, nginx = collector.haproxy(errors), collector.nginx(errors)
|
||||
assert [(row["name"], row["port"]) for row in document["entries"]] == (
|
||||
host_agent.target_names(haproxy, nginx)), (
|
||||
"the recorded names are not the ones these configs declare — re-run "
|
||||
"./scripts/capture-gateway-config.sh and re-tally in the same commit")
|
||||
_section, directive = collector.nginx_document(errors)
|
||||
assert document["config_directive"] == directive, (
|
||||
"the fixture's `resolver` line is not what the agent reads out of the same text")
|
||||
|
||||
for row in document["entries"]:
|
||||
assert set(row) == {"name", "port", "status", "addresses", "ts"}
|
||||
assert row["status"] in {"ok", "nxdomain", "error", "literal"}, row
|
||||
if row["status"] in {"ok", "literal"}:
|
||||
assert row["addresses"], f"{row['name']} is {row['status']} with no address"
|
||||
else:
|
||||
assert row["addresses"] == [], f"{row['name']} publishes an address it did not resolve"
|
||||
assert row["addresses"] == host_agent.sort_resolver_addresses(row["addresses"]), (
|
||||
f"{row['name']} is not in the order the payload will publish it")
|
||||
|
||||
|
||||
def test_the_lookup_the_agent_performs_is_one_real_libc_accepts(monkeypatch, tmp_path) -> None:
|
||||
"""The one pin in this file with **no stub in sight**, because a stub cannot refuse a keyword.
|
||||
|
||||
`socket.getaddrinfo(name, port, proto=socket.SOCK_STREAM)` — the spelling that reads as "give
|
||||
me the TCP addresses" — is not that. `proto` is a **protocol number** (`IPPROTO_TCP` is 6) and
|
||||
`SOCK_STREAM` is 1, so libc is asked for a stream socket over protocol 1 and answers
|
||||
`EAI_SERVICE` for every name. A fake `getaddrinfo` accepts any keyword, so every resolver test
|
||||
in this file would have stayed green while the agent reported `error` on a healthy gateway and
|
||||
left every route on `/gateway` uncounted.
|
||||
|
||||
So: the module-wide `no_real_dns` fixture is removed for this test, and the lookup runs against
|
||||
real libc on a name libc answers from `/etc/hosts` — no resolver, no network, no skip. It
|
||||
cannot be replaced with a numeric literal either: an address literal is the `literal` arm and
|
||||
never reaches libc at all, which is the other half of the same pass.
|
||||
"""
|
||||
monkeypatch.setattr(host_agent.socket, "getaddrinfo", socket.getaddrinfo)
|
||||
collector = host_agent.Collector(str(gateway_tree(tmp_path / "root")),
|
||||
smartctl="/nonexistent/smartctl")
|
||||
|
||||
entry = collector._resolve_target("localhost", 9120)
|
||||
|
||||
assert entry["status"] == "ok", (
|
||||
f"real libc refused the call the agent makes ({entry}); if it says `error` with "
|
||||
"`EAI_SERVICE` in it, the keyword on that call has moved back to `proto=`")
|
||||
assert entry["addresses"], "a name libc resolves must publish at least one address"
|
||||
for value in entry["addresses"]:
|
||||
assert ipaddress.ip_address(value) # type: ignore[call-overload]
|
||||
|
||||
|
||||
def test_the_recorded_resolver_document_publishes_no_globally_routable_address() -> None:
|
||||
"""Invariant 6 over the JSON, for the same reason and with one hazard of its own.
|
||||
|
||||
A resolver answer is an identifier the same way a socket row is: `getent` on this router answers
|
||||
the WAN address of anything published through it, and a global IPv6 answer is EUI-64 — this
|
||||
box's NIC, in hex. `scripts/capture-gateway-config.sh` runs the same substitution over the
|
||||
resolver's answers as over the socket tables, so nothing routable is tracked.
|
||||
|
||||
The second assertion is specific to *this* fixture and is a warning, not a rule: on this fleet
|
||||
every declared target answers a private LAN address, so nothing needs scrubbing. If a capture
|
||||
ever moves an address into documentation space, that replacement is numbered per document in
|
||||
first-seen order — it is therefore **not** guaranteed to be the same documentation address the
|
||||
socket fixtures gave the same endpoint, and `tests/test_gateway_source.py`'s join of the two
|
||||
would silently stop joining. Re-check that join before trusting the numbers.
|
||||
"""
|
||||
document = json.loads((FIXTURES / "gateway_resolver.json").read_text())
|
||||
routable: list[str] = []
|
||||
documented: list[str] = []
|
||||
for row in document["entries"]:
|
||||
for value in row["addresses"]:
|
||||
address = ipaddress.ip_address(value)
|
||||
subject = getattr(address, "ipv4_mapped", None) or address
|
||||
if subject.is_global:
|
||||
routable.append(f"{row['name']} -> {value}")
|
||||
if not subject.is_private:
|
||||
documented.append(f"{row['name']} -> {value}")
|
||||
assert not routable, ("globally routable resolver answers are in a tracked fixture: "
|
||||
+ ", ".join(routable)
|
||||
+ " — scripts/capture-gateway-config.sh is the thing that stops them")
|
||||
assert not documented, ("the recorded resolver document had to be scrubbed: "
|
||||
+ ", ".join(documented)
|
||||
+ " — reconcile the replacement with the socket fixtures before "
|
||||
"trusting the join in tests/test_gateway_source.py")
|
||||
|
||||
|
||||
def test_the_recorded_socket_tables_publish_no_globally_routable_endpoint() -> None:
|
||||
"""AGENTS.md invariant 6, on the artifacts: identifiers never go in the repo.
|
||||
|
||||
@@ -3035,3 +3233,502 @@ def test_the_recorded_gateway_answers_all_three_sections_at_once(tmp_path: Path)
|
||||
"temperatures", "processes", "errors"):
|
||||
assert section in snap, f"the pre-1.2.0 payload lost {section}"
|
||||
|
||||
|
||||
|
||||
# ---- the resolver: the names the configs declare, resolved on this box (phase 32 task 01) -- #
|
||||
|
||||
def resolver_ledger(monkeypatch: pytest.MonkeyPatch, *,
|
||||
nx: tuple[str, ...] = (), delay: float = 0.0,
|
||||
fail: tuple[str, ...] = ()) -> list[tuple[str, int | None]]:
|
||||
"""Stub `getaddrinfo` with a ledger of every `(name, port)` it was asked about.
|
||||
|
||||
The three knobs are the three things a resolver does that a page has to survive: it answers
|
||||
(`203.0.113.x` + `2001:db8::…`, documentation space so a test payload can never carry a real
|
||||
endpoint), it refuses a name (`nxdomain`), and it is slow or broken (`delay`, `fail`).
|
||||
"""
|
||||
calls: list[tuple[str, int | None]] = []
|
||||
|
||||
def stub(name: object, port: object = None, *args: Any, **kwargs: Any) -> Any:
|
||||
text = str(name)
|
||||
calls.append((text, port if isinstance(port, int) else None))
|
||||
if text in fail:
|
||||
raise socket.gaierror(socket.EAI_NONAME, "and the resolver refused to say")
|
||||
if delay:
|
||||
time.sleep(delay)
|
||||
if text in nx:
|
||||
raise socket.gaierror(socket.EAI_NONAME, "Name or service not known")
|
||||
host_port = port if isinstance(port, int) else 0
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 0, "", (f"203.0.113.{len(calls)}", host_port))]
|
||||
|
||||
monkeypatch.setattr(host_agent.socket, "getaddrinfo", stub)
|
||||
return calls
|
||||
|
||||
|
||||
def resolver_snapshot(root: Path, **kwargs: Any) -> dict[str, Any]:
|
||||
"""One snapshot of a captured-gateway tree, the resolver's `getaddrinfo` stubbed by the caller."""
|
||||
return host_agent.Collector(str(root), smartctl="/nonexistent/smartctl",
|
||||
**kwargs).snapshot()
|
||||
|
||||
|
||||
def test_only_the_targets_the_configs_declare_are_resolved(tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The allow-list is the config, in config order, once each — decision 3, pinned as a call set.
|
||||
|
||||
Everything the resolver dialed is a target the two parsed sections publish, in the order the
|
||||
files write them, with no duplicate — and nothing that came from `/proc/net/tcp` is in there.
|
||||
That last half is the point: a socket table is full of addresses this gateway talks to, and
|
||||
resolving one of those would be a fact about whoever connected, not about the config.
|
||||
"""
|
||||
calls = resolver_ledger(monkeypatch)
|
||||
snap = resolver_snapshot(gateway_tree(tmp_path / "root"))
|
||||
declared = host_agent.target_names(snap["haproxy"], snap["nginx"])
|
||||
looked_up = [pair for pair in declared if host_agent.address_family(pair[0]) is None]
|
||||
|
||||
assert calls == looked_up, "the lookups are exactly the declared targets, in config order"
|
||||
assert len(calls) == len(set(calls)), "one name on one port is one lookup, not two"
|
||||
assert len(looked_up) < len(declared), "the tree writes literals too, and they are kept"
|
||||
socket_addresses = {row["remote"]["addr"] for row in snap["sockets"]["established"]}
|
||||
socket_addresses |= {row["local"]["addr"] for row in snap["sockets"]["established"]}
|
||||
socket_addresses |= {row["addr"] for row in snap["sockets"]["listens"]}
|
||||
assert not socket_addresses & {name for name, _ in calls}, (
|
||||
"a name from the socket table was resolved: the config is the only allow-list")
|
||||
# haproxy first (its six `server` lines), and this fleet's two Minecraft ports are two targets
|
||||
assert calls[:6] == [("reeseapps-gitea.reeselink.com", 2222),
|
||||
("stackexpected-gitea.reeselink.com", 2222),
|
||||
("web.reeselink.com", 5201),
|
||||
("unifi-external.reeselink.com", 8080),
|
||||
("minecraft.reeselink.com", 25565),
|
||||
("minecraft.reeselink.com", 25566)]
|
||||
|
||||
|
||||
def test_an_address_literal_in_a_config_is_reported_and_never_dialled(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""`proxy_pass http://127.0.0.1:8081` is already an address: the entry says so, no lookup runs.
|
||||
|
||||
Looking a literal up would be a round trip to be told what the config wrote, and on a box with
|
||||
a slow resolver it is time spent for nothing. The address still travels, because the gateway
|
||||
page's join keys on it.
|
||||
"""
|
||||
calls = resolver_ledger(monkeypatch)
|
||||
snap = resolver_snapshot(gateway_tree(tmp_path / "root"))
|
||||
entries = {host_agent.resolver_key(e["name"], e["port"]): e for e in snap["resolver"]["entries"]}
|
||||
|
||||
literal = entries[host_agent.resolver_key("127.0.0.1", 8081)]
|
||||
assert literal["status"] == "literal"
|
||||
assert literal["addresses"] == ["127.0.0.1"]
|
||||
assert ("127.0.0.1", 8081) not in calls, "a literal costs no lookup"
|
||||
assert not [call for call in calls if host_agent.address_family(call[0])], \
|
||||
"no lookup was ever asked about an address"
|
||||
|
||||
|
||||
def test_the_section_names_how_and_where_and_when(tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""`via`, `config_directive` and the timestamps are the provenance of every address here.
|
||||
|
||||
`libc` is what `getaddrinfo` is — the resolution haproxy performs on this box — and the nginx
|
||||
config's own `resolver` line travels beside it verbatim (read out of `stream.d/stream.conf`,
|
||||
which is where this fleet writes it, so an include is as good as the main file). The
|
||||
`nginx` section itself grows no key: that is what keeps the phase's additivity control a
|
||||
top-level comparison.
|
||||
"""
|
||||
resolver_ledger(monkeypatch)
|
||||
snap = resolver_snapshot(gateway_tree(tmp_path / "root"))
|
||||
section = snap["resolver"]
|
||||
assert section["via"] == host_agent.RESOLVE_VIA == "libc"
|
||||
assert section["config_directive"] == "10.1.0.1"
|
||||
assert "resolver" not in snap["nginx"], "the nginx document stays byte-stable"
|
||||
assert isinstance(section["ts"], float)
|
||||
for entry in section["entries"]:
|
||||
assert entry["ts"] <= section["ts"] + 1.0, "an entry is answered at or before the pass ends"
|
||||
assert set(entry) == {"name", "port", "status", "addresses", "ts"}
|
||||
|
||||
|
||||
def test_the_resolver_cache_means_one_lookup_per_target(tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Two snapshots, one set of lookups; an expiry refreshes in the background and never on the request.
|
||||
|
||||
The dashboard polls every 10 s. Without the cache that is 30 DNS lookups per host per minute
|
||||
for names that move when the operator moves them; with it, the poll that finds an expired
|
||||
cache is answered from the cache and carries the **original** per-entry `ts`, so it cannot
|
||||
present a five-minute-old address as a fresh one.
|
||||
"""
|
||||
calls = resolver_ledger(monkeypatch)
|
||||
root = gateway_tree(tmp_path / "root")
|
||||
collector = host_agent.Collector(str(root), smartctl="/nonexistent/smartctl")
|
||||
|
||||
first = collector.snapshot()["resolver"]
|
||||
baseline = len(calls)
|
||||
assert baseline and len(calls) == len(set(calls))
|
||||
|
||||
second = collector.snapshot()["resolver"]
|
||||
assert len(calls) == baseline, "a fresh cache costs no lookup"
|
||||
assert second["entries"] == first["entries"]
|
||||
|
||||
collector.resolve_cache_seconds = 0.0
|
||||
third = collector.snapshot()["resolver"]
|
||||
assert third["entries"] == first["entries"], "the expired cache is served as it was"
|
||||
assert [e["ts"] for e in third["entries"]] == [e["ts"] for e in first["entries"]], \
|
||||
"a cache-served entry keeps the time it was answered, not the time it was served"
|
||||
wait_for_resolver_refresh(collector)
|
||||
assert len(calls) > baseline, "the expiry did re-resolve, off the request path"
|
||||
|
||||
|
||||
def test_a_slow_resolver_cannot_hold_the_snapshot_open(tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The pass stops at its budget and says so once — the SMART pass-budget lesson, applied to DNS.
|
||||
|
||||
`getaddrinfo` takes no timeout argument, so the only defence against a resolver that has
|
||||
started black-holing is to stop asking. What the section must still be: `available: true`, the
|
||||
targets it reached present, one line naming how many of them that was, and `/snapshot`
|
||||
answering in a small fraction of the time the full set would have taken — because the *system*
|
||||
page's timeout is what a slow pass would otherwise cost.
|
||||
"""
|
||||
calls = resolver_ledger(monkeypatch, delay=0.02)
|
||||
root = gateway_tree(tmp_path / "root")
|
||||
collector = host_agent.Collector(str(root), smartctl="/nonexistent/smartctl")
|
||||
collector.resolve_pass_budget_seconds = 0.1
|
||||
|
||||
started = time.monotonic()
|
||||
snap = collector.snapshot()
|
||||
elapsed = time.monotonic() - started
|
||||
section = snap["resolver"]
|
||||
budget_lines = [line for line in section["errors"] if "pass budget" in line]
|
||||
|
||||
assert section["available"] is True, "a truncated pass is still an answer"
|
||||
assert len(budget_lines) == 1, snap["errors"]
|
||||
reached = [e for e in section["entries"] if e["status"] != "literal"]
|
||||
assert budget_lines[0] == (f"resolver: pass budget 0.1s exhausted, "
|
||||
f"{len(reached)} of 30 targets looked up"), budget_lines[0]
|
||||
assert len(reached) < 29, "the pass stopped rather than finished"
|
||||
assert len(calls) == len(reached), "the ledger and the section agree on what was reached"
|
||||
assert [e for e in section["entries"] if e["status"] == "literal"], \
|
||||
"an address the config spells survives a pass that stopped early"
|
||||
assert len(section["entries"]) == len(reached) + 2, "two literals, and nothing invented"
|
||||
assert elapsed < 1.0, f"a bounded pass cost {elapsed:.2f}s"
|
||||
assert snap["cpu"]["modes"]["total"] > 0 and snap["host"]["hostname"]
|
||||
assert core_errors(snap) == [], snap["errors"]
|
||||
|
||||
|
||||
def test_the_name_bound_reports_the_excess_in_one_line(tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""More declared targets than the bound: the excess is one line, not 40, and `true` stands."""
|
||||
calls = resolver_ledger(monkeypatch)
|
||||
root = gateway_tree(tmp_path / "root")
|
||||
collector = host_agent.Collector(str(root), smartctl="/nonexistent/smartctl")
|
||||
collector.resolve_max_names = 3
|
||||
|
||||
section = collector.snapshot()["resolver"]
|
||||
bound_lines = [line for line in section["errors"] if "name bound" in line]
|
||||
assert len(bound_lines) == 1, section["errors"]
|
||||
assert bound_lines[0].startswith("resolver: 27 of 30 targets are beyond the 3-name bound"), \
|
||||
bound_lines[0]
|
||||
assert section["available"] is True
|
||||
assert len(calls) == 3, "the bound is the cost of the section, not a display choice"
|
||||
literals = [e for e in section["entries"] if e["status"] == "literal"]
|
||||
assert len(section["entries"]) == 3 + len(literals)
|
||||
|
||||
|
||||
def wait_for_resolver_refresh(collector: Any, attempts: int = 500) -> None:
|
||||
"""Join the single-flight refresher, as far as a test can: it is a daemon thread on purpose.
|
||||
|
||||
The wait is on the flag rather than a `Thread.join()` because the agent keeps no handle to it
|
||||
(a snapshot must never be able to wait on DNS), and 5 s of 10 ms polls is more than a stubbed
|
||||
pass takes on any machine this suite runs on.
|
||||
"""
|
||||
for _ in range(attempts):
|
||||
if not collector._resolver_refreshing:
|
||||
return
|
||||
time.sleep(0.01)
|
||||
raise AssertionError("the background resolver refresh never finished")
|
||||
|
||||
|
||||
def test_a_failed_background_refresh_keeps_the_last_addresses(tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A refresh that dies is one line, the old entries, and no hammering.
|
||||
|
||||
The thread is the only thing that ever re-reads DNS after the first pass, so a pass that
|
||||
raises must leave the snapshot answering with the last truth it had (SMART's rule) and must
|
||||
not turn every subsequent poll into a retry. One line, replaced rather than stacked: a
|
||||
resolver that has been broken for a week is one fact the operator acts on once, not 2 000.
|
||||
"""
|
||||
resolver_ledger(monkeypatch)
|
||||
collector = host_agent.Collector(str(gateway_tree(tmp_path / "root")),
|
||||
smartctl="/nonexistent/smartctl")
|
||||
first = collector.snapshot()["resolver"]
|
||||
assert first["available"] is True and first["entries"]
|
||||
|
||||
def explode(*args: object) -> None:
|
||||
raise RuntimeError("the resolver daemon is gone")
|
||||
|
||||
monkeypatch.setattr(host_agent.Collector, "resolver_pass", explode)
|
||||
collector.resolve_cache_seconds = 0.0
|
||||
for attempt in range(2):
|
||||
served = collector.snapshot()["resolver"]
|
||||
assert served["entries"] == first["entries"], "the last good answers stay published"
|
||||
wait_for_resolver_refresh(collector)
|
||||
# Read the collector's state, not this response: the snapshot was served *while* the
|
||||
# refresh was in flight, so the line it could not have is not a defect in it.
|
||||
failures = [line for line in collector._resolver_errors
|
||||
if "background refresh failed" in line]
|
||||
assert len(failures) == 1, (attempt, collector._resolver_errors)
|
||||
assert failures[0].startswith("resolver: background refresh failed: RuntimeError")
|
||||
assert collector._resolver_cache_ts is not None, "a failed pass still re-stamps the cache"
|
||||
# …and the line reaches the page eventually, beside the entries it explains.
|
||||
later = collector.snapshot()["resolver"]
|
||||
assert any("background refresh failed" in line for line in later["errors"]), later["errors"]
|
||||
assert later["entries"] == first["entries"]
|
||||
|
||||
|
||||
def test_nxdomain_is_a_status_and_never_a_failure(tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""One name that does not answer is one entry with no addresses — the snapshot is unbothered.
|
||||
|
||||
The route stays on the page and carries no count (task 03 does the joining); what must not
|
||||
happen is a `0`, a dropped entry, or a poll that goes red because a domain left the zone file.
|
||||
"""
|
||||
resolver_ledger(monkeypatch, nx=("reeseapps-brain-of-reese.reeselink.com",))
|
||||
snap = resolver_snapshot(gateway_tree(tmp_path / "root"))
|
||||
section = snap["resolver"]
|
||||
entries = {e["name"]: e for e in section["entries"]}
|
||||
|
||||
failed = entries["reeseapps-brain-of-reese.reeselink.com"]
|
||||
assert failed["status"] == "nxdomain"
|
||||
assert failed["addresses"] == [], "no address, never a 0 and never last month's"
|
||||
answered = [e for e in section["entries"] if e["status"] == "ok"]
|
||||
assert answered, "the others answered"
|
||||
assert section["available"] is True
|
||||
assert section["errors"] == [], "a name that does not resolve is not this section's failure"
|
||||
assert snap["cpu"]["modes"]["total"] > 0
|
||||
assert core_errors(snap) == [], snap["errors"]
|
||||
|
||||
|
||||
def test_a_resolver_that_errors_is_an_error_and_not_an_nxdomain(tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
""""The name has no record" and "the lookup could not be performed" are two fixes.
|
||||
|
||||
A temporary failure (`EAI_AGAIN`) is the one that most looks like `nxdomain` and must not be
|
||||
reported as it: an operator reading `nxdomain` edits a zone file, an operator reading `error`
|
||||
checks whether the resolver is up.
|
||||
"""
|
||||
def stub(name: object, port: object = None, *args: Any, **kwargs: Any) -> Any:
|
||||
text = str(name)
|
||||
if text == "gitea.reeselink.com":
|
||||
raise socket.gaierror(socket.EAI_AGAIN, "Temporary failure in name resolution")
|
||||
if text == "jellyfin.reeselink.com":
|
||||
raise OSError("resolver socket refused to open")
|
||||
if text == "bricktracker.reeselink.com":
|
||||
raise ValueError("a platform-shaped answer nobody predicted")
|
||||
host_port = port if isinstance(port, int) else 0
|
||||
return [(socket.AF_INET, socket.SOCK_STREAM, 0, "", ("203.0.113.7", host_port))]
|
||||
|
||||
monkeypatch.setattr(host_agent.socket, "getaddrinfo", stub)
|
||||
snap = resolver_snapshot(gateway_tree(tmp_path / "root"))
|
||||
entries = {e["name"]: e for e in snap["resolver"]["entries"]}
|
||||
|
||||
assert entries["gitea.reeselink.com"]["status"] == "error", "EAI_AGAIN is not nxdomain"
|
||||
assert entries["jellyfin.reeselink.com"]["status"] == "error"
|
||||
assert entries["bricktracker.reeselink.com"]["status"] == "error"
|
||||
assert entries["bricktracker.reeselink.com"]["detail"].startswith("ValueError")
|
||||
assert entries["gitea.reeselink.com"]["addresses"] == []
|
||||
assert snap["resolver"]["available"] is True
|
||||
assert core_errors(snap) == [], snap["errors"]
|
||||
|
||||
|
||||
def test_the_addresses_are_unique_and_sorted_v4_then_v6(tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Two polls of one healthy gateway are byte-identical, whatever order libc answered in.
|
||||
|
||||
`getaddrinfo` returns whatever the resolver and `/etc/gai.conf` feel like today, and the entry
|
||||
list is something a page renders: an unsorted set makes a healthy box look like its addresses
|
||||
are shuffling.
|
||||
"""
|
||||
def stub(name: object, port: object = None, *args: Any, **kwargs: Any) -> Any:
|
||||
host_port = port if isinstance(port, int) else 0
|
||||
return [(socket.AF_INET6, socket.SOCK_STREAM, 0, "", ("2001:db8::2", host_port, 0, 0)),
|
||||
(socket.AF_INET, socket.SOCK_STREAM, 0, "", ("203.0.113.9", host_port)),
|
||||
(socket.AF_INET, socket.SOCK_STREAM, 0, "", ("203.0.113.9", host_port)),
|
||||
(socket.AF_INET, socket.SOCK_STREAM, 0, "", ("203.0.113.4", host_port))]
|
||||
|
||||
monkeypatch.setattr(host_agent.socket, "getaddrinfo", stub)
|
||||
snap = resolver_snapshot(gateway_tree(tmp_path / "root"))
|
||||
entry = snap["resolver"]["entries"][0]
|
||||
assert entry["status"] == "ok"
|
||||
assert entry["addresses"] == ["203.0.113.4", "203.0.113.9", "2001:db8::2"]
|
||||
|
||||
|
||||
def test_a_value_that_is_not_an_address_sorts_last_instead_of_raising() -> None:
|
||||
"""The sort is defensive for the one input the section should never see.
|
||||
|
||||
`getaddrinfo` answers addresses, so this arm is a stub or a platform oddity — and the answer
|
||||
to that is a stable order with the odd value at the end, not a `ValueError` on a snapshot path
|
||||
and not the whole section lost because one entry could not be compared.
|
||||
"""
|
||||
assert host_agent.sort_resolver_addresses(["nope", "10.0.0.2", "::1", "also-bad", "10.0.0.2"]) \
|
||||
== ["10.0.0.2", "::1", "also-bad", "nope"]
|
||||
|
||||
|
||||
def test_no_resolve_removes_the_key_in_silence(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""`--no-resolve` is `--no-smart`'s precedent: no key, no error line — "not collected".
|
||||
|
||||
An empty section would read as "collected, and nothing resolves", which is a different fact
|
||||
and the wrong one to print for a host the operator told not to ask.
|
||||
"""
|
||||
calls = resolver_ledger(monkeypatch)
|
||||
root = gateway_tree(tmp_path / "root")
|
||||
|
||||
snap = host_agent.Collector(str(root), smartctl="/nonexistent/smartctl",
|
||||
resolve=False).snapshot()
|
||||
assert "resolver" not in snap
|
||||
assert calls == [], "the flag is honoured before anything is dialled"
|
||||
assert snap["nginx"]["available"] is True and snap["sockets"]["available"] is True, \
|
||||
"the rest of the family is untouched"
|
||||
assert [line for line in snap["errors"] if line.startswith("resolver:")] == []
|
||||
|
||||
|
||||
def test_no_proxy_removes_the_resolver_with_the_rest_of_the_family(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""A box that collects no proxy has no declared names, so `--no-proxy` removes this too."""
|
||||
calls = resolver_ledger(monkeypatch)
|
||||
snap = host_agent.Collector(str(gateway_tree(tmp_path / "root")),
|
||||
smartctl="/nonexistent/smartctl", proxy=False).snapshot()
|
||||
for section in ("haproxy", "nginx", "sockets", "resolver"):
|
||||
assert section not in snap
|
||||
assert calls == []
|
||||
|
||||
|
||||
def test_a_host_with_no_proxy_sections_says_so_once(tmp_path: Path) -> None:
|
||||
"""`available: false` and exactly one `resolver:` line naming **not found** — invariant 3.
|
||||
|
||||
Two of this fleet's three hosts run no proxy at all. Their agents must report "nothing to
|
||||
resolve" without putting a red banner, an overview alert, or fifty lines in `errors` anywhere.
|
||||
"""
|
||||
snap = host_agent.Collector(str(tmp_path / "root")).snapshot()
|
||||
resolver_lines = [line for line in snap["errors"] if line.startswith("resolver:")]
|
||||
|
||||
assert snap["resolver"]["available"] is False
|
||||
assert snap["resolver"]["entries"] == []
|
||||
assert len(resolver_lines) == 1, snap["errors"]
|
||||
assert "not found" in resolver_lines[0]
|
||||
assert "--no-resolve" in resolver_lines[0]
|
||||
assert snap["resolver"]["errors"] == resolver_lines
|
||||
assert core_errors(snap) == [], "an unconfigured add-on is not a system degradation"
|
||||
|
||||
|
||||
@pytest.mark.skipif(os.geteuid() == 0, reason="root reads a 0000 file")
|
||||
def test_a_config_that_is_present_but_unreadable_says_that_instead(tmp_path: Path) -> None:
|
||||
"""The `false` arm names the right reason: an unreadable file is not a missing one.
|
||||
|
||||
`not found` and `unreadable` are two operator actions — install the proxy, or fix the mode on
|
||||
its config — and one line that says which is the difference between a `chmod` and a
|
||||
`dnf install`. Still one line, still `available: false`, still no system degradation.
|
||||
"""
|
||||
root = tmp_path / "root"
|
||||
nginx_tree(root, http=None, stream=None)
|
||||
(root / "etc/nginx/nginx.conf").chmod(0o000)
|
||||
|
||||
snap = host_agent.Collector(str(root), smartctl="/nonexistent/smartctl").snapshot()
|
||||
line = next(line for line in snap["errors"] if line.startswith("resolver:"))
|
||||
assert snap["resolver"]["available"] is False
|
||||
assert "haproxy not found" in line
|
||||
assert "nginx unreadable" in line
|
||||
assert len([x for x in snap["errors"] if x.startswith("resolver:")]) == 1
|
||||
|
||||
|
||||
def test_a_config_that_reads_cleanly_and_names_no_target_is_true_and_empty(
|
||||
tmp_path: Path) -> None:
|
||||
"""Configs read, no targets declared: `available: true` with `entries: []` and no line.
|
||||
|
||||
Absence of a target is not a fault, and the third arm (`null`) is reserved for a pass that
|
||||
could not run at all — otherwise "no routes" and "no resolver" arrive as one value.
|
||||
"""
|
||||
root = tmp_path / "root"
|
||||
nginx_tree(root, main="http { server { listen 8080; server_name quiet.test; } }\n",
|
||||
http=None, stream=None)
|
||||
snap = host_agent.Collector(str(root), smartctl="/nonexistent/smartctl").snapshot()
|
||||
|
||||
assert snap["nginx"]["available"] is True
|
||||
assert snap["resolver"]["available"] is True, snap["errors"]
|
||||
assert snap["resolver"]["entries"] == []
|
||||
assert snap["resolver"]["errors"] == []
|
||||
assert core_errors(snap) == [], snap["errors"]
|
||||
|
||||
|
||||
def test_the_resolver_lines_never_fail_the_snapshot(tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Every arm of the new section leaves `/snapshot` answering, with its key and its own line.
|
||||
|
||||
The family rule phase 28 pinned for `haproxy`, `nginx` and `sockets` extended to the fourth
|
||||
section: whatever the resolver does — raises inside, cannot find a config, cannot reach a
|
||||
name — the document keeps the key (so a reader can tell "the pass failed" from "the agent
|
||||
predates it"), the other sections are intact, and the line is the add-on's own rather than a
|
||||
system `degraded` (invariant 3).
|
||||
"""
|
||||
def explode(*args: object) -> dict[str, Any]:
|
||||
raise RuntimeError("resolver exploded")
|
||||
|
||||
monkeypatch.setattr(host_agent.Collector, "resolver", explode)
|
||||
snap = host_agent.Collector(str(gateway_tree(tmp_path / "root")),
|
||||
smartctl="/nonexistent/smartctl").snapshot()
|
||||
assert snap["resolver"] == {"available": None, "via": "libc", "config_directive": None,
|
||||
"ts": None, "entries": [], "errors": []}
|
||||
assert any(line.startswith("resolver:") for line in snap["errors"])
|
||||
assert core_errors(snap) == [], snap["errors"]
|
||||
assert snap["haproxy"]["available"] is True and snap["nginx"]["available"] is True
|
||||
assert snap["cpu"]["modes"]["total"] > 0
|
||||
|
||||
|
||||
def test_the_resolver_section_is_documented_in_the_schema() -> None:
|
||||
"""`GET /schema` answers the docstring, so the section an operator can `curl` is described."""
|
||||
assert schema_documents("resolver")
|
||||
|
||||
|
||||
def test_the_resolver_reaches_the_command_line(tmp_path: Path, capsys: pytest.CaptureFixture[str],
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""`--once` prints the resolver block, and prints none with `--no-resolve` added.
|
||||
|
||||
This is the completion criterion an operator can run on a host with no dashboard: one command,
|
||||
and the answer says which names this box turned into which addresses.
|
||||
"""
|
||||
resolver_ledger(monkeypatch)
|
||||
root = gateway_tree(tmp_path / "root")
|
||||
# The config paths are the ones the **config** writes: an absolute path resolves under
|
||||
# `--root`, which is how a deployment reads /etc and a test reads a fake /etc.
|
||||
nginx, haproxy = "/etc/nginx/nginx.conf", "/etc/haproxy/haproxy.cfg"
|
||||
|
||||
assert host_agent.main(["--once", "--no-smart", "--root", str(root),
|
||||
"--haproxy-config", haproxy, "--nginx-config", nginx]) == 0
|
||||
printed = json.loads(capsys.readouterr().out)
|
||||
assert printed["resolver"]["available"] is True
|
||||
assert printed["resolver"]["via"] == "libc"
|
||||
assert printed["resolver"]["config_directive"] == "10.1.0.1"
|
||||
assert printed["resolver"]["entries"]
|
||||
|
||||
assert host_agent.main(["--once", "--no-smart", "--no-resolve", "--root", str(root),
|
||||
"--haproxy-config", haproxy, "--nginx-config", nginx]) == 0
|
||||
without = json.loads(capsys.readouterr().out)
|
||||
assert "resolver" not in without
|
||||
assert without["nginx"]["available"] is True, "the rest of the family still prints"
|
||||
|
||||
|
||||
def test_the_target_list_is_the_two_sections_and_survives_a_section_that_is_not_one() -> None:
|
||||
"""`target_names` reads haproxy then nginx (upstreams, proxy_pass, maps) and nothing stranger.
|
||||
|
||||
A `proxy_pass $var` contributes nothing — the map decides, and the map's entries are already
|
||||
in the list — and a `None`, a bare string, or a section of the wrong shape contributes an
|
||||
empty list rather than a `TypeError` on a snapshot path.
|
||||
"""
|
||||
haproxy = {"backends": [{"servers": [{"name": "a", "target": "a.test", "port": 1},
|
||||
{"name": "b", "target": "b.test", "port": None},
|
||||
{"name": "c", "target": None, "port": 3},
|
||||
{"name": "d", "target": "a.test", "port": 1}]}]}
|
||||
nginx = {"upstreams": [{"servers": [{"host": "pool.test", "port": 80}]}],
|
||||
"servers": [{"proxy_passes": [{"target": {"host": "vhost.test", "port": 443}},
|
||||
{"target": None, "variable": "backend"}]}],
|
||||
"maps": [{"entries": [{"key": "x", "target": {"host": "map.test", "port": 443}},
|
||||
{"key": "default", "target": None}]}]}
|
||||
|
||||
assert host_agent.target_names(haproxy, nginx) == [
|
||||
("a.test", 1), ("b.test", None), ("pool.test", 80),
|
||||
("vhost.test", 443), ("map.test", 443)]
|
||||
assert host_agent.target_names(None, None) == []
|
||||
assert host_agent.target_names({"backends": "nonsense"}, {"maps": [7]}) == []
|
||||
|
||||
@@ -24,6 +24,7 @@ import inspect
|
||||
from typing import Any
|
||||
|
||||
from dashboards import alerts, severity
|
||||
from dashboards.collectors import ADDON_ERROR_PREFIXES
|
||||
from dashboards.config import Settings
|
||||
from dashboards.sources import nut
|
||||
from dashboards.sources.nut_sim import NutSimulator
|
||||
@@ -351,6 +352,39 @@ def test_a_degraded_agent_warns_and_names_the_sections_that_failed(
|
||||
assert "disk_io" in out[0]["message"]
|
||||
|
||||
|
||||
def test_an_addon_absence_line_raises_no_row_and_a_section_that_failed_raises_one(
|
||||
host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
|
||||
"""Invariant 3's last link, `resolver:` included: filtered out means no row, unfiltered means one.
|
||||
|
||||
`collectors.ADDON_ERROR_PREFIXES` is what keeps an agent's add-on lines out of `degraded`
|
||||
(`tests/test_collectors.py` pins that), and `degraded` is what this module turns into the
|
||||
overview row pinned just above. So: a host whose agent said it runs no haproxy, no nginx, no
|
||||
smartmontools and had **nothing to resolve** is all-clear on every panel — the phase-32 pair of
|
||||
lines (`nothing to resolve`, `pass budget … exhausted`) included, because a truncated lookup is
|
||||
one route without a count and not a broken box. And a line that *did* reach `degraded` — which
|
||||
is what dropping a prefix from that list would do — does raise exactly one row, naming it.
|
||||
"""
|
||||
addon_lines = (
|
||||
"resolver: nothing to resolve — haproxy not found and nginx not found; --no-resolve opts out",
|
||||
"resolver: pass budget 3.0s exhausted, 41 of 58 targets looked up",
|
||||
"haproxy: /etc/haproxy/haproxy.cfg not found — haproxy is not installed here",
|
||||
"smart: smartctl 7.5 not found",
|
||||
)
|
||||
assert all(line.lower().startswith(ADDON_ERROR_PREFIXES) for line in addon_lines), (
|
||||
"a prefix dropped from collectors' list would put this fleet's non-gateway hosts on warn")
|
||||
|
||||
clean = host_scrape(agent_snapshot, cpu_busy=10.0)
|
||||
clean["agent"]["errors"] = list(addon_lines)
|
||||
assert "degraded" not in clean, "the collector filtered them, so there is nothing to grade"
|
||||
assert alerts.build(SETTINGS, [state()], [clean]) == []
|
||||
|
||||
for line in addon_lines:
|
||||
broken = host_scrape(agent_snapshot, cpu_busy=10.0)
|
||||
broken["degraded"] = [line]
|
||||
out = alerts.build(SETTINGS, [state()], [broken])
|
||||
assert [row["id"] for row in out] == ["driveripper:agent.sections"], line
|
||||
|
||||
|
||||
def test_a_degraded_list_from_an_older_agent_is_readable_too(
|
||||
host_scrape: Any, agent_snapshot: dict[str, Any]) -> None:
|
||||
payload = host_scrape(agent_snapshot, cpu_busy=10.0)
|
||||
|
||||
+603
-65
@@ -4907,9 +4907,17 @@ def test_the_gateway_job_is_handed_no_rate_estimator(tmp_path: Path) -> None:
|
||||
|
||||
def make_gateway_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
||||
**overrides: Any) -> Any:
|
||||
"""An app whose only host is the ingress, on clocks fast enough to poll inside a test."""
|
||||
"""An app whose only host is the ingress, on clocks fast enough to poll inside a test.
|
||||
|
||||
The intervals are the poller's own floor, not 0.05: `Job` clamps its cadence at 0.5 s,
|
||||
while the staleness threshold is three times the *configured* interval. At 0.05 s a healthy
|
||||
job's gap (≥ 0.5 s) always outlived its own threshold (0.15 s), so its `:stale` row
|
||||
appeared in the alert list 70 % of the time and any assertion on the list's length was
|
||||
a test of when the event loop happened to tick. At 0.5 s the threshold (1.5 s) is
|
||||
three times the gap a healthy job can make, and the premise jobs stay quiet whatever
|
||||
the machine is doing."""
|
||||
return build_client(tmp_path, monkeypatch, host_targets=(GATEWAY_TARGET,),
|
||||
host_interval=0.05, disks_interval=0.05, gateway_interval=0.05,
|
||||
host_interval=0.5, disks_interval=0.5, gateway_interval=0.5,
|
||||
**overrides)
|
||||
|
||||
|
||||
@@ -4975,25 +4983,56 @@ def test_the_summary_gateway_block_is_the_stub_of_a_source_that_never_answered(
|
||||
assert block["href"] == GATEWAY_HREF
|
||||
|
||||
|
||||
def endpoint_identity(out: dict[str, Any]) -> tuple[Any, ...] | None:
|
||||
"""One endpoint, in the currency the kernel writes — derived here, not by the source.
|
||||
|
||||
An endpoint written as an address **is** that address; an endpoint written as a name is every
|
||||
address the agent's resolver published for it (`out.resolved`), and `ipaddress` folds the
|
||||
`::ffff:`-mapped and zero-padded spellings so one address in two tables is one key. `None`
|
||||
means the join has nothing to name the far end with, which contributes nothing to a total.
|
||||
|
||||
This is deliberately a **third** reading of "what is one endpoint" (the others are
|
||||
`gateway.endpoint_of` behind `api._endpoint_values`, and the join itself): since phase 32 two
|
||||
route lines can reach one address while the configs spell it as two different names, and a
|
||||
total keyed on the config's text would book it once per name.
|
||||
"""
|
||||
port = out.get("port")
|
||||
if not isinstance(port, int):
|
||||
return None
|
||||
spellings = ([out["host"]] if out.get("address") is True
|
||||
else list(out.get("resolved") or []))
|
||||
keys: list[str] = []
|
||||
for spelling in spellings:
|
||||
try:
|
||||
address = ipaddress.ip_address(str(spelling))
|
||||
except ValueError:
|
||||
continue
|
||||
mapped = getattr(address, "ipv4_mapped", None)
|
||||
keys.append(str(mapped if mapped is not None else address))
|
||||
return (min(keys), port) if keys else None
|
||||
|
||||
|
||||
def endpoint_sums(payload: dict[str, Any]) -> tuple[int | None, int | None]:
|
||||
"""The two live totals, derived here so the block is checked against a **second** reading.
|
||||
|
||||
Inbound is the sum over trunks. Outbound is the sum over the **distinct** `(host, port)` the
|
||||
route lines name: `sources/gateway.py` repeats an endpoint's whole count on every line that
|
||||
reaches it (`shared: true`), so summing the lines would book the same sockets twice and print
|
||||
a router busier than the kernel has connections for. Neither is a sum when nothing answered.
|
||||
Inbound is the sum over trunks. Outbound is the sum over the **distinct endpoints** the route
|
||||
lines reach: `sources/gateway.py` repeats an endpoint's whole count on every line that reaches
|
||||
it (`shared: true`), so summing the lines would book the same sockets twice and print a router
|
||||
busier than the kernel has connections for. Neither is a sum when nothing answered.
|
||||
"""
|
||||
live_in = [row["connections"] for row in payload["listeners"]
|
||||
if isinstance(row["connections"], int)]
|
||||
endpoints: dict[tuple[Any, Any], Any] = {}
|
||||
endpoints: dict[tuple[Any, ...], Any] = {}
|
||||
for row in payload["connections"]:
|
||||
endpoints.setdefault((row["out"]["host"], row["out"]["port"]), row["connections"])
|
||||
key = endpoint_identity(row["out"])
|
||||
if key is not None:
|
||||
endpoints.setdefault(key, row["connections"])
|
||||
live_out = [value for value in endpoints.values() if isinstance(value, int)]
|
||||
return (sum(live_in) if live_in else None,
|
||||
sum(live_out) if live_out else None)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode", ["full", "predates", "sockets_silent"])
|
||||
@pytest.mark.parametrize("mode", ["full", "resolved", "predates", "sockets_silent"])
|
||||
def test_the_summary_gateway_block_is_the_payloads_sums_not_the_pages_arithmetic(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, gateway_snapshot: Any,
|
||||
mode: str) -> None:
|
||||
@@ -5003,12 +5042,18 @@ def test_the_summary_gateway_block_is_the_payloads_sums_not_the_pages_arithmetic
|
||||
hand-written payload, so a rename in the source's shape fails here instead of printing a
|
||||
plausible zero (rule 7).
|
||||
|
||||
The split that replaced `partial` is the point of the three arms. `listeners` / `connections`
|
||||
The split that replaced `partial` is the point of the arms. `listeners` / `connections`
|
||||
are **row counts** — a fact readable from a config file, so they are numbers even when the
|
||||
socket table never answered — while `live_in` / `live_out` are measurements and go `null` with
|
||||
it. The old block could only report one number per side and needed a `partial` flag to say it
|
||||
was a subtotal; there is no residual to qualify now, because a socket outside the two files is
|
||||
counted nowhere by decision.
|
||||
|
||||
The `resolved` arm is phase 32's, and it is here because the **block** must not change when the
|
||||
join widens: twelve fields, the same names, and `live_out` still the sum over distinct
|
||||
endpoints — now endpoints reached through a resolved name as well as an address, each booked
|
||||
once however many lines reach it. A block that summed the lines instead would move from 15 to a
|
||||
bigger, wronger number on the same capture.
|
||||
"""
|
||||
payload = gateway_source.parse(gateway_snapshot(mode), "gateway")
|
||||
expected_in, expected_out = endpoint_sums(payload)
|
||||
@@ -5054,22 +5099,40 @@ def test_the_summary_gateway_block_of_the_captured_ingress_reads_the_live_join(
|
||||
"""The real capture, one figure at a time — the block an operator would actually get.
|
||||
|
||||
The parametrised test above proves the rule; this one pins the numbers, so a change to the join
|
||||
that moves a total is visible somewhere rather than only in a browser. The outbound figure is
|
||||
small **on purpose**: this capture's configs write almost every target as a domain, and a
|
||||
domain is not something this stack can join to a socket, so only the two `127.0.0.1` endpoints
|
||||
carry a count at all (`:4443` with 2 and `:8081` with 0).
|
||||
that moves a total is visible somewhere rather than only in a browser. The three config-shaped
|
||||
figures are literal, because only the operator moves them; the two live totals are derived from
|
||||
the capture rather than quoted, because a re-capture of a busy router lawfully moves them.
|
||||
|
||||
Since phase 32 task 03 the outbound total is no longer pinned to the two `127.0.0.1` endpoints:
|
||||
the recorded `gateway_resolver.json` places every name this ingress routes to, so the routes
|
||||
that used to be dashes are counted through the address the gateway dialled. What is still pinned
|
||||
is the **rule** the number comes from — one count per distinct endpoint, and never more than the
|
||||
kernel has connections for.
|
||||
"""
|
||||
payload = gateway_source.parse(gateway_snapshot("resolved"), "gateway")
|
||||
with make_gateway_client(tmp_path, monkeypatch) as client:
|
||||
answer_as(client, "gateway", gateway_source.parse(gateway_snapshot("full"), "gateway"))
|
||||
answer_as(client, "gateway", payload)
|
||||
deadline = time.time() + 10
|
||||
while time.time() < deadline and not client.app.state.ctx.hub.jobs["gateway"].state.ok:
|
||||
time.sleep(0.05)
|
||||
block = gateway_block(client)
|
||||
assert block["listeners"] == 10, "6 haproxy binds + 4 nginx listen ports"
|
||||
assert block["connections"] == 32, "every route the two captured configs declare"
|
||||
assert block["live_in"] == 9, "9 of the 18 live sockets face a configured listen port"
|
||||
assert block["live_out"] == 2, "the two loopback endpoints, counted once each"
|
||||
assert block["map_targets"] == 17, "the stream map's routing entries are the fan-out"
|
||||
assert block["connections"] == 33, "every route the two captured configs declare"
|
||||
assert block["map_targets"] == 18, "the stream map's routing entries are the fan-out"
|
||||
# The two live totals are the volatile half: a capture of a running router is a different set of
|
||||
# connections every second, so they are compared with the payload the block was built from
|
||||
# rather than with a number typed here (the doctrine this suite already keeps for the socket
|
||||
# fixtures themselves — assert the shape and the rule, never a particular row). What stays
|
||||
# literal is what only the two config files decide, above, and what stays asserted as a
|
||||
# relationship is the direction of the error, below.
|
||||
assert block["live_in"] == sum(row["connections"] or 0 for row in payload["listeners"])
|
||||
_, expected_out = endpoint_sums(payload)
|
||||
assert block["live_out"] == expected_out, "one count per distinct endpoint, never one per line"
|
||||
assert sum(row["connections"] or 0 for row in payload["connections"]) >= (expected_out or 0), (
|
||||
"the lines repeat a shared endpoint's count, which is exactly why the block adds the "
|
||||
"endpoints up instead")
|
||||
assert block["live_in"] + block["live_out"] <= payload["sockets"]["established"], (
|
||||
"the reported figures may under-count the kernel's, never exceed it")
|
||||
assert block["warnings"] == 0, (
|
||||
"this fleet's two config files connect to themselves, and 0 is the truth rather than a "
|
||||
"missing measurement — the overview line prints nothing for it either way")
|
||||
@@ -5106,6 +5169,13 @@ def test_one_dangling_use_backend_is_one_warn_row_on_the_overview(
|
||||
deadline = time.time() + 15
|
||||
while time.time() < deadline and not client.app.state.ctx.hub.jobs["gateway"].state.ok:
|
||||
time.sleep(0.05)
|
||||
# The two host jobs are this test's premise, not its subject: on a slow machine the wait
|
||||
# loops above outlive their 1.5 s staleness window, and the alert then gains a `:stale`
|
||||
# row for the premise and the assertion below reads a three-row list. Stamping their
|
||||
# `last_ok` pins the premise the docstring states ("the agent answers") and leaves the
|
||||
# gateway row the only line.
|
||||
for host_key in ("host:gateway", "disks:gateway"):
|
||||
client.app.state.ctx.hub.jobs[host_key].state.last_ok = time.time()
|
||||
rows = gateway_alerts(client)
|
||||
block = client.get("/api/summary").json()["gateway"]
|
||||
assert len(rows) == 1, rows
|
||||
@@ -5123,6 +5193,8 @@ def test_one_dangling_use_backend_is_one_warn_row_on_the_overview(
|
||||
deadline = time.time() + 15
|
||||
while time.time() < deadline and client.get("/api/summary").json()["gateway"]["warnings"]:
|
||||
time.sleep(0.05)
|
||||
for host_key in ("host:gateway", "disks:gateway"):
|
||||
client.app.state.ctx.hub.jobs[host_key].state.last_ok = time.time()
|
||||
assert gateway_alerts(client) == [], "a clean ingress is an all clear, not a green line"
|
||||
|
||||
|
||||
@@ -5164,6 +5236,13 @@ def test_a_gateway_job_that_stops_answering_keeps_one_row_not_two(
|
||||
time.sleep(0.05)
|
||||
else:
|
||||
pytest.fail("the gateway job never went from answering to failing")
|
||||
# The two host jobs are this test's premise, not its subject: on a slow machine the wait
|
||||
# loops above outlive their 1.5 s staleness window, and the alert then gains a `:stale`
|
||||
# row for the premise and the assertion below reads a three-row list. Stamping their
|
||||
# `last_ok` pins the premise the docstring states ("the agent answers") and leaves the
|
||||
# gateway row the only line.
|
||||
for host_key in ("host:gateway", "disks:gateway"):
|
||||
client.app.state.ctx.hub.jobs[host_key].state.last_ok = time.time()
|
||||
alerts = gateway_alerts(client)
|
||||
assert [row["id"] for row in alerts] == ["gateway:gateway:down"], (
|
||||
"the reachability row is the gateway's only line while it is down")
|
||||
@@ -5428,17 +5507,16 @@ def find_keys(node: Any, name: str) -> list[Any]:
|
||||
return []
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gateway_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
||||
gateway_snapshot: Any) -> Iterator[TestClient]:
|
||||
"""The ingress answering with the captured configs and sockets, its gauges already stored.
|
||||
def _gateway_routes_client(tmp_path: Path, mode: str,
|
||||
gateway_snapshot: Any) -> Iterator[TestClient]:
|
||||
"""One app per capture mode, its gauges already stored, its payload already in the job state.
|
||||
|
||||
Both halves come from `sources/gateway` over the **captured** tree, so every path a page reads
|
||||
and every series name a chart asks for is one the source really emits (rule 7). The target
|
||||
points at a dead port: no poll can overwrite the payload, dial out, or add a second sample
|
||||
between the store read and the route assertion below.
|
||||
"""
|
||||
db = str(tmp_path / "gateway-routes.sqlite3")
|
||||
db = str(tmp_path / f"gateway-routes-{mode}.sqlite3")
|
||||
store = SampleStore(db)
|
||||
app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
||||
pbkdf2_iterations=FAST_PBKDF2,
|
||||
@@ -5446,7 +5524,7 @@ def gateway_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
||||
with TestClient(app) as test_client:
|
||||
sign_in(test_client)
|
||||
now = time.time()
|
||||
snapshot = gateway_snapshot("full")
|
||||
snapshot = gateway_snapshot(mode)
|
||||
snapshot["ts"] = now # "the agent answered just now", so charts are in range
|
||||
result = gateway_source.parse_gateway_snapshot("gateway", snapshot, now=now)
|
||||
store.add_many(result.samples)
|
||||
@@ -5455,6 +5533,31 @@ def gateway_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def gateway_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
||||
gateway_snapshot: Any) -> Iterator[TestClient]:
|
||||
"""The ingress answering as agent **1.2.0** (or 1.3.0 with `--no-resolve`) does.
|
||||
|
||||
No `resolver` section, so the routes whose targets the two config files write as domains carry
|
||||
no count, no gauge and no stored series — phase 30 verbatim, which is what the phase-32 pins
|
||||
below compare against.
|
||||
"""
|
||||
yield from _gateway_routes_client(tmp_path, "full", gateway_snapshot)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def resolved_gateway_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
||||
gateway_snapshot: Any) -> Iterator[TestClient]:
|
||||
"""The same ingress answering as agent **1.3.0**: the captured resolver section attached.
|
||||
|
||||
The document is the recorded `gateway_resolver.json`, so every series name this fixture's store
|
||||
holds is one the deployed ingress will really write after task 05 — including the **name-keyed**
|
||||
outbound gauges phase 30 stopped writing, which are writable again precisely because the
|
||||
gateway published the address behind the name.
|
||||
"""
|
||||
yield from _gateway_routes_client(tmp_path, "resolved", gateway_snapshot)
|
||||
|
||||
|
||||
def dial_spies(monkeypatch: pytest.MonkeyPatch, calls: list[str]) -> None:
|
||||
"""Two spies that raise: a route that dials before its gate fails here, not in a timeout.
|
||||
|
||||
@@ -5568,11 +5671,19 @@ def test_gateway_snapshot_answers_the_pipes_the_projection_joined(
|
||||
"the deleted join is back in the payload")
|
||||
|
||||
# the config half, verbatim from the capture the agent read (phase 28's six frontends, the
|
||||
# stream map's 17 routing entries) — the echo the pipes were built from
|
||||
# stream map's 18 routing entries) — the echo the pipes were built from
|
||||
assert len(body["haproxy"]["frontends"]) == 6
|
||||
assert len(body["nginx"]["map_targets"]) == 17
|
||||
assert len(body["nginx"]["map_targets"]) == 18
|
||||
# `/proc/net/tcp` is one row per live connection, so every count below is derived from the same
|
||||
# capture the payload was built from instead of typed here: the rule is the pin, the number is a
|
||||
# reading. The config-shaped figures above stay literal, because only the operator moves those.
|
||||
established = gateway_snapshot("full")["sockets"]["established"]
|
||||
|
||||
stream = next(pipe for pipe in body["listeners"] if pipe["port"] == 443)
|
||||
assert stream["connections"] == 6 and stream["metric"] == "gateway.in_conns@443", (
|
||||
assert stream["connections"] == sum(1 for row in established
|
||||
if row["local"]["port"] == 443), (
|
||||
"the inbound pipe carries the sockets whose LOCAL port it is, which is the whole join")
|
||||
assert stream["metric"] == "gateway.in_conns@443", (
|
||||
"the pipe carries the series it is, so the page never rebuilds the name")
|
||||
assert stream["clients"], "the inbound pipe names the addresses that are using it"
|
||||
secure = next(pipe for pipe in body["listeners"] if pipe["port"] == 4443)
|
||||
@@ -5581,14 +5692,19 @@ def test_gateway_snapshot_answers_the_pipes_the_projection_joined(
|
||||
# The graph: every route the two captured configs declare, each line's inbound end a trunk this
|
||||
# same payload draws, and a count on **only** the lines whose endpoint the config wrote as an
|
||||
# address — this capture has exactly two (`127.0.0.1:4443` and `127.0.0.1:8081`).
|
||||
assert len(body["connections"]) == 32
|
||||
assert len(body["connections"]) == 33
|
||||
listener_ports = {listener["port"] for listener in body["listeners"]}
|
||||
assert listener_ports == {22, 2000, 5201, 8080, 25565, 25566, 80, 443, 4443, 8443}
|
||||
assert all(row["in"]["port"] in listener_ports for row in body["connections"]), (
|
||||
"a connection on a port no listener carries is an orphan the page cannot draw")
|
||||
def sockets_to(host: str, port: int) -> int:
|
||||
return sum(1 for row in established
|
||||
if (row["remote"]["addr"], row["remote"]["port"]) == (host, port))
|
||||
|
||||
counted = {row["id"]: row["connections"] for row in body["connections"]
|
||||
if row["connections"] is not None}
|
||||
assert counted == {"nginx:443->127.0.0.1:4443": 2, "nginx:8443->127.0.0.1:8081": 0}, (
|
||||
assert counted == {"nginx:443->127.0.0.1:4443": sockets_to("127.0.0.1", 4443),
|
||||
"nginx:8443->127.0.0.1:8081": sockets_to("127.0.0.1", 8081)}, (
|
||||
"a route written as a domain carries no count, and one written as an address carries the "
|
||||
"endpoint's own — 0 included, because the socket table answered")
|
||||
assert all(row["out"]["address"] is True for row in body["connections"]
|
||||
@@ -5600,11 +5716,88 @@ def test_gateway_snapshot_answers_the_pipes_the_projection_joined(
|
||||
if pipe["connections"] is not None)
|
||||
endpoints = {(row["out"]["host"], row["out"]["port"]) for row in body["connections"]
|
||||
if row["connections"] is not None}
|
||||
assert known_in + sum(counted.values()) <= body["sockets"]["established"] == 18
|
||||
assert known_in + sum(counted.values()) <= body["sockets"]["established"] == len(established)
|
||||
assert len(endpoints) == 2
|
||||
assert [row["key"] for row in body["states"]] == ["gateway"]
|
||||
|
||||
|
||||
def test_the_served_snapshot_counts_a_resolved_name_and_says_which_address_it_used(
|
||||
resolved_gateway_client) -> None:
|
||||
"""`/api/gateway/snapshot` on the 1.3.0 document: the dash is gone, and the reason is in the row.
|
||||
|
||||
One route, both halves of the claim. `brain.reeseapps.com` leaves this ingress for
|
||||
`reeseapps-brain-of-reese.reeselink.com:443`; the row now carries `address: false` (the config
|
||||
wrote a name — verbatim, still), `resolved: ["10.1.2.64"]`, `via: "libc"` and that entry's own
|
||||
`resolved_ts`, **and** a finite `connections`. A number with no provenance beside it would be
|
||||
the invention this app refuses to print, which is why the two halves are one test.
|
||||
"""
|
||||
body = resolved_gateway_client.get("/api/gateway/snapshot").json()
|
||||
assert body["resolver"]["available"] is True and body["resolver"]["via"] == "libc"
|
||||
brain = next(row for row in body["connections"]
|
||||
if row["out"]["host"] == "reeseapps-brain-of-reese.reeselink.com")
|
||||
assert brain["out"]["selectors"] == ["brain.reeseapps.com"]
|
||||
assert brain["out"]["address"] is False
|
||||
assert brain["out"]["resolved"] == ["10.1.2.64"]
|
||||
assert brain["out"]["via"] == "libc"
|
||||
assert isinstance(brain["out"]["resolved_ts"], (int, float))
|
||||
assert isinstance(brain["connections"], int) and not isinstance(brain["connections"], bool), (
|
||||
"resolved and idle is a `0`, not the dash: the socket table answered and this endpoint has none")
|
||||
assert brain["metric"] == "gateway.out_conns@reeseapps-brain-of-reese.reeselink.com:443"
|
||||
|
||||
ntfy = next(row for row in body["connections"]
|
||||
if row["out"]["host"] == "reeseapps-ntfy.reeselink.com")
|
||||
assert ntfy["connections"] > 0, "the capture has live ntfy sockets behind that name"
|
||||
counted = [row for row in body["connections"] if row["connections"] is not None]
|
||||
assert len(counted) == len(body["connections"]), (
|
||||
"every route this ingress declares is placed by the recorded resolver, so no route on the "
|
||||
"live page is a dash any more — the operator's complaint, in one assertion")
|
||||
assert all(row["out"]["address"] is True or row["out"]["resolved"] for row in counted), (
|
||||
"a count never arrives without the address it was counted through — one the config wrote, "
|
||||
"or one the gateway published; `resolved` stays null on the address rows, because copying "
|
||||
"the agent's `literal` echo of a string the config already wrote would claim a lookup that "
|
||||
"never happened")
|
||||
|
||||
|
||||
def test_an_injected_nxdomain_row_is_a_dash_in_the_served_document(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch, gateway_snapshot: Any) -> None:
|
||||
"""Both arms on one route list: placed is a number, unplaced is `null` / `null`.
|
||||
|
||||
The capture resolves everything, so the second arm is injected: one entry of the recorded
|
||||
document emptied and marked `nxdomain`, exactly what agent 1.3.0 publishes when `getaddrinfo`
|
||||
says a name has no record. Its line keeps its place in the graph with `resolved: null`,
|
||||
`via: null`, `connections: null` and `metric: null` — no `0`, no series, no sample — while the
|
||||
routes around it carry their counts, so a page cannot read one failed name as a dead ingress.
|
||||
"""
|
||||
broken = gateway_snapshot("resolved")
|
||||
entries = [entry for entry in broken["resolver"]["entries"]
|
||||
if entry["name"] == "reeseapps-immich.reeselink.com"]
|
||||
assert entries, "the capture resolves immich, so the injection has something to un-resolve"
|
||||
for entry in entries:
|
||||
entry["addresses"] = []
|
||||
entry["status"] = "nxdomain"
|
||||
payload = gateway_source.parse(broken, "gateway")
|
||||
|
||||
with make_gateway_client(tmp_path, monkeypatch) as client:
|
||||
answer_as(client, "gateway", payload)
|
||||
deadline = time.time() + 10
|
||||
while time.time() < deadline and not client.app.state.ctx.hub.jobs["gateway"].state.ok:
|
||||
time.sleep(0.05)
|
||||
body = client.get("/api/gateway/snapshot").json()
|
||||
|
||||
row = next(item for item in body["connections"]
|
||||
if item["out"]["host"] == "reeseapps-immich.reeselink.com")
|
||||
assert row["out"]["resolved"] is None, "never `[]`, so the page's dash arm is one `is null` test"
|
||||
assert row["out"]["via"] is None and row["out"]["resolved_ts"] is None
|
||||
assert row["connections"] is None and row["metric"] is None
|
||||
assert row["in"] == {"port": 443}, "the route is still drawn: a name with no record is not no route"
|
||||
placed = [item for item in body["connections"] if item["connections"] is not None]
|
||||
assert len(placed) == len(body["connections"]) - 1, (
|
||||
"one unresolved name dims one line and nothing else")
|
||||
documented = {entry["name"]: entry["status"] for entry in body["resolver"]["entries"]}
|
||||
assert documented["reeseapps-immich.reeselink.com"] == "nxdomain", (
|
||||
"the entry the index refused to use is still the entry the page can quote")
|
||||
|
||||
|
||||
def test_no_port_the_two_configs_do_not_name_appears_anywhere(gateway_client,
|
||||
gateway_snapshot: Any) -> None:
|
||||
"""The live router's own agent port is `9120`, and this payload never mentions it.
|
||||
@@ -5700,10 +5893,12 @@ def test_gateway_history_serves_the_pipe_gauges_the_store_holds(
|
||||
facts), and the `source` says the `gateway` family — a page must never ask the 10-second `host`
|
||||
series for a connection count.
|
||||
|
||||
The outbound name asked for is an **address** (`127.0.0.1:4443`), and that is the whole point of
|
||||
the phase-30 change to `samples()`: a route whose target the config wrote as a domain publishes
|
||||
no series at all, so a name-keyed `gateway.out_conns@<domain>:<port>` is no longer stored and
|
||||
the store's `gateway` family is exactly the endpoints the join could count.
|
||||
The outbound name asked for is an **address** (`127.0.0.1:4443`) because this fixture answers as
|
||||
a 1.2.0 agent. The rule `samples()` keeps is about the **address**, not the spelling: a route
|
||||
publishes a gauge when the kernel can name its far end — an address the config wrote, or (since
|
||||
phase 32 task 03, pinned beside this one on the `resolved` arm) a name the gateway itself
|
||||
resolved. A name nobody resolved publishes nothing, which is what
|
||||
`test_gateway_history_answers_empty_for_a_pipe_it_never_measured` still proves.
|
||||
"""
|
||||
store: SampleStore = gateway_client.app.state.ctx.store # type: ignore[attr-defined]
|
||||
body = gateway_client.get("/api/gateway/history", params={
|
||||
@@ -5743,6 +5938,65 @@ def test_gateway_history_answers_empty_for_a_pipe_it_never_measured(gateway_clie
|
||||
assert series_source(body) == {"gateway"}
|
||||
|
||||
|
||||
def test_a_resolved_name_publishes_its_gauge_and_history_answers_it(
|
||||
resolved_gateway_client) -> None:
|
||||
"""The name-keyed series phase 30 stopped writing, written again on an honest footing.
|
||||
|
||||
Phase 30 task 02 retired `gateway.out_conns@<domain>:<port>` because the only figure it could
|
||||
have carried was the *port's* — the port-only join's count filed under whatever domain happened
|
||||
to share it. Phase 32 task 03 gives the name a figure of its own: the sockets to the address the
|
||||
**gateway** resolved that name to. So the name is stored, `/api/metrics` lists it, and
|
||||
`/api/gateway/history` answers points for it — read through `store.query`, the third path, as
|
||||
every other history pin is.
|
||||
"""
|
||||
store: SampleStore = resolved_gateway_client.app.state.ctx.store # type: ignore[attr-defined]
|
||||
metric = "gateway.out_conns@reeseapps-ntfy.reeselink.com:443"
|
||||
body = resolved_gateway_client.get("/api/gateway/history", params={
|
||||
"hosts": "gateway", "metrics": metric, "range": "1h"}).json()
|
||||
|
||||
assert [series["metric"] for series in body["series"]] == [metric]
|
||||
assert series_source(body) == {"gateway"}
|
||||
rows = store.query("gateway", "gateway", metric, res=RES_RAW)
|
||||
assert rows, "the resolved route's gauge was not stored"
|
||||
assert [(point["ts"], point["value"]) for point in body["series"][0]["points"]] == rows
|
||||
assert metric in resolved_gateway_client.get("/api/metrics",
|
||||
params={"source": "gateway"}).json()["metrics"]
|
||||
|
||||
|
||||
def test_one_shared_endpoint_costs_one_insert_per_poll_even_through_a_name(
|
||||
tmp_path: Path, gateway_snapshot: Any) -> None:
|
||||
"""Two lines, one gauge name, **one row per poll** — counted in the store, not asserted.
|
||||
|
||||
The captured ingress reaches `keycloak.reeselink.com:8080` from `:4443` and from `:8443`, and
|
||||
since task 03 the endpoint is reachable through a **resolved name** rather than a written
|
||||
address. `shared: true` says the two lines carry one number; `samples()` de-duplicating by
|
||||
metric is what stops that number being inserted twice for one reading, and the row count is the
|
||||
only proof worth having. Two polls, two rows: one per reading, never one per line.
|
||||
"""
|
||||
db = str(tmp_path / "gateway-shared.sqlite3")
|
||||
store = SampleStore(db)
|
||||
now = time.time()
|
||||
metric = "gateway.out_conns@keycloak.reeselink.com:8080"
|
||||
# both readings in the past: `store.query` bounds its window at now, so a simulated second poll
|
||||
# that lands in the future is a row that never comes back, not a lost insert
|
||||
for offset in (-60.0, -30.0):
|
||||
snapshot = gateway_snapshot("resolved")
|
||||
snapshot["ts"] = now + offset
|
||||
result = gateway_source.parse_gateway_snapshot("gateway", snapshot, now=now + offset)
|
||||
shared = [row for row in result.payload["connections"] if row["shared"]]
|
||||
assert {row["metric"] for row in shared} == {metric}, (
|
||||
"this capture's shared endpoint is the keycloak pair, and one poll writes it once")
|
||||
assert len({row["connections"] for row in shared}) == 1, (
|
||||
"both lines carry the same number, which is what makes one insert the honest count")
|
||||
out = [row for row in result.samples if row.metric == metric]
|
||||
assert len(out) == 1, "one sample per poll, however many lines reach the endpoint"
|
||||
store.add_many(result.samples)
|
||||
|
||||
rows = store.query("gateway", "gateway", metric, res=RES_RAW)
|
||||
assert len(rows) == 2, "two polls, two rows — not four (one per line) and not one (lost poll)"
|
||||
store.close()
|
||||
|
||||
|
||||
def test_gateway_history_names_the_ingress_and_no_other_host(gateway_client) -> None:
|
||||
"""With no `hosts=` the answer is the one host this source reads — not the whole fleet.
|
||||
|
||||
@@ -5853,6 +6107,10 @@ GATEWAY_PAGE_FIELDS = (
|
||||
"connections.id", "connections.source", "connections.in.port",
|
||||
"connections.out.host", "connections.out.port", "connections.out.name",
|
||||
"connections.out.kind", "connections.out.selectors", "connections.out.address",
|
||||
# Phase 32 task 04: the provenance a counted line prints and the dash refuses. `resolved` is what
|
||||
# the filter matches an IP against, `via` + `resolved_ts` are the tooltip's `address from libc,
|
||||
# resolved 3m` — so a rename of any of the three fails by name here, not as a permanent silence.
|
||||
"connections.out.resolved", "connections.out.via", "connections.out.resolved_ts",
|
||||
"connections.connections", "connections.shared", "connections.metric",
|
||||
"warnings.id", "warnings.kind", "warnings.subject", "warnings.message",
|
||||
)
|
||||
@@ -6415,11 +6673,16 @@ def test_the_trunk_pipe_is_an_accessible_image_of_its_port(client, gateway_clien
|
||||
|
||||
|
||||
def _leaf_haystack(row: dict[str, Any]) -> str:
|
||||
"""The one string the filter can see about one route, as `page-gateway.js` composes it."""
|
||||
"""The one string the filter can see about one route, as `page-gateway.js` composes it.
|
||||
|
||||
Phase 32 task 04 adds the **sixth** field: `out.resolved`, the addresses the gateway published for
|
||||
a name target. Typing an IP is how an operator asks "which route owns the socket I am looking at",
|
||||
and a haystack of names alone cannot answer it.
|
||||
"""
|
||||
out = row.get("out") or {}
|
||||
port = (row.get("in") or {}).get("port")
|
||||
parts = ["" if port is None else str(port), out.get("host"), out.get("name"), out.get("kind"),
|
||||
*(out.get("selectors") or [])]
|
||||
*(out.get("selectors") or []), *(out.get("resolved") or [])]
|
||||
return " ".join(p for p in parts if isinstance(p, str) and p).lower()
|
||||
|
||||
|
||||
@@ -6434,7 +6697,8 @@ def filter_view(snapshot: dict[str, Any], query: str) -> dict[str, Any]:
|
||||
|
||||
There is no JS runner (a locked anchor), so this is `rendered_leaf`'s idiom: the rules are
|
||||
re-applied here from the payload, and the served-byte pins in the test below are what keep the
|
||||
mirror honest — `haystack` indexes `in.port`, `out.host`, `out.name`, `out.kind`, `out.selectors`;
|
||||
mirror honest — `haystack` indexes `in.port`, `out.host`, `out.name`, `out.kind`,
|
||||
`out.selectors`, `out.resolved`;
|
||||
`trunkHaystack` indexes `:${port}`, `domains`, `clients`; a trunk that matches keeps **all** its
|
||||
leaves; a group stands when it or one of its routes answers. A rename on either side therefore
|
||||
fails a pin *and* changes what this mirror prints.
|
||||
@@ -6476,7 +6740,8 @@ def filter_view(snapshot: dict[str, Any], query: str) -> dict[str, Any]:
|
||||
return {"groups": groups, "standing": standing, "total": len(snapshot["connections"])}
|
||||
|
||||
|
||||
def test_the_filter_narrows_the_map_without_asking_anything(client, gateway_client) -> None:
|
||||
def test_the_filter_narrows_the_map_without_asking_anything(client, gateway_client,
|
||||
gateway_snapshot: Any) -> None:
|
||||
"""One field, four rules, and not one extra request.
|
||||
|
||||
The operator's complaint is a reading-speed one: *"I want to find a domain."* This ingress routes
|
||||
@@ -6485,9 +6750,12 @@ def test_the_filter_narrows_the_map_without_asking_anything(client, gateway_clie
|
||||
feature could go wrong:
|
||||
|
||||
* **It matches payload text and nothing else.** `in.port`, `out.host`, `out.name`, `out.kind`,
|
||||
`out.selectors` for a route, `:${port}` + `domains` + `clients` for a trunk. No DNS (the module
|
||||
still resolves no name), no re-ranking, no rule about `.reeselink.com` — a query can never
|
||||
invent a route, because there is no other text to match.
|
||||
`out.selectors` and `out.resolved` for a route, `:${port}` + `domains` + `clients` for a trunk.
|
||||
The sixth field is phase 32 task 04's: an address the **gateway** published is text the payload
|
||||
already carries, so typing the far end of a socket finds the route that owns it. No DNS (the
|
||||
page still resolves no name — it matches a string, it never looks one up), no re-ranking, no
|
||||
rule about `.reeselink.com` — a query can never invent a route, because there is no other text
|
||||
to match.
|
||||
* **It costs nothing.** The repaint runs on `state.snapshot`; a keystroke that re-fetched would be
|
||||
a poll per character and a lie about what changed.
|
||||
* **It hides rows, it does not remove them.** `li.hidden` in the patch pass, so a route the query
|
||||
@@ -6498,18 +6766,27 @@ def test_the_filter_narrows_the_map_without_asking_anything(client, gateway_clie
|
||||
"""
|
||||
js = _code(client.get("/static/page-gateway.js").text)
|
||||
|
||||
# (1) the matcher's whole vocabulary: five payload fields for a route, three for a trunk.
|
||||
# (1) the matcher's whole vocabulary: six payload fields for a route, three for a trunk.
|
||||
hay = js[js.index("function haystack("):js.index("function trunkHaystack(")]
|
||||
assert "String((conn.in || {}).port ?? '')" in hay, \
|
||||
"the inbound port is no longer searchable — `4443` is the second thing an operator types"
|
||||
for field in ("out.host", "out.name", "out.kind", "out.selectors"):
|
||||
for field in ("out.host", "out.name", "out.kind", "out.selectors", "out.resolved"):
|
||||
assert field in hay, f"the filter no longer sees {field}"
|
||||
# Phase 32 task 04: an address the **gateway** resolved is payload text like any other, so typing
|
||||
# the far end of a socket finds the route that owns it. Names alone could not answer that question.
|
||||
assert "...(out.resolved || [])" in hay, \
|
||||
"a name-written route stopped being findable by the address it resolves to"
|
||||
assert "toLowerCase()" in hay, "the match became case-sensitive on a page of lowercase domains"
|
||||
for absent in ("conn.connections", "severity", "metric", "getaddrinfo", "split(",
|
||||
"endsWith", "startsWith", "slice("):
|
||||
assert absent not in hay, (
|
||||
f"the matcher reached for {absent} — it may read what the payload calls a route, never "
|
||||
"what it reports about it, and never resolve a name")
|
||||
# …and the three things that would stop it being an eye over the document in hand: a dial, a read
|
||||
# of what is painted rather than of what the payload says, and a literal `null` a query could
|
||||
# match against a row that published nothing.
|
||||
for not_here in ("fetchJSON", "textContent", "innerText", '"null"', "'null'"):
|
||||
assert not_here not in hay, f"the haystack reached for {not_here}"
|
||||
trunk = js[js.index("function trunkHaystack("):js.index("function filtering(")]
|
||||
for field in ("group.port", "listener.domains", "listener.clients"):
|
||||
assert field in trunk, f"the trunk no longer answers for {field}"
|
||||
@@ -6616,6 +6893,25 @@ def test_the_filter_narrows_the_map_without_asking_anything(client, gateway_clie
|
||||
assert all(not leaf["hidden"] for leaf in home["leaves"]), \
|
||||
"a trunk that matched lost leaves of its fan-out"
|
||||
|
||||
# (7) phase 32 task 04: an address the **gateway** resolved is searchable too, which is the field
|
||||
# an operator reaches for when they are holding a socket rather than a domain. Over the 1.2.0
|
||||
# document there is nothing to match (no resolver section, and the page resolves nothing), so
|
||||
# this leg runs on the recorded 1.3.0 one.
|
||||
placed = gateway_source.parse(gateway_snapshot("resolved"), "gateway")
|
||||
assert not filter_view(payload, "10.1.2.64")["standing"], \
|
||||
"the page found an address it was never given — it resolves nothing, and must keep not doing it"
|
||||
brain_ip = filter_view(placed, "10.1.2.64")
|
||||
found = [leaf["id"] for group in brain_ip["groups"] for leaf in group["leaves"]
|
||||
if not leaf["hidden"]]
|
||||
assert found and all("reeseapps-brain-of-reese" in route_id for route_id in found), \
|
||||
f"typing the brain's address left {found}"
|
||||
assert all(group["hidden"] or group["port"] == 443 for group in brain_ip["groups"]), \
|
||||
"an address match let a trunk of another port stand"
|
||||
shared_ip = filter_view(placed, "10.1.1.78")
|
||||
expected = sum(1 for row in placed["connections"] if "10.1.1.78" in (row["out"]["resolved"] or []))
|
||||
assert expected > 1 and shared_ip["standing"] == expected, (
|
||||
f"one address reached by {expected} routes was found by {shared_ip['standing']} of them")
|
||||
|
||||
|
||||
def test_the_filter_field_is_markup_the_sheet_and_nobody_else_uses(client) -> None:
|
||||
"""The field is pinned markup, on one page, wearing the sheet's one field skin.
|
||||
@@ -6972,19 +7268,39 @@ def test_the_page_says_where_a_line_comes_from_and_where_its_count_cannot(client
|
||||
"""Four footnotes, one per claim the page is allowed to make, and not one more.
|
||||
|
||||
Why the page draws a line at all (the config declares the route) and what its motion means (a
|
||||
socket is open this instant, so a resting line is not a dead one); why an endpoint written as a
|
||||
**name** carries no number (nothing here resolves one, so a dash is not a fault); what the switch
|
||||
does **not** do (toggling a route needs a config write path this app does not have); and — since
|
||||
phase 31 task 04 — what the filter box does to the map and what it leaves alone (the tallies, the
|
||||
warnings and the charts keep describing the whole ingress, because the field asks for nothing).
|
||||
Each lives in the footer exactly once — the page does not repeat a why-line per panel, and it
|
||||
does not need to: an uncounted line prints `—` and the footnote is the sentence that explains it
|
||||
socket is open this instant, so a resting line is not a dead one); why a route carries no number
|
||||
(phase 32 task 04: **the gateway could not resolve that name in this reading** — the gateway does
|
||||
the resolving, so the sentence blames the reading and points at the hover that shows the address a
|
||||
counted line was matched to, instead of blaming the config for having written a name or the page
|
||||
for having no resolver); what the switch does **not** do (toggling a route needs a config write
|
||||
path this app does not have); and — since phase 31 task 04 — what the filter box does to the map
|
||||
and what it leaves alone (the tallies, the warnings and the charts keep describing the whole
|
||||
ingress, because the field asks for nothing).
|
||||
Each lives in the footer exactly once — the page does not repeat a why-line per panel, and it does
|
||||
not need to: an uncounted line prints `—` and the footnote is the sentence that explains it
|
||||
(rule 2).
|
||||
"""
|
||||
html = client.get(GATEWAY_PAGE).text
|
||||
footer = html[html.index("<footer"):]
|
||||
assert footer.count("resolves nothing") == 1, (
|
||||
"the no-DNS rule is the page's single footnote about a missing count — printed once")
|
||||
assert footer.count("a dash means") == 1, (
|
||||
"the page's single footnote about a missing count, printed once, in the phase-32 reason")
|
||||
assert "could not resolve that name" in footer, (
|
||||
"a dash means the gateway's reading came back with nothing, not that the config wrote a name")
|
||||
# The two sentences phase 32 made stale may not come back from either file: the blame on the page
|
||||
# (`the page knows no`) and its no-DNS limit as a per-row excuse, which task 04 replaced with the
|
||||
# thing to do about a dash rather than a reason to accept it.
|
||||
assert "the page knows no" not in footer, (
|
||||
"the dash stopped blaming the gateway's reading and went back to blaming the page")
|
||||
# The served source wraps this paragraph, so the two halves are matched separately: what is
|
||||
# pinned is that the sentence names a cause **and** the place to look.
|
||||
flat = " ".join(footer.split())
|
||||
assert "hover a counted line for the address it matched" in flat, (
|
||||
"the footnote names a cause and no longer says where the address and its age are")
|
||||
script = client.get("/static/page-gateway.js").text
|
||||
assert "the gateway could not resolve this name in this reading" in script, (
|
||||
"the per-row tooltip must say the same thing the footnote says")
|
||||
assert "nothing here resolves names" not in script, (
|
||||
"the old tooltip sentence blames the name, which is no longer why a line has no count")
|
||||
assert "animates" in footer, "the animation rule lost its footnote"
|
||||
band = footer[:footer.index("</footer>")]
|
||||
# Four paragraphs and nothing else: one sentence per claim, in one ink band under the panels.
|
||||
@@ -7054,6 +7370,21 @@ def test_the_gateway_page_renders_each_degradation_arm_in_the_payloads_own_words
|
||||
assert sum(1 for field in ("listeners", "connections") if stub[field]) == 0, (
|
||||
"the stub arm's rows are not empty, so the header's `n/a` would not be about the same thing")
|
||||
|
||||
# (5) a **resolver** that did not answer (agent 1.2.0, `--no-resolve`, or a pass that could not
|
||||
# run) is the fifth arm and the quietest: the section is not one of the two config files, so
|
||||
# it raises no banner and no colour, and every name-written line simply carries the dash with
|
||||
# the resolution sentence. `full` is that document — no `resolver` key at all.
|
||||
assert silent["resolver"]["available"] is None and silent["resolver"]["entries"] == [], (
|
||||
"the `full` capture is supposed to be the no-resolver arm this case measures")
|
||||
assert silent["why"] is None, "an absent resolver section became a page-wide complaint"
|
||||
assert "severity" not in json.dumps(silent), "the fifth arm arrived with a verdict on it"
|
||||
dashes = [row for row in silent["connections"] if row["connections"] is None]
|
||||
assert dashes, "a document with no resolver counted a name-written route"
|
||||
assert all(count_surfaces(row)["why"] == COUNT_WHY_UNRESOLVED for row in dashes), \
|
||||
"a line nobody resolved is being explained by something other than resolution"
|
||||
assert not any(count_surfaces(row)["glyph"] == "0" for row in dashes), \
|
||||
"the uncounted became a claimed zero"
|
||||
|
||||
|
||||
def test_the_gateway_page_keeps_its_last_drawing_when_a_poll_fails(client) -> None:
|
||||
"""Invariant 4 on the fifth page: a failed read leaves the last picture on screen.
|
||||
@@ -7072,6 +7403,43 @@ def test_the_gateway_page_keeps_its_last_drawing_when_a_poll_fails(client) -> No
|
||||
"and the reason has to be printed where the operator is already looking"
|
||||
|
||||
|
||||
COUNT_WHY_UNRESOLVED = "the gateway could not resolve this name in this reading"
|
||||
COUNT_WHY_NO_SOCKETS = "the socket table did not answer this reading"
|
||||
|
||||
|
||||
def count_surfaces(row: dict[str, Any]) -> dict[str, Any]:
|
||||
"""The count column's two strings for one route, as `countWhy` / `countTitle` build them.
|
||||
|
||||
A **second** reading of the same decision, with the served-byte pins below as its other half:
|
||||
|
||||
* `glyph` — the em dash for a null, the number otherwise (`0` included, because a resolved
|
||||
endpoint with no socket is a measurement and not a gap);
|
||||
* `why` — for a null, which of the two unknowns the payload describes: an address is **known**
|
||||
when the config wrote one (`out.address is True`) or the gateway published one
|
||||
(`out.resolved` non-empty), and a row with neither is a name nothing could resolve;
|
||||
* `title` — for a counted row, the number plus the provenance the tooltip owes it. The **age**
|
||||
fragment is left out here and pinned as a prefix instead, because it is a clock: the test
|
||||
asserts what comes before it and what the remainder looks like.
|
||||
"""
|
||||
out = row["out"]
|
||||
shown = row["connections"]
|
||||
if shown is None:
|
||||
known = out["address"] is True or bool(out["resolved"])
|
||||
why = COUNT_WHY_NO_SOCKETS if known else COUNT_WHY_UNRESOLVED
|
||||
return {"glyph": "\u2014", "why": why, "title": f"no number: {why}",
|
||||
"aria": f"connections not attributed: {why}", "age": False}
|
||||
title = f"{shown} open connection(s) on this endpoint"
|
||||
via = out["via"] if isinstance(out["via"], str) and out["via"] else None
|
||||
if out["resolved"] is None or via is None:
|
||||
return {"glyph": str(shown), "why": None, "title": title, "age": False,
|
||||
"aria": f"{shown} live connections"}
|
||||
title += f" \u00b7 address from {via}"
|
||||
# `Dash.duration(Dash.ageSeconds(ts))`, whose text is a clock and therefore not this mirror's to
|
||||
# reproduce — only the *decision* to print it is.
|
||||
return {"glyph": str(shown), "why": None, "title": title,
|
||||
"age": out["resolved_ts"] is not None, "aria": f"{shown} live connections"}
|
||||
|
||||
|
||||
def rendered_leaf(row: dict[str, Any]) -> dict[str, str]:
|
||||
"""The two lines and the two long strings one leaf row prints, derived as a **second** reading.
|
||||
|
||||
@@ -7082,6 +7450,10 @@ def rendered_leaf(row: dict[str, Any]) -> dict[str, str]:
|
||||
`leafAria` compose those same two from the literals asserted verbatim below. A rename on either
|
||||
side therefore fails a pin **and** changes what this mirror prints, which is the pair of
|
||||
guarantees a page with no test runner can actually make.
|
||||
|
||||
The row's `aria-label` ends with the count's claim, and phase 32 task 04 made that claim carry its
|
||||
cause: `count_surfaces()` is the one place that sentence is assembled, so the row and the count
|
||||
element cannot disagree about which unknown a dash is.
|
||||
"""
|
||||
out = row["out"]
|
||||
host = out["host"] or "?"
|
||||
@@ -7099,8 +7471,7 @@ def rendered_leaf(row: dict[str, Any]) -> dict[str, str]:
|
||||
if row["shared"] is True:
|
||||
bits.append("this endpoint is shared with another route")
|
||||
bits.append("in the config")
|
||||
bits.append("connections not attributed" if row["connections"] is None
|
||||
else f"{row['connections']} live connections")
|
||||
bits.append(count_surfaces(row)["aria"])
|
||||
return {"line1": inbound,
|
||||
"line2": f"→ {endpoint}" + (f" · {kind}" if kind else ""),
|
||||
"title": pair + shared,
|
||||
@@ -7440,7 +7811,7 @@ def test_the_stream_map_is_rendered_inside_nginx_and_nowhere_else(
|
||||
|
||||
Pinned from both ends: the script ships exactly the two panel definitions and never indexes
|
||||
`nginx.map_targets`, and the live document's map connections are all nginx's, all on one port,
|
||||
and exactly as many as the map has routing entries (17 on this capture — 16 `*.reeselink.com`
|
||||
and exactly as many as the map has routing entries (18 on this capture — 17 `*.reeselink.com`
|
||||
plus the `default` the file writes for SNI nobody mapped).
|
||||
"""
|
||||
html = _code(client.get(GATEWAY_PAGE).text)
|
||||
@@ -7570,22 +7941,31 @@ def test_warnings_render_in_the_overviews_alert_vocabulary_and_hide_when_clean(
|
||||
|
||||
|
||||
def test_an_unattributed_endpoint_prints_a_dash_and_never_a_zero(client, gateway_client) -> None:
|
||||
"""A leaf the join cannot count prints `—`, and the tooltip says why in one sentence.
|
||||
"""A leaf the join cannot count prints `—`, and the tooltip says **which** unknown it is.
|
||||
|
||||
Sixteen of this fleet's 17 map entries name a **domain**, and `/proc` reports addresses, so no
|
||||
count exists for them and none can ever exist (nothing here resolves a name). `0` would be a
|
||||
claim about traffic (rule 2) and a column of `n/a` would be twelve identical words competing with
|
||||
the page's one footnote — so the leaf prints an em dash whose `title` names the reason, while the
|
||||
trunk keeps `n/a`, which is a different fact: the socket table itself went quiet.
|
||||
`/proc` reports addresses, so a route whose far end has no address cannot be counted, and `0` would
|
||||
be a claim about traffic (rule 2) while a column of `n/a` would be twelve identical words
|
||||
competing with the page's one footnote — so the leaf prints an em dash whose `title` names the
|
||||
reason, while the trunk keeps `n/a`, which is a different fact: the socket table itself went quiet.
|
||||
Phase 32 task 04 sharpened the mechanism and not only the wording: the dash now has **two** causes
|
||||
that `countCause` separates from `out.address` / `out.resolved` — a name the gateway could not
|
||||
resolve, and an address that is known beside a table that answered nothing — and the sentence is
|
||||
chosen from data, never guessed. A `0` is neither of them: it is a measurement.
|
||||
"""
|
||||
js = _code(client.get("/static/page-gateway.js").text)
|
||||
head = js[:js.index("function countText(")]
|
||||
assert "const NO_COUNT = '—'" in head, "the unattributed leaf count is no longer an em dash"
|
||||
assert "nothing here resolves names" in head, "the dash stopped explaining itself"
|
||||
assert "the gateway could not resolve this name in this reading" in head, \
|
||||
"the dash stopped explaining itself"
|
||||
assert "function countCause(" in head and "out.address === true" in head \
|
||||
and "out.resolved" in head, \
|
||||
"the dash's cause is no longer read from the two payload fields that decide it"
|
||||
assert "nothing here resolves names" not in head, (
|
||||
"the old sentence blamed the name, which phase 32 task 03 made a false reason")
|
||||
leaf = js[js.index("function renderLeaf("):js.index("function groupElement(")]
|
||||
assert "shown == null ? NO_COUNT : shown" in leaf, "a null count is not the dash any more"
|
||||
assert "count.title = shown == null ? NO_COUNT_WHY" in leaf, \
|
||||
"the dash lost the sentence that separates it from a zero"
|
||||
assert "count.title = shown == null ? countWhy(conn) : countTitle(conn, shown)" in leaf, \
|
||||
"the dash lost its sentence, or a counted line lost the provenance beside its number"
|
||||
trunk = js[js.index("function renderTrunk("):js.index("function renderLeaves(")]
|
||||
assert "live" in trunk and "n/a" not in trunk, (
|
||||
"the trunk now prints the leaf's dash too — a quiet socket table is a different fact, and "
|
||||
@@ -7600,6 +7980,164 @@ def test_an_unattributed_endpoint_prints_a_dash_and_never_a_zero(client, gateway
|
||||
"table's arm, not the name arm, and it prints a different sentence")
|
||||
|
||||
|
||||
def test_an_unresolved_name_and_an_idle_route_say_two_different_sentences(
|
||||
client, gateway_snapshot: Any) -> None:
|
||||
"""The em dash and the measured `0` are two facts, and every surface of the row says so.
|
||||
|
||||
Phase 32 task 03 made a name-written route countable, which left the page's one dash sentence
|
||||
explaining two different things: a name the gateway could not resolve, and a route that **was**
|
||||
resolved and simply has nothing open. Task 04 splits them, and the split has to be on every
|
||||
surface a reader can meet — the glyph, the `title`, the `aria-label` — because a `0` read as a
|
||||
missing measurement (or a dash read as an idle route) sends the operator to the wrong box.
|
||||
|
||||
Three arms, all from the recorded capture and one injection, all rendered through
|
||||
`count_surfaces` (the mirror) with the served-byte pins below as its other half:
|
||||
|
||||
* **resolved and idle** — `brain.reeseapps.com` reaches
|
||||
`reeseapps-brain-of-reese.reeselink.com:443`, the gateway resolved it to `10.1.2.64`, and the
|
||||
kernel reports no socket: the glyph is `0`, and a `0` is a measurement.
|
||||
* **resolved and busy** — the ntfy route, whose glyph is its own number.
|
||||
* **not resolved** — one entry of the recorded document emptied and marked `nxdomain`, exactly
|
||||
what agent 1.3.0 publishes for a name with no record: the glyph is the em dash and the sentence
|
||||
is the resolution one, never the socket one.
|
||||
|
||||
And a fourth, because the third arm's sentence must not become a catch-all: with the kernel's
|
||||
table unreadable, a line whose address the **config wrote** is uncounted too, and blaming
|
||||
resolution there would be false — so it gets the socket-table sentence.
|
||||
"""
|
||||
js = _code(client.get("/static/page-gateway.js").text)
|
||||
head = js[: js.index("function countText(")]
|
||||
|
||||
# (1) the two sentences exist, differ, name no `n/a`, and neither is the other's wording.
|
||||
assert "const WHY_UNRESOLVED = 'the gateway could not resolve this name in this reading'" in head
|
||||
assert "const WHY_NO_SOCKETS = 'the socket table did not answer this reading'" in head
|
||||
assert COUNT_WHY_UNRESOLVED != COUNT_WHY_NO_SOCKETS
|
||||
for sentence in (COUNT_WHY_UNRESOLVED, COUNT_WHY_NO_SOCKETS):
|
||||
assert "n/a" not in sentence, "an unknown that has a cause stopped naming its cause"
|
||||
# The choice is data: the same two fields `sources/gateway.py` joins on, and nothing else.
|
||||
chooser = js[js.index("function countCause("):js.index("function countWhy(")]
|
||||
assert "out.address === true" in chooser and "out.resolved" in chooser, \
|
||||
"the dash's cause is no longer chosen from the address the config wrote or the gateway found"
|
||||
assert "countCause(conn)" in js[js.index("function leafAria("):js.index("function leafTitle(")], \
|
||||
"the `aria-label` no longer carries the cause the tooltip carries"
|
||||
|
||||
# (2) the three arms, rendered.
|
||||
resolved = gateway_source.parse(gateway_snapshot("resolved"), "gateway")
|
||||
|
||||
def route(host: str, port: int) -> dict[str, Any]:
|
||||
return next(row for row in resolved["connections"]
|
||||
if row["out"]["host"] == host and row["in"]["port"] == port)
|
||||
|
||||
idle = count_surfaces(route("reeseapps-brain-of-reese.reeselink.com", 443))
|
||||
busy = count_surfaces(route("reeseapps-ntfy.reeselink.com", 443))
|
||||
assert idle["glyph"] == "0", "a resolved endpoint with nothing open is a measurement, not a gap"
|
||||
assert busy["glyph"] == "3" and busy["why"] is None
|
||||
assert idle["title"].startswith("0 open connection(s) on this endpoint") \
|
||||
and busy["title"].startswith("3 open connection(s) on this endpoint")
|
||||
|
||||
broken = gateway_snapshot("resolved")
|
||||
for entry in broken["resolver"]["entries"]:
|
||||
if entry["name"] == "reeseapps-immich.reeselink.com":
|
||||
entry["addresses"] = []
|
||||
entry["status"] = "nxdomain"
|
||||
unplaced = gateway_source.parse(broken, "gateway")
|
||||
dash = count_surfaces(next(row for row in unplaced["connections"]
|
||||
if row["out"]["host"] == "reeseapps-immich.reeselink.com"))
|
||||
assert dash["glyph"] == "\u2014", "a name with no record grew someone else's number"
|
||||
assert dash["why"] == COUNT_WHY_UNRESOLVED and dash["title"] == f"no number: {dash['why']}"
|
||||
assert dash["aria"] == f"connections not attributed: {COUNT_WHY_UNRESOLVED}"
|
||||
|
||||
# (3) the fourth arm: an address the config wrote, beside a kernel that said nothing.
|
||||
quiet = gateway_source.parse(gateway_snapshot("sockets_silent"), "gateway")
|
||||
unwritten = count_surfaces(next(row for row in quiet["connections"]
|
||||
if row["out"]["address"] is True))
|
||||
assert unwritten["glyph"] == "\u2014" and unwritten["why"] == COUNT_WHY_NO_SOCKETS, (
|
||||
"a line whose address the config wrote was told to go resolve something — the socket table "
|
||||
"is what went quiet, and that is the sentence the trunk already prints as `n/a`")
|
||||
|
||||
# (4) five lines read aloud: distinct claims, no invented number, no `null` leaking as text.
|
||||
spoken = [count_surfaces(row) for row in resolved["connections"][:5]]
|
||||
claims = {entry["title"] for entry in spoken} | {entry["aria"] for entry in spoken}
|
||||
assert len(claims) > 1, "five lines said one thing"
|
||||
for entry in [*spoken, idle, busy, dash, unwritten]:
|
||||
for text in (entry["glyph"], entry["title"] or "", entry["aria"]):
|
||||
assert "undefined" not in text and "NaN" not in text and "null" not in text, \
|
||||
f"a row printed {text!r}"
|
||||
|
||||
|
||||
def test_a_counted_line_names_where_its_address_came_from(client, gateway_snapshot: Any,
|
||||
resolved_gateway_client) -> None:
|
||||
"""Every counted line says, on hover, which address it was counted to and when that was learned.
|
||||
|
||||
A number that arrived through the gateway's own `libc` resolution is a different claim from one
|
||||
the config wrote as an address, and the tooltip is the one place both can say it without growing
|
||||
a column of ink per row. Three arms, one rule: the provenance is the payload's (`out.via`,
|
||||
`out.resolved_ts`) copied, never paraphrased, and a field that is not there prints **nothing**
|
||||
rather than `n/a` or `NaN`.
|
||||
|
||||
* a **resolved name** — `… · address from libc`, and `, resolved <age>` while the entry has a
|
||||
timestamp (the age is `Dash.duration(Dash.ageSeconds(…))`, so its text is a clock and only its
|
||||
presence is pinned here);
|
||||
* the same row with `resolved_ts` removed — the age fragment goes, the rest of the sentence
|
||||
stays, and nothing prints the shared `n/a` where an age belongs;
|
||||
* an **address the config wrote** — the bare sentence. It was never resolved, and stamping the
|
||||
agent's `literal` echo of it as a lookup would be rule 2 with a tooltip's clothes on.
|
||||
"""
|
||||
js = _code(client.get("/static/page-gateway.js").text)
|
||||
builder = js[js.index("function countTitle("):js.index("const NO_TARGET")]
|
||||
for field in ("out.via", "out.resolved", "out.resolved_ts"):
|
||||
assert field in builder, f"the counted line's tooltip stopped reading {field}"
|
||||
for helper in ("duration(", "ageSeconds("):
|
||||
assert helper in builder, f"the age stopped going through the shared helper {helper}"
|
||||
assert "out.resolved_ts == null" in builder, \
|
||||
"a missing timestamp now prints whatever duration() makes of Infinity"
|
||||
assert "address from" in builder and "resolved " in builder
|
||||
assert "n/a" not in builder and "\u2014" not in builder, \
|
||||
"the counted tooltip grew a fallback glyph; a missing fragment is silence, not a mark"
|
||||
|
||||
resolved = gateway_source.parse(gateway_snapshot("resolved"), "gateway")
|
||||
brain = next(row for row in resolved["connections"]
|
||||
if row["out"]["host"] == "reeseapps-brain-of-reese.reeselink.com")
|
||||
printed = count_surfaces(brain)
|
||||
assert printed["title"] == "0 open connection(s) on this endpoint \u00b7 address from libc"
|
||||
assert printed["age"] is True
|
||||
served = client.get("/static/page-gateway.js").text
|
||||
assert "\u00b7 address from ${via}" in served and ", resolved ${age}`" in served, \
|
||||
"the two provenance fragments are no longer composed from the payload's own words"
|
||||
|
||||
aged = dict(brain, out=dict(brain["out"], resolved_ts=None))
|
||||
lost = count_surfaces(aged)
|
||||
assert lost["title"] == printed["title"], "losing the timestamp lost the address it came from"
|
||||
assert lost["age"] is False, "an age was printed for a row that publishes no timestamp"
|
||||
assert "NaN" not in lost["title"] and "n/a" not in lost["title"]
|
||||
|
||||
written = next(row for row in resolved["connections"] if row["out"]["address"] is True)
|
||||
plain = count_surfaces(written)
|
||||
assert plain["title"] == f"{written['connections']} open connection(s) on this endpoint", \
|
||||
"an address the config wrote was given a resolution provenance it never had"
|
||||
assert "address from" not in plain["title"]
|
||||
|
||||
# …and the field table, in **both** directions (the `SLOT_FIELDS` device): the three names the
|
||||
# tooltip and the filter index are named there, and the 1.3.0 document answers each with real
|
||||
# data, so a rename in `sources/gateway.py` fails by name here instead of printing a tooltip that
|
||||
# quietly stopped carrying its provenance.
|
||||
provenance = ("connections.out.resolved", "connections.out.via",
|
||||
"connections.out.resolved_ts")
|
||||
live = resolved_gateway_client.get("/api/gateway/snapshot").json()
|
||||
for path in provenance:
|
||||
assert path in GATEWAY_PAGE_FIELDS, (
|
||||
f"the counted line reads {path.removeprefix('connections.')}, which the page's field "
|
||||
"table does not name")
|
||||
found, values = _payload_field(live, path)
|
||||
assert found and any(value is not None for value in values), (
|
||||
f"{path} is named but the resolved document never answers it, so the pin above would "
|
||||
"pass over a payload that carries nothing")
|
||||
assert any(value is None for path in provenance
|
||||
for value in _payload_field(live, path)[1]), (
|
||||
"every row published provenance, including the addresses the config wrote — the arm that "
|
||||
"must print nothing is no longer distinguishable")
|
||||
|
||||
|
||||
def test_the_docs_name_the_page_its_knob_and_its_gauges() -> None:
|
||||
"""The phase's own completion criterion, pinned so it cannot rot quietly.
|
||||
|
||||
|
||||
@@ -261,6 +261,11 @@ async def test_a_host_without_a_proxy_config_does_not_degrade_the_system_source(
|
||||
`collectors.ADDON_ERROR_PREFIXES` is the one list of these prefixes; this asserts every
|
||||
name by behaviour, so dropping any of them fails here, and `tests/test_agent.py` pins both
|
||||
the wording of the lines the agent actually writes and the equality of the two lists.
|
||||
|
||||
The `resolver:` line is phase 32's and the same case with a DNS-shaped name: a host with no
|
||||
proxy config has nothing to resolve, and a truncated lookup pass (`resolver: pass budget …`)
|
||||
is one route without a count, not a broken box. Both arms belong in this list or every
|
||||
upgraded host gets an overview WARN it has no way to clear.
|
||||
"""
|
||||
missing_haproxy = ("haproxy: /etc/haproxy/haproxy.cfg not found — haproxy is not installed "
|
||||
"here, or its config lives elsewhere (--haproxy-config); --no-proxy "
|
||||
@@ -269,9 +274,12 @@ async def test_a_host_without_a_proxy_config_does_not_degrade_the_system_source(
|
||||
"its config lives elsewhere (--nginx-config); --no-proxy opts out")
|
||||
no_sockets = ("sockets: /proc/net/tcp not found, /proc/net/tcp6 not found — this host "
|
||||
"publishes no kernel socket table; --no-proxy opts out")
|
||||
nothing_to_resolve = ("resolver: nothing to resolve — haproxy not found and nginx not found; "
|
||||
"--no-resolve opts out")
|
||||
truncated_lookups = ("resolver: pass budget 3.0s exhausted, 41 of 58 targets looked up")
|
||||
body = agent_body()
|
||||
body["errors"] = [missing_haproxy, missing_nginx, no_sockets,
|
||||
"smart: smartctl 7.5 not found"]
|
||||
"smart: smartctl 7.5 not found", nothing_to_resolve, truncated_lookups]
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(200, json=body)
|
||||
@@ -280,7 +288,7 @@ async def test_a_host_without_a_proxy_config_does_not_degrade_the_system_source(
|
||||
result = await fetch_host_snapshot(client, Target("gw", "http://agent.test"),
|
||||
HostCollector(RateEstimator()))
|
||||
assert "degraded" not in result.payload, "unconfigured is not broken (invariant 3)"
|
||||
for prefix in ("haproxy:", "nginx:", "sockets:"):
|
||||
for prefix in ("haproxy:", "nginx:", "sockets:", "resolver:"):
|
||||
assert any(prefix in str(e) for e in result.payload["agent"]["errors"]), \
|
||||
"the sentence survives for the page that explains it"
|
||||
|
||||
|
||||
@@ -27,6 +27,7 @@ from __future__ import annotations
|
||||
|
||||
import difflib
|
||||
import importlib.util
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
@@ -2391,6 +2392,23 @@ case "$command" in
|
||||
[ -f "$file" ] || { echo "cat: $path: No such file or directory" >&2; exit 1; }
|
||||
cat "$file"
|
||||
;;
|
||||
"getent "*)
|
||||
# libc resolution on the fake box. Answers come from `$FAKE_REMOTE/getent/<name>.<family>`;
|
||||
# a file that is not there is `getent`'s own "no such record" (exit 2, no output), which the
|
||||
# script records as a status rather than a failure. Two env switches stand in for the two
|
||||
# reads that are *not* a fact about the name: `GETENT_MODE=transport` is ssh dying, and
|
||||
# `GETENT_MODE=missing` is a gateway with no getent to ask.
|
||||
family="${command#getent }"
|
||||
family="${family%% *}"
|
||||
name="${command##* }"
|
||||
case "${GETENT_MODE:-ok}" in
|
||||
transport) echo "ssh: connect to host $host port 22: Connection refused" >&2; exit 255 ;;
|
||||
missing) echo "bash: line 1: getent: command not found" >&2; exit 127 ;;
|
||||
esac
|
||||
file="$FAKE_REMOTE/getent/$name.${family#ahosts}"
|
||||
[ -f "$file" ] || exit 2
|
||||
cat "$file"
|
||||
;;
|
||||
"ls "*)
|
||||
spec="${command#ls }"
|
||||
dir="${spec%/*}"
|
||||
@@ -2444,7 +2462,10 @@ CAPTURE_FAKE_GATEWAY: dict[str, str] = {
|
||||
"map $ssl_preread_server_name $targetBackend {\n"
|
||||
" default 127.0.0.1:4443;\n"
|
||||
" brain.example reeseapps-brain-of-reese.reeselink.com:443;\n}\n"
|
||||
"server {\n listen 443;\n proxy_pass $targetBackend;\n}\n"
|
||||
# `resolver` here, inside the stream server, is where this fleet writes it: the fixture's
|
||||
# `config_directive` must come out of an *included* document, read from the bytes the run
|
||||
# already captured, with no round trip of its own.
|
||||
"server {\n listen 443;\n proxy_pass $targetBackend;\n resolver 10.9.8.7;\n}\n"
|
||||
),
|
||||
# 8.8.8.8 and 2603:6013::1 are globally routable, `::ffff:8.8.8.8` is a visitor behind a v6
|
||||
# socket, and 203.0.113.7 is already documentation space. The tx/rx queue pair on the last row
|
||||
@@ -2470,12 +2491,40 @@ CAPTURE_FAKE_GATEWAY: dict[str, str] = {
|
||||
}
|
||||
|
||||
|
||||
#: What the fake gateway's libc answers for the names its configs declare. The three arms are the
|
||||
#: three the capture has to tell apart: an answer (private, so unscrubbed — this is what the real
|
||||
#: fleet looks like), an answer that is **globally routable** (so it must become documentation
|
||||
#: space), and no answer at all (a status, not a failure). `nextcloud` deliberately answers a v6
|
||||
#: that is `::ffff:`-mapped, because `getent ahostsv6` synthesises that spelling for a name with no
|
||||
#: AAAA record and the agent's own `getaddrinfo` never publishes it.
|
||||
CAPTURE_FAKE_ANSWERS: dict[str, dict[str, str]] = {
|
||||
# Private and answered twice over — the A record, and the `::ffff:` spelling `getent ahostsv6`
|
||||
# synthesises for a name with no AAAA record. One address goes in; nothing moves.
|
||||
"gitea.reeselink.com": {
|
||||
"v4": "10.1.0.137 STREAM gitea.reeselink.com\n10.1.0.137 DGRAM \n",
|
||||
"v6": "::ffff:10.1.0.137 STREAM \n",
|
||||
},
|
||||
# Two globally routable answers (one per family) beside one already in documentation space: the
|
||||
# first two become `203.0.113.1` / `2001:db8::1` and the third stays exactly as it was, which is
|
||||
# what makes a second run a no-op instead of a renumbering.
|
||||
"nextcloud.reeselink.com": {
|
||||
"v4": ("8.8.8.8 STREAM nextcloud.reeselink.com\n"
|
||||
"203.0.113.9 STREAM nextcloud.reeselink.com\n"),
|
||||
"v6": "2603:6013:3100:101:5054:ff:fe84:b2e0 STREAM nextcloud.reeselink.com\n",
|
||||
},
|
||||
# `reeseapps-brain-of-reese.reeselink.com` is answered by nothing on purpose: the absence of
|
||||
# the file *is* the nxdomain arm, a status among successes rather than a failed run.
|
||||
}
|
||||
|
||||
|
||||
def _capture_sandbox(
|
||||
tmp_path: Path,
|
||||
gateway: dict[str, str] | None = None,
|
||||
*,
|
||||
recorded: dict[str, str] | None = None,
|
||||
fail_ssh: bool = False,
|
||||
fail_getent: str | None = None,
|
||||
answers: dict[str, dict[str, str]] | None = None,
|
||||
alias: str | None = None,
|
||||
) -> tuple[Path, Path, Path, dict[str, str]]:
|
||||
"""A copy of the capture script, a fake gateway, and a PATH whose `ssh` reaches only the fake.
|
||||
@@ -2483,16 +2532,28 @@ def _capture_sandbox(
|
||||
Returns (script, fixtures dir, stub call log, env). The script is copied into
|
||||
`<tmp>/scripts/` because it derives `tests/fixtures/` from its own location, which is what
|
||||
keeps the repository's fixtures out of reach of a test.
|
||||
|
||||
The parser step needs the repo, so `agent/host_agent.py` is copied beside the copied script:
|
||||
the sandbox is a checkout shape, and the plan step imports `target_names` from it exactly as
|
||||
it does when the real script runs. That copy is the pin — a script that re-implemented the
|
||||
parser would not need it, and one that imported a different module would not find it.
|
||||
"""
|
||||
script = tmp_path / "scripts" / CAPTURE.name
|
||||
script.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copyfile(CAPTURE, script)
|
||||
(tmp_path / "agent").mkdir(parents=True, exist_ok=True)
|
||||
shutil.copyfile(ROOT / "agent/host_agent.py", tmp_path / "agent/host_agent.py")
|
||||
|
||||
fake = tmp_path / "gateway"
|
||||
for path, body in {**CAPTURE_FAKE_GATEWAY, **(gateway or {})}.items():
|
||||
target = fake / path.lstrip("/")
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(body, encoding="utf-8")
|
||||
for name, families in (CAPTURE_FAKE_ANSWERS if answers is None else answers).items():
|
||||
for family, body in families.items():
|
||||
answer = fake / "getent" / f"{name}.{family}"
|
||||
answer.parent.mkdir(parents=True, exist_ok=True)
|
||||
answer.write_text(body, encoding="utf-8")
|
||||
|
||||
fixtures = tmp_path / "tests" / "fixtures"
|
||||
fixtures.mkdir(parents=True, exist_ok=True)
|
||||
@@ -2511,7 +2572,11 @@ def _capture_sandbox(
|
||||
"PATH": f"{stubs}{os.pathsep}{os.environ['PATH']}",
|
||||
"STUB_LOG": str(call_log),
|
||||
"FAKE_REMOTE": str(fake),
|
||||
"FAIL_SSH": "1" if fail_ssh else "0"}
|
||||
"FAIL_SSH": "1" if fail_ssh else "0",
|
||||
"GETENT_MODE": fail_getent or "ok",
|
||||
# The script prepends its own root to this, which is how `agent.host_agent` resolves
|
||||
# when uv has no project to run in (this sandbox is not one).
|
||||
"PYTHONPATH": str(tmp_path)}
|
||||
if alias is not None:
|
||||
env["GATEWAY_SSH"] = alias
|
||||
return script, fixtures, call_log, env
|
||||
@@ -2529,12 +2594,23 @@ def _fixture_body(path: Path) -> str:
|
||||
|
||||
|
||||
def test_the_gateway_capture_exists_and_asks_for_no_privilege() -> None:
|
||||
"""Six world-readable reads over an ssh key: nothing here needs root or a rewrite tool."""
|
||||
"""Six world-readable reads plus `getent`, over an ssh key: no root, no rewrite tool.
|
||||
|
||||
`getent` belongs in this list because it is the seventh read and the only one that is not a
|
||||
file: it is libc resolution on the box that dials, which is what `via: "libc"` claims, and it
|
||||
is world-runnable for the same reason the six files are world-readable.
|
||||
"""
|
||||
assert CAPTURE.is_file(), "scripts/capture-gateway-config.sh is missing (phase 28 task 04)"
|
||||
assert os.access(CAPTURE, os.X_OK)
|
||||
text = CAPTURE.read_text()
|
||||
for path in CAPTURE_TARGETS:
|
||||
assert path in text, f"{path} is no longer captured"
|
||||
for family in ("getent ahostsv4", "getent ahostsv6"):
|
||||
assert family in text, f"{family} is no longer how the resolver document is recorded"
|
||||
for tool in ("dig", "nslookup", "resolvectl", "drill"):
|
||||
assert not re.search(rf"(?<![\w-]){tool}\s+[A-Za-z0-9.*_-]", text), (
|
||||
f"{tool} assumes a DNS client the gateway may not ship; getent is libc, which is the "
|
||||
"vantage the agent's own lookup performs")
|
||||
code = "\n".join(line for line in text.splitlines() if not line.lstrip().startswith("#"))
|
||||
assert "sudo" not in code, (
|
||||
"the capture must read exactly what dashboards-agent can read; a pass that needed root "
|
||||
@@ -2586,7 +2662,9 @@ def test_the_gateway_capture_records_every_document_of_the_tree_it_read(tmp_path
|
||||
expected = {"gateway_haproxy.cfg", "gateway_nginx_main.conf",
|
||||
"gateway_nginx_conf_d_http.conf", "gateway_nginx_conf_d_tls.conf",
|
||||
"gateway_nginx_stream_d_stream.conf",
|
||||
"gateway_proc_net_tcp.txt", "gateway_proc_net_tcp6.txt"}
|
||||
"gateway_proc_net_tcp.txt", "gateway_proc_net_tcp6.txt",
|
||||
# The resolver document, derived from the configs in this same set.
|
||||
"gateway_resolver.json"}
|
||||
assert {p.name for p in fixtures.iterdir()} == expected, result.stdout
|
||||
assert _fixture_body(fixtures / "gateway_haproxy.cfg") == \
|
||||
CAPTURE_FAKE_GATEWAY["/etc/haproxy/haproxy.cfg"]
|
||||
@@ -2703,6 +2781,189 @@ def test_the_socket_capture_publishes_no_globally_routable_endpoint(tmp_path: Pa
|
||||
assert len(v6[2]) == len(raw_v6[2])
|
||||
|
||||
|
||||
# --- the resolver document (phase 32 task 02) ------------------------------------------------
|
||||
|
||||
#: One entry per declared target of the fake gateway, in the plan's own order: haproxy's `server`
|
||||
#: line first, then the nginx vhosts' `proxy_pass` targets, then the stream `map`'s entries — with
|
||||
#: the map's `default` an address literal, which is why it costs no read on the remote.
|
||||
CAPTURE_RESOLVER_TARGETS: tuple[tuple[str, int | None], ...] = (
|
||||
("gitea.reeselink.com", 2222),
|
||||
("nextcloud.reeselink.com", 11000),
|
||||
("reeseapps-brain-of-reese.reeselink.com", 443),
|
||||
("127.0.0.1", 4443),
|
||||
)
|
||||
|
||||
RESOLVER_KEEP = {
|
||||
"gateway_resolver.json": '{"available": true, "entries": ["keep me"]}\n',
|
||||
}
|
||||
|
||||
def _resolver_fixture(path: Path) -> dict[str, Any]:
|
||||
"""The recorded document, parsed — and a clear failure when the run refused to write it."""
|
||||
assert path.is_file(), f"{path.name} was not written"
|
||||
return json.loads(path.read_text())
|
||||
|
||||
|
||||
def _resolver_run(tmp_path: Path, **kwargs: Any) -> tuple[subprocess.CompletedProcess[str], Path]:
|
||||
"""One sandboxed capture run, returning its result and the fixture directory it wrote into."""
|
||||
script, fixtures, _log, env = _capture_sandbox(tmp_path, **kwargs)
|
||||
result = subprocess.run([BASH, str(script)], capture_output=True, text=True, env=env,
|
||||
check=False)
|
||||
return result, fixtures
|
||||
|
||||
|
||||
#: The config documents of the captured tree, the same mapping `tests/conftest.py` lays down for
|
||||
#: `sources/gateway.py`: an agent `--root` under which the agent can read this repo's captures.
|
||||
def test_the_capture_asks_its_own_parser_and_not_its_memory() -> None:
|
||||
"""The name list is `target_names()`, imported — never a second list somebody maintained.
|
||||
|
||||
This is the phase's rule 3 in the capture's own hands: the fixture can only ever name what the
|
||||
agent will resolve, so `target_names()` changing moves the fixture with it instead of leaving a
|
||||
document of names the agent stopped asking about.
|
||||
"""
|
||||
text = CAPTURE.read_text()
|
||||
assert "from agent.host_agent import" in text, "the plan does not import the agent's parser"
|
||||
assert "target_names(" in text, "the plan does not ask the agent what its configs declare"
|
||||
assert "Collector(" in text, "the plan does not read the captured configs the agent reads them"
|
||||
# And no domain is written into the script: the only domains in it are comments and the fake
|
||||
# gateway's own fixtures live in the test, not here.
|
||||
code = "\n".join(line for line in text.splitlines() if not line.lstrip().startswith("#"))
|
||||
assert ".reeselink.com" not in code and ".reeseapps.com" not in code, (
|
||||
"a target name is hardcoded in the capture instead of derived from the configs")
|
||||
|
||||
|
||||
def test_the_resolver_document_names_every_target_the_configs_declare(tmp_path: Path) -> None:
|
||||
"""The happy path: one libc read per declared name, in the agent's order, in its own shape.
|
||||
|
||||
Four assertions the fixture's whole usefulness rests on: the entry set is the plan, `via` says
|
||||
how the addresses were obtained, `config_directive` came out of the **captured** nginx text (so
|
||||
it matches what the agent will read from the same bytes, with no extra round trip), and the
|
||||
remote was asked exactly twice per name — and never about the literal the map writes.
|
||||
"""
|
||||
result, fixtures = _resolver_run(tmp_path)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
document = _resolver_fixture(fixtures / "gateway_resolver.json")
|
||||
|
||||
assert document["available"] is True
|
||||
assert document["via"] == "libc"
|
||||
assert document["config_directive"] == "10.9.8.7", (
|
||||
"the `resolver` directive did not come out of the included stream config")
|
||||
assert [(row["name"], row["port"]) for row in document["entries"]] == list(
|
||||
CAPTURE_RESOLVER_TARGETS), "the entries are the plan, in the agent's order"
|
||||
assert document["errors"] == []
|
||||
for row in document["entries"]:
|
||||
assert set(row) == {"name", "port", "status", "addresses", "ts"}
|
||||
assert row["ts"] == document["ts"], "a fixed capture ts keeps the fixture byte-stable"
|
||||
|
||||
script, _fixtures, log, _env = _capture_sandbox(tmp_path / "ledger")
|
||||
subprocess.run([BASH, str(script)], capture_output=True, text=True, env=_env, check=False)
|
||||
asked = [line for line in log.read_text().splitlines() if "getent " in line]
|
||||
for name, _port in CAPTURE_RESOLVER_TARGETS:
|
||||
if ipaddress.ip_address(name) if name[0].isdigit() else False:
|
||||
assert not [line for line in asked if name in line], (
|
||||
"an address literal cost a lookup on the remote")
|
||||
continue
|
||||
families = [line for line in asked if "getent ahostsv" in line and line.endswith(name)]
|
||||
assert len(families) == 2, f"{name} was not read once per family: {families}"
|
||||
|
||||
|
||||
def test_the_resolver_document_scrubs_global_addresses_and_keeps_everything_else(
|
||||
tmp_path: Path) -> None:
|
||||
"""What the document edits, and what it must not, in one run.
|
||||
|
||||
Three arms at once, because they are one rule: `8.8.8.8` and a global EUI-64 become
|
||||
documentation space (invariant 6 — a resolved address is as much an identifier as the socket
|
||||
table's), `10.1.0.137` and an address already in documentation space stay exactly as they were
|
||||
(which is what makes a re-run a no-op), and the `::ffff:` spelling `getent ahostsv6` synthesises
|
||||
for a name with no AAAA record is dropped — the agent's one `getaddrinfo` never publishes it, so
|
||||
recording it would claim an address the agent does not. A name nobody answers is `nxdomain`, the
|
||||
status the page splits out from "resolved and idle".
|
||||
"""
|
||||
result, fixtures = _resolver_run(tmp_path)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
entries = {row["name"]: row for row in
|
||||
_resolver_fixture(fixtures / "gateway_resolver.json")["entries"]}
|
||||
|
||||
private = entries["gitea.reeselink.com"]
|
||||
assert private["status"] == "ok"
|
||||
assert private["addresses"] == ["10.1.0.137"], private
|
||||
assert not any(a.startswith("::ffff:") for a in private["addresses"]), (
|
||||
"the synthesised v4-mapped spelling survived: the agent never publishes it")
|
||||
|
||||
moved = entries["nextcloud.reeselink.com"]
|
||||
assert moved["status"] == "ok"
|
||||
assert moved["addresses"] == ["203.0.113.1", "203.0.113.9", "2001:db8::1"], moved
|
||||
assert "8.8.8.8" not in json.dumps(moved), "a globally routable answer is in a tracked file"
|
||||
assert "2603:6013" not in json.dumps(moved), "an EUI-64 address is in a tracked file"
|
||||
|
||||
missing = entries["reeseapps-brain-of-reese.reeselink.com"]
|
||||
assert missing["status"] == "nxdomain", "one name failing among successes is a fact, kept"
|
||||
assert missing["addresses"] == [], "an unresolved name publishes no address, not 0 and not a guess"
|
||||
|
||||
literal = entries["127.0.0.1"]
|
||||
assert literal["status"] == "literal" and literal["addresses"] == ["127.0.0.1"]
|
||||
|
||||
|
||||
def test_the_resolver_document_is_idempotent_so_a_re_run_does_not_churn(tmp_path: Path) -> None:
|
||||
"""Two runs over the same answers write the same bytes.
|
||||
|
||||
The `ts` is a constant and the scrub is first-seen order over a deterministic plan, so a
|
||||
re-capture of a gateway nobody changed is a no-op — the property that makes `git diff` on this
|
||||
directory a statement about the ingress rather than about the clock.
|
||||
"""
|
||||
once, first = _resolver_run(tmp_path)
|
||||
assert once.returncode == 0, once.stdout + once.stderr
|
||||
again, second = _resolver_run(tmp_path / "again")
|
||||
assert again.returncode == 0, again.stdout + again.stderr
|
||||
assert (second / "gateway_resolver.json").read_bytes() == (
|
||||
first / "gateway_resolver.json").read_bytes()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("mode, needle", [
|
||||
pytest.param("transport", "did not answer", id="ssh-dies"),
|
||||
pytest.param("missing", "getent is not available", id="no-getent"),
|
||||
])
|
||||
def test_a_broken_resolver_vantage_is_not_a_fixture(tmp_path: Path, mode: str,
|
||||
needle: str) -> None:
|
||||
"""The configs capture fine and only the resolver is broken — and it still refuses.
|
||||
|
||||
This is the arm the socket rule cannot cover: `getent` answering *nothing* for every name is a
|
||||
fact about the box's resolver, not about the names, and a document of `nxdomain` rows would
|
||||
make every live route on the page a permanent dash and every join test a test of nothing.
|
||||
"""
|
||||
result, fixtures = _resolver_run(tmp_path, recorded=dict(RESOLVER_KEEP), fail_getent=mode)
|
||||
assert result.returncode == 1, result.stdout
|
||||
assert needle in result.stdout, result.stdout
|
||||
assert "gateway_resolver.json" in result.stdout
|
||||
assert (fixtures / "gateway_resolver.json").read_text() == RESOLVER_KEEP[
|
||||
"gateway_resolver.json"], "a broken vantage overwrote the recorded document"
|
||||
|
||||
|
||||
def test_not_one_name_answering_is_refused_the_way_a_dead_transport_is(tmp_path: Path) -> None:
|
||||
"""Every name answered `nxdomain`: each individual read succeeded, and the run still refuses.
|
||||
|
||||
A resolver that returns "no records" for the whole config is a firewall, a dead upstream DNS,
|
||||
or a box that has never been pointed at one — and the fixture it would produce is the same
|
||||
nothing the other two arms produce.
|
||||
"""
|
||||
result, fixtures = _resolver_run(tmp_path, recorded=dict(RESOLVER_KEEP), answers={})
|
||||
assert result.returncode == 1, result.stdout
|
||||
assert "none of the" in result.stdout and "vantage is broken" in result.stdout, result.stdout
|
||||
assert (fixtures / "gateway_resolver.json").read_text() == RESOLVER_KEEP[
|
||||
"gateway_resolver.json"]
|
||||
|
||||
|
||||
def test_a_dead_gateway_does_not_attempt_the_resolver_document(tmp_path: Path) -> None:
|
||||
"""With no configs there is no name list, and a document of the wrong question is worse than none."""
|
||||
result = _capture(tmp_path, recorded=dict(RESOLVER_KEEP), fail_ssh=True)
|
||||
fixtures = tmp_path / "tests" / "fixtures"
|
||||
assert result.returncode == 1, result.stdout
|
||||
assert "not attempted" in result.stdout, result.stdout
|
||||
assert (fixtures / "gateway_resolver.json").read_text() == RESOLVER_KEEP[
|
||||
"gateway_resolver.json"]
|
||||
assert sorted(p.name for p in fixtures.iterdir()) == ["gateway_resolver.json"], (
|
||||
"a refused resolver capture left a temp file in tests/fixtures/")
|
||||
|
||||
|
||||
def test_the_socket_capture_is_idempotent_so_a_re_run_does_not_churn(tmp_path: Path) -> None:
|
||||
"""Re-recording the same moment must not diff, or every capture moves the fixture twice.
|
||||
|
||||
|
||||
+741
-26
@@ -18,13 +18,18 @@ outside the two configs is *reported*, because it no longer is —
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import ipaddress
|
||||
import json
|
||||
import socket as socket_module
|
||||
from collections.abc import Mapping, Sequence
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from dashboards.sources import gateway
|
||||
|
||||
#: This repo's recorded captures, for the pins that read a fixture directly.
|
||||
FIXTURES = Path(__file__).resolve().parent / "fixtures"
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# synthetic builders: a section exactly as the agent serves it, minus the file
|
||||
# --------------------------------------------------------------------------- #
|
||||
@@ -111,10 +116,31 @@ def conn(local: tuple[str, int], remote: tuple[str, int], *,
|
||||
"remote": {"addr": remote[0], "port": remote[1]}}
|
||||
|
||||
|
||||
def resolver_section(entries: Sequence[dict[str, Any]] = (), *, available: bool | None = True,
|
||||
via: str | None = "libc", config_directive: str | None = "10.1.0.1",
|
||||
ts: float = 1_760_000_000.0,
|
||||
errors: Sequence[str] = ()) -> dict[str, Any]:
|
||||
"""Agent 1.3.0's fourth section, in the shape `host_agent.Collector.resolver` publishes."""
|
||||
return {"available": available, "via": via, "config_directive": config_directive, "ts": ts,
|
||||
"entries": list(entries), "errors": list(errors)}
|
||||
|
||||
|
||||
def resolved(name: str, port: int | None, addresses: Sequence[str], *,
|
||||
status: str = "ok", ts: float | None = 1_760_000_000.0) -> dict[str, Any]:
|
||||
"""One resolver entry: what libc answered for `(name, port)`, and when."""
|
||||
return {"name": name, "port": port, "status": status, "addresses": list(addresses), "ts": ts}
|
||||
|
||||
|
||||
def snapshot(*, haproxy: dict[str, Any] | None = None, nginx: dict[str, Any] | None = None,
|
||||
sockets: dict[str, Any] | None = None, errors: Sequence[str] = (),
|
||||
sockets: dict[str, Any] | None = None, resolver: dict[str, Any] | None = None,
|
||||
errors: Sequence[str] = (),
|
||||
agent_version: str | None = "1.2.0") -> dict[str, Any]:
|
||||
"""The document one agent `/snapshot` would answer, with only what a test names."""
|
||||
"""The document one agent `/snapshot` would answer, with only what a test names.
|
||||
|
||||
No `resolver` key is the default, because that is the arm most of this file predates: a 1.2.0
|
||||
agent, or a 1.3.0 one run with `--no-resolve`. A test that wants a name to be countable passes
|
||||
`resolver=resolver_section([...])` and says so.
|
||||
"""
|
||||
doc: dict[str, Any] = {"ok": True, "schema": 1, "ts": 1_760_000_000.0, "host": "gateway",
|
||||
"errors": list(errors)}
|
||||
if agent_version is not None:
|
||||
@@ -125,6 +151,8 @@ def snapshot(*, haproxy: dict[str, Any] | None = None, nginx: dict[str, Any] | N
|
||||
doc["nginx"] = nginx
|
||||
if sockets is not None:
|
||||
doc["sockets"] = sockets
|
||||
if resolver is not None:
|
||||
doc["resolver"] = resolver
|
||||
return doc
|
||||
|
||||
|
||||
@@ -162,6 +190,52 @@ def out_metrics(payload: Mapping[str, Any]) -> list[str]:
|
||||
return [row["metric"] for row in payload["connections"] if row["metric"] is not None]
|
||||
|
||||
|
||||
def endpoint_keys(row: Mapping[str, Any]) -> set[tuple[str, int]]:
|
||||
"""The normalised `(address, port)` keys one route reaches, derived **in the test**.
|
||||
|
||||
Read straight off `out.address` / `out.resolved` with `ipaddress`, deliberately not by calling
|
||||
`gateway.endpoint_of`: a total a test compares against must not come from the function under
|
||||
test. The normalisation is the same one the join makes, because `::ffff:10.1.2.64` and
|
||||
`10.1.2.64` are one endpoint and a ledger that counted them apart would reconcile to nothing.
|
||||
"""
|
||||
out = row["out"]
|
||||
port = out.get("port")
|
||||
if not isinstance(port, int):
|
||||
return set()
|
||||
spellings = [out["host"]] if out.get("address") is True else list(out.get("resolved") or [])
|
||||
keys: set[tuple[str, int]] = set()
|
||||
for spelling in spellings:
|
||||
try:
|
||||
address = ipaddress.ip_address(str(spelling))
|
||||
except ValueError:
|
||||
continue
|
||||
mapped = getattr(address, "ipv4_mapped", None)
|
||||
keys.add((str(mapped if mapped is not None else address), port))
|
||||
return keys
|
||||
|
||||
|
||||
def endpoint_groups(payload: Mapping[str, Any]) -> dict[tuple[str, int], list[Mapping[str, Any]]]:
|
||||
"""The route lines grouped by the **distinct endpoint** they reach, keyed by min(key).
|
||||
|
||||
Lines whose key sets are identical share a group, which is every case a capture can produce
|
||||
(one address per name, or one name written by two vhosts). `gateway._endpoint_groups` merges
|
||||
*overlapping* sets as well, for a multi-homed name sharing one address with a single-homed one;
|
||||
no capture has that shape, and if one ever does this helper disagrees by keeping two groups —
|
||||
which fails the totals test below loudly rather than quietly.
|
||||
"""
|
||||
groups: dict[tuple[str, int], list[Mapping[str, Any]]] = {}
|
||||
for row in payload["connections"]:
|
||||
keys = endpoint_keys(row)
|
||||
if keys:
|
||||
groups.setdefault(min(keys), []).append(row)
|
||||
return groups
|
||||
|
||||
|
||||
def distinct_endpoint_totals(payload: Mapping[str, Any]) -> dict[tuple[str, int], Any]:
|
||||
"""One count per distinct endpoint — the figure a total may add, never one per line."""
|
||||
return {key: rows[0]["connections"] for key, rows in endpoint_groups(payload).items()}
|
||||
|
||||
|
||||
def walk_keys(node: Any) -> set[str]:
|
||||
"""Every key name anywhere in a payload — the device the "no severity / no window" pins use."""
|
||||
if isinstance(node, Mapping):
|
||||
@@ -200,7 +274,7 @@ def test_the_captured_configs_yield_every_configured_pipe(gateway_snapshot: Any)
|
||||
"five vhosts share 4443 and their domains belong to the one pipe for that port")
|
||||
|
||||
assert payload["nginx"]["map_targets"], "the stream map is the fan-out the page draws"
|
||||
assert len(payload["nginx"]["map_targets"]) == 17
|
||||
assert len(payload["nginx"]["map_targets"]) == 18
|
||||
homepage = next(row for row in payload["nginx"]["map_targets"]
|
||||
if row["key"] == "homepage.reeseapps.com")
|
||||
assert homepage == {"key": "homepage.reeseapps.com",
|
||||
@@ -229,7 +303,12 @@ def test_the_source_reads_the_keys_the_agent_actually_publishes(gateway_snapshot
|
||||
assert {row["port"] for row in payload["nginx"]["servers"]} == agent_listen_ports
|
||||
# and the three section names themselves, as the snapshot publishes them
|
||||
assert {"haproxy", "nginx", "sockets"} <= set(snap)
|
||||
assert host_agent_module.AGENT_VERSION == "1.2.0", "the sections these keys belong to"
|
||||
# At least 1.2.0, the version that shipped these three keys. It is a floor and not an
|
||||
# equality on purpose: phase 32 added a fourth section and bumped the constant, and the
|
||||
# coupling this test pins is that the three the source joins still exist in what an agent
|
||||
# publishes, whatever the number on it has since grown.
|
||||
version = tuple(int(part) for part in host_agent_module.AGENT_VERSION.split("."))
|
||||
assert version >= (1, 2, 0), "the sections these keys belong to"
|
||||
|
||||
|
||||
def test_the_live_counts_reconcile_with_the_socket_table(gateway_snapshot: Any) -> None:
|
||||
@@ -247,29 +326,60 @@ def test_the_live_counts_reconcile_with_the_socket_table(gateway_snapshot: Any)
|
||||
What stays pinned is the direction of the error and the absence of a residual: the reported
|
||||
figures are always **under** the kernel's number, and there is no field anywhere that quietly
|
||||
absorbs the difference to make the two agree.
|
||||
|
||||
Run against the **`resolved`** capture, because that is what the deployed ingress serves since
|
||||
phase 32 task 05: the widened match set is the one the arithmetic has to hold for. Three
|
||||
figures, read three ways:
|
||||
|
||||
* `in_total` — the sum over the listeners (each socket at most once, inbound);
|
||||
* `out_total` — the sum over **distinct endpoints**, derived here from `out.address` /
|
||||
`out.resolved` rather than from `gateway.endpoint_of`;
|
||||
* `lines_total` — the sum over the **lines**, which is deliberately larger wherever a line
|
||||
repeats a number (`shared`), and is pinned to be exactly `out_total` plus the repetitions.
|
||||
|
||||
The `127.0.0.1:4443` loopback hop is the boundary case: it is booked **once inbound** (the rows
|
||||
whose local port is 4443, on the `:4443` trunk) and **once outbound** (the rows whose remote is
|
||||
4443, on the `:443 → 127.0.0.1:4443` pipe) and no single `/proc` row is in both.
|
||||
"""
|
||||
snap = gateway_snapshot("full")
|
||||
snap = gateway_snapshot("resolved")
|
||||
payload = gateway.parse(snap, "gateway")
|
||||
established = len(snap["sockets"]["established"])
|
||||
established = snap["sockets"]["established"]
|
||||
configured = {row["port"] for row in payload["listeners"]}
|
||||
assert configured == {22, 2000, 5201, 8080, 25565, 25566, 80, 443, 4443, 8443}
|
||||
assert all(row["port"] in configured for row in payload["listeners"]), (
|
||||
"an inbound count on a port no config names cannot exist in this payload")
|
||||
in_total = sum(row["connections"] for row in payload["listeners"])
|
||||
endpoints = {(row["out"]["host"], row["out"]["port"]): row["connections"]
|
||||
for row in payload["connections"] if row["metric"] is not None}
|
||||
out_total = sum(value or 0 for value in endpoints.values())
|
||||
totals = distinct_endpoint_totals(payload)
|
||||
out_total = sum(value or 0 for value in totals.values())
|
||||
lines_total = sum(row["connections"] or 0 for row in payload["connections"])
|
||||
repeated = sum((len(rows) - 1) * (rows[0]["connections"] or 0)
|
||||
for rows in endpoint_groups(payload).values() if len(rows) > 1)
|
||||
|
||||
rows_in_table = len(established)
|
||||
assert payload["sockets"] == {"listens": len(snap["sockets"]["listens"]),
|
||||
"established": established}
|
||||
assert in_total + out_total <= established
|
||||
assert in_total + out_total < established, (
|
||||
"established": rows_in_table}
|
||||
assert in_total + out_total <= rows_in_table
|
||||
assert in_total + out_total < rows_in_table, (
|
||||
"this capture has traffic outside both config files; if it ever stops having any, this "
|
||||
"test is proving nothing and should be re-written against a capture that does")
|
||||
assert "unmatched" not in payload and "partial" not in payload
|
||||
# the loopback hop is the case that used to double count: `127.0.0.1:4443` is both a listener's
|
||||
# remote end and an outbound endpoint, and it may be booked on only one of them
|
||||
assert sum(row["connections"] or 0 for row in payload["connections"]) == out_total
|
||||
assert lines_total == out_total + repeated, (
|
||||
"the only difference between the lines and the endpoints is a repeated count — no socket "
|
||||
"is booked twice and no endpoint is booked once per name that reaches it")
|
||||
|
||||
# the loopback hop, both sides, from the raw table rather than from either counter
|
||||
accepted = [row for row in established if (row["local"]["port"]) == 4443]
|
||||
dialled = [row for row in established
|
||||
if row["local"]["port"] != 4443 and (row["remote"]["addr"], row["remote"]["port"])
|
||||
== ("127.0.0.1", 4443)]
|
||||
assert accepted and dialled, "this capture is the loop the stream map's `default` produces"
|
||||
assert listener_at(payload, 4443)["connections"] == len(accepted)
|
||||
assert endpoint_count(payload, "127.0.0.1", 4443) == len(dialled)
|
||||
assert not [row for row in established
|
||||
if row["local"]["port"] == 4443
|
||||
and (row["remote"]["addr"], row["remote"]["port"]) == ("127.0.0.1", 4443)], (
|
||||
"a row that was both would be counted on two pipes by two rules, and the ≤ above would "
|
||||
"still hold while every figure in the payload was wrong")
|
||||
|
||||
|
||||
def test_a_socket_neither_config_describes_is_counted_nowhere(gateway_snapshot: Any) -> None:
|
||||
@@ -293,7 +403,7 @@ def test_every_connection_the_captured_configs_declare_is_drawn(gateway_snapshot
|
||||
"""The graph as a **set**, so a missing or invented route fails by name.
|
||||
|
||||
The join this task rewrote stamps counts onto these rows, so the rows themselves are pinned
|
||||
here too: 6 haproxy `server` lines, 17 stream-`map` entries (the 16 `*.reeselink.com:443` fan-
|
||||
here too: 6 haproxy `server` lines, 18 stream-`map` entries (the 17 `*.reeselink.com:443` fan-
|
||||
out plus the literal `default` → `127.0.0.1:4443`), and 9 vhost `proxy_pass` targets across
|
||||
`:4443` and `:8443`. `:80` is the refusal that lives on this fleet: the catch-all vhost declares
|
||||
no `proxy_pass`, so it is a trunk with no connection and no invented line beside it.
|
||||
@@ -317,6 +427,7 @@ def test_every_connection_the_captured_configs_declare_is_drawn(gateway_snapshot
|
||||
(443, "reeseapps-gitea.reeselink.com", 443),
|
||||
(443, "reeseapps-homepage.reeselink.com", 443),
|
||||
(443, "reeseapps-immich.reeselink.com", 443),
|
||||
(443, "reeseapps-whiteboard.reeselink.com", 443),
|
||||
(443, "reeseapps-keycloak.reeselink.com", 443),
|
||||
(443, "reeseapps-litellm.reeselink.com", 443),
|
||||
(443, "reeseapps-ntfy.reeselink.com", 443),
|
||||
@@ -362,9 +473,9 @@ def test_the_map_fanout_is_routes_and_never_a_variable_host(gateway_snapshot: An
|
||||
"a proxy_pass $var carries no address, so it must not become a line of its own")
|
||||
brain = next(row for row in map_routes
|
||||
if row["out"]["host"] == "reeseapps-brain-of-reese.reeselink.com")
|
||||
assert brain["out"]["address"] is False, "the config wrote a name, and this stack resolves none"
|
||||
assert brain["connections"] is None, "so the line carries no count"
|
||||
assert brain["metric"] is None, "and publishes no series"
|
||||
assert brain["out"]["address"] is False, "the config wrote a name, and the config is verbatim"
|
||||
assert brain["connections"] is None, "and on this arm (no resolver section) nothing placed it"
|
||||
assert brain["metric"] is None, "so it publishes no series"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
@@ -476,10 +587,12 @@ def test_a_name_target_is_never_credited_for_a_socket_on_its_port() -> None:
|
||||
"""The rule that replaced `by_port`, stated as a fact about the payload and not a chip.
|
||||
|
||||
`proxy_pass https://nextcloud.example.test:11000` and the socket to `10.1.0.254:11000` are the
|
||||
same connection in the operator's head and two unrelated strings in this stack. Attributing one
|
||||
to the other needed a resolver, and the port-only guess the old join made in its place was a
|
||||
guess about a **port**, not about a host — which is why its count was the port's and its chip
|
||||
said so. Now the line simply has no number, no gauge, and no apology to render.
|
||||
same connection in the operator's head and two unrelated strings **in this stack**. The bridge is
|
||||
the gateway's own resolution (phase 32, tested below), and the port-only guess the old join made
|
||||
in its place was a guess about a **port**, not about a host — which is why its count was the
|
||||
port's and its chip said so. With nothing to bridge them the line has no number, no gauge, and no
|
||||
apology to render; with the agent's document it is counted through the address, and this test
|
||||
stays true because the snapshot it builds has no resolver section.
|
||||
"""
|
||||
snap = snapshot(
|
||||
nginx=nginx_section([nginx_server([4443], ["cloud.example.test"],
|
||||
@@ -503,7 +616,7 @@ def test_a_name_target_is_never_credited_for_a_socket_on_its_port() -> None:
|
||||
|
||||
|
||||
def test_a_port_shared_by_several_names_still_resolves_none_of_them() -> None:
|
||||
"""The real `stream.conf` shape: sixteen names on 443, none of them counted.
|
||||
"""The real `stream.conf` shape: sixteen names on 443, none of them counted **without DNS**.
|
||||
|
||||
The answer is the same `null` the old join gave, minus the machinery that produced it and the
|
||||
`unmatched.outbound` total that used to double-count the same sockets.
|
||||
@@ -557,6 +670,68 @@ def test_no_join_ever_asks_a_resolver() -> None:
|
||||
assert calls == []
|
||||
|
||||
|
||||
def test_the_recorded_resolver_document_reaches_every_name_written_route(
|
||||
gateway_snapshot: Any) -> None:
|
||||
"""What phase 32 buys, priced against the captures — and spent, since task 03.
|
||||
|
||||
31 of this ingress's 33 routes carried an em dash because this module joins sockets by
|
||||
**address** and the two config files write those endpoints as **names**. The recorded
|
||||
`gateway_resolver.json` is the thing that removed that reason, so this pins the two properties
|
||||
the join needs and no more:
|
||||
|
||||
* every name-written route has an entry in the document, for that name **and that port**, with
|
||||
at least one address — so no route on `/gateway` is dashed forever because the resolver had
|
||||
nothing to say about its endpoint;
|
||||
* the addresses are the same currency the socket table is written in: the two captures overlap
|
||||
on live remote endpoints, which is what a join has to have to join anything.
|
||||
|
||||
and, in the last few lines, that spending it is a fact about the **document** rather than a
|
||||
coincidence of the port table: the name-written routes are counted exactly where the entry puts
|
||||
an address and nowhere else. This module still resolves nothing — the spy pin above stays exactly
|
||||
as it was, and is the reason the fixture has to exist.
|
||||
"""
|
||||
snap = gateway_snapshot("full")
|
||||
payload = gateway.parse(snap, "gateway")
|
||||
document = json.loads((FIXTURES / "gateway_resolver.json").read_text())
|
||||
answered = {(row["name"], row["port"]) for row in document["entries"] if row["addresses"]}
|
||||
|
||||
written = {(row["out"]["host"], row["out"]["port"]) for row in payload["connections"]
|
||||
if not row["out"]["address"]}
|
||||
assert written, "this capture writes its targets as names, which is the gap the fixture fills"
|
||||
unreached = written - answered
|
||||
assert not unreached, ("name-written routes the resolver document cannot reach: "
|
||||
f"{sorted(unreached)} — re-run ./scripts/capture-gateway-config.sh")
|
||||
|
||||
resolved = {address for row in document["entries"] for address in row["addresses"]}
|
||||
remote = {row["remote"]["addr"] for row in snap["sockets"]["established"]}
|
||||
assert resolved & remote, (
|
||||
"the resolver document and the socket table share no address, so a join built on it could "
|
||||
"attribute nothing — the two captures are from different boxes or one scrub moved a number")
|
||||
assert all(row["connections"] is None for row in payload["connections"]
|
||||
if (row["out"]["host"], row["out"]["port"]) in written), (
|
||||
"this arm is the snapshot with no `resolver` key at all, so a name-written route still "
|
||||
"carries no count here — and adding one address to that section is what changes it")
|
||||
|
||||
spent = gateway.parse(gateway_snapshot("resolved"), "gateway")
|
||||
rows = {(row["out"]["host"], row["out"]["port"]): row for row in spent["connections"]}
|
||||
trunks = {row["port"] for row in spent["listeners"]}
|
||||
for name, port in sorted(written):
|
||||
row = rows[(name, port)]
|
||||
entry = next(item for item in document["entries"]
|
||||
if (item["name"], item["port"]) == (name, port))
|
||||
assert row["out"]["resolved"] == entry["addresses"], f"{name}:{port} rewrites the capture"
|
||||
assert row["connections"] is not None, f"{name}:{port} is placed and still carries no count"
|
||||
# the exact figure, re-counted from the raw table by this test: the rows that reach this
|
||||
# endpoint, minus the ones the inbound arm claimed first (a socket is counted at most once)
|
||||
keys = {(address, port) for address in entry["addresses"]}
|
||||
live = [item for item in snap["sockets"]["established"]
|
||||
if item["local"]["port"] not in trunks
|
||||
and (item["remote"]["addr"], item["remote"]["port"]) in keys]
|
||||
assert row["connections"] == len(live), (
|
||||
f"{name}:{port} resolves to {sorted(keys)} and the table holds {len(live)} such "
|
||||
f"sockets, but the line reads {row['connections']}")
|
||||
|
||||
|
||||
def test_two_services_on_one_host_are_two_pipes_not_one_sum() -> None:
|
||||
"""The reason the outbound series name carries its port (`minecraft` 25565 and 25566)."""
|
||||
snap = snapshot(
|
||||
@@ -750,6 +925,546 @@ def test_an_endpoint_reached_by_one_line_is_not_shared() -> None:
|
||||
assert row["connections"] == 1
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# phase 32: the resolver section — a name counted through the address the gateway dialled
|
||||
#
|
||||
# Everything above this line runs on a snapshot with **no** `resolver` key, which is agent 1.2.0 (or
|
||||
# 1.3.0 with `--no-resolve`) and therefore phase 30 verbatim. Everything below it spends the
|
||||
# document `scripts/capture-gateway-config.sh` recorded with `getent` on the router: the resolution
|
||||
# haproxy and nginx perform **on the box that opens the connection**, which is the only vantage that
|
||||
# can join `reeseapps-brain-of-reese.reeselink.com` to the `10.1.2.64` `/proc` reports. The
|
||||
# dashboard asking the same name through its own `resolv.conf` would be a different vantage
|
||||
# answering a different question — and `test_the_dashboard_still_never_resolves` is the spy that
|
||||
# keeps it that way.
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
def test_a_name_route_counts_through_the_address_the_gateway_published(
|
||||
gateway_snapshot: Any) -> None:
|
||||
"""The whole point of phase 32, measured against the captures.
|
||||
|
||||
`stream.conf` routes `brain.reeseapps.com` to `reeseapps-brain-of-reese.reeselink.com:443`,
|
||||
`/proc` would report a socket to `10.1.2.64:443`, and until this section arrived nothing could
|
||||
join the two. Now the line carries the config's text and the agent's answer side by side, and
|
||||
the count comes through the answer.
|
||||
"""
|
||||
snap = gateway_snapshot("resolved")
|
||||
payload = gateway.parse(snap, "gateway")
|
||||
brain = next(row for row in payload["connections"]
|
||||
if row["out"]["host"] == "reeseapps-brain-of-reese.reeselink.com")
|
||||
assert brain["in"] == {"port": 443}
|
||||
assert brain["out"]["selectors"] == ["brain.reeseapps.com"], "the config's word, untouched"
|
||||
assert brain["out"]["address"] is False, "the config wrote a name, and still says so"
|
||||
assert brain["out"]["resolved"] == ["10.1.2.64"], "the agent's answer, verbatim"
|
||||
assert brain["out"]["via"] == "libc", "the vantage is the agent's own string, copied"
|
||||
assert brain["out"]["resolved_ts"] == 1_792_886_400.0, "that entry's own timestamp"
|
||||
assert brain["metric"] == "gateway.out_conns@reeseapps-brain-of-reese.reeselink.com:443", (
|
||||
"the gauge keeps the config's spelling, so history continues under one name")
|
||||
assert brain["connections"] == 0, "resolved and idle is `0`, never the dash"
|
||||
|
||||
ntfy = next(row for row in payload["connections"]
|
||||
if row["out"]["host"] == "reeseapps-ntfy.reeselink.com")
|
||||
live = [row for row in snap["sockets"]["established"]
|
||||
if (row["remote"]["addr"], row["remote"]["port"]) == ("10.1.1.176", 443)]
|
||||
assert live, "the recorded table has live ntfy sockets for the join to find"
|
||||
assert ntfy["connections"] == len(live), "every socket to the address the name became"
|
||||
assert ntfy["shared"] is False, "one route reaches that endpoint, so nothing to repeat"
|
||||
|
||||
|
||||
def test_an_endpoint_the_config_wrote_as_an_address_claims_no_lookup(
|
||||
gateway_snapshot: Any) -> None:
|
||||
"""`resolved` answers *what did this name become*, and an address was never a name.
|
||||
|
||||
The agent publishes a `literal` entry for a target that is already an address, and the resolver
|
||||
block lists it verbatim. Stamping that echo onto the route as `resolved` beside `via: "libc"`
|
||||
would report a `getaddrinfo` this gateway never ran, so the three provenance fields stay `null`
|
||||
on an `address: true` row while the count and the gauge keep working exactly as phase 30 left
|
||||
them. Two facts stay two sentences: `address` is what the config wrote, `resolved` is what the
|
||||
gateway looked up.
|
||||
"""
|
||||
payload = gateway.parse(gateway_snapshot("resolved"), "gateway")
|
||||
written = [row for row in payload["connections"] if row["out"]["address"] is True]
|
||||
assert written, "this capture writes two targets as addresses, which is phase 30's joinable pair"
|
||||
for row in written:
|
||||
assert row["out"]["host"] in {entry["name"]
|
||||
for entry in payload["resolver"]["entries"]}, (
|
||||
"the agent did publish this literal, and the block still lists it")
|
||||
assert row["out"]["resolved"] is None and row["out"]["via"] is None \
|
||||
and row["out"]["resolved_ts"] is None, (
|
||||
f"{row['id']}: an address row must not borrow a lookup claim")
|
||||
assert row["metric"] == f"gateway.out_conns@{row['out']['host']}:{row['out']['port']}"
|
||||
assert row["connections"] is not None, "phase 30's address join is untouched by this"
|
||||
|
||||
|
||||
def test_a_socket_lands_on_the_name_route_that_resolved_to_its_address() -> None:
|
||||
"""The synthetic version, so the claim does not depend on what a capture happened to contain."""
|
||||
snap = snapshot(
|
||||
nginx=nginx_section([nginx_server([443], [], [proxy_pass(None, variable="targetBackend")])],
|
||||
maps=[mapping_block("$targetBackend",
|
||||
[("brain.example.test",
|
||||
("brain-backend.example.test", 443))])]),
|
||||
sockets=sockets_section(
|
||||
listens=[listen(443)],
|
||||
established=[conn(("::", 51000), ("10.1.2.64", 443), family="v6"),
|
||||
conn(("10.1.1.27", 443), ("203.0.113.7", 900))]),
|
||||
resolver=resolver_section([resolved("brain-backend.example.test", 443, ["10.1.2.64"])]))
|
||||
payload = gateway.parse(snap, "gateway")
|
||||
row = routes_to(payload, "brain-backend.example.test")[0]
|
||||
assert row["out"]["address"] is False and row["out"]["resolved"] == ["10.1.2.64"]
|
||||
assert row["connections"] == 1
|
||||
assert row["metric"] == "gateway.out_conns@brain-backend.example.test:443"
|
||||
assert listener_at(payload, 443)["connections"] == 1, "the inbound half is untouched"
|
||||
gauge = [g for g in gateway.samples(payload, 5.0)
|
||||
if g.metric == "gateway.out_conns@brain-backend.example.test:443"]
|
||||
assert [(g.metric, g.value) for g in gauge] == [
|
||||
("gateway.out_conns@brain-backend.example.test:443", 1.0)]
|
||||
|
||||
|
||||
def test_the_match_is_an_address_not_a_string() -> None:
|
||||
"""One address, two spellings, in either direction: the join still lands.
|
||||
|
||||
`/proc`'s `inet_ntop` writes an IPv4 peer seen through the IPv6 table as `::ffff:10.1.2.64`
|
||||
while a resolver answers `10.1.2.64`, and an IPv6 in one table can carry leading zeros and
|
||||
capitals the other does not. `ipaddress` canonicalises both, so the **match key** is one address
|
||||
while the **payload** keeps publishing whatever the config and the agent wrote — a page that
|
||||
printed the normalised form would be quoting a string nobody configured.
|
||||
|
||||
A join that compared strings instead would report a resolved, dialled route as idle forever:
|
||||
the exact failure this phase exists to remove, arriving back through a spelling nobody thought
|
||||
about. The two directions are asserted separately because each half is a different table.
|
||||
"""
|
||||
def build(written: str, socketed: str) -> dict[str, Any]:
|
||||
snap = snapshot(
|
||||
haproxy=haproxy_section([frontend("peer", [8001], default="peer-be")],
|
||||
[backend("peer-be", [server("p", "peer.example.test", 443)])]),
|
||||
sockets=sockets_section(
|
||||
established=[conn(("::", 40000), (socketed, 443), family="v6")]),
|
||||
resolver=resolver_section([resolved("peer.example.test", 443, [written])]))
|
||||
return gateway.parse(snap, "gateway")
|
||||
|
||||
for written, socketed in (("10.1.2.64", "::ffff:10.1.2.64"),
|
||||
("::ffff:10.1.2.64", "10.1.2.64"),
|
||||
("2001:db8::1", "2001:0DB8::0001"),
|
||||
("2001:0DB8::0001", "2001:db8::1")):
|
||||
payload = build(written, socketed)
|
||||
row = routes_to(payload, "peer.example.test")[0]
|
||||
assert row["connections"] == 1, f"{written} vs {socketed} is one address and did not match"
|
||||
assert row["out"]["resolved"] == [written], "the payload keeps the agent's own spelling"
|
||||
|
||||
|
||||
def test_a_zero_padded_ipv4_is_no_address_to_anyone_and_matches_nothing() -> None:
|
||||
"""The one spelling this module refuses to fold, and why refusing is the safe half.
|
||||
|
||||
`ipaddress` rejects `010.1.2.64` because `inet_aton` reads a leading `0` as **octal**: the
|
||||
decimal and octal readings are two different addresses, and picking one here would book a
|
||||
socket on a route by coin flip. So a padded target is a **name** to this module — and the
|
||||
resolver section, which is libc's own answer, is what turns it into an address the kernel agrees
|
||||
with. Nothing is guessed in between.
|
||||
"""
|
||||
snap = snapshot(
|
||||
haproxy=haproxy_section([frontend("be", [8080], default="be")],
|
||||
[backend("be", [server("odd", "010.1.2.64", 443)])]),
|
||||
sockets=sockets_section(
|
||||
established=[conn(("10.1.1.27", 40000), ("10.1.2.64", 443)),
|
||||
conn(("10.1.1.27", 40002), ("8.1.2.64", 443))]))
|
||||
row = routes_to(gateway.parse(snap, "gateway"), "010.1.2.64")[0]
|
||||
assert row["out"]["address"] is False, "not an address here, so it is a name"
|
||||
assert row["connections"] is None and row["metric"] is None, (
|
||||
"and with no resolver entry it is counted nowhere — neither reading is chosen for it")
|
||||
|
||||
# …and the fix is the resolver's, not this module's: libc's own answer places the same route,
|
||||
# the padded spelling travels in the payload untouched, and neither socket is read as an octet.
|
||||
placed = snapshot(
|
||||
haproxy=haproxy_section([frontend("be", [8080], default="be")],
|
||||
[backend("be", [server("odd", "010.1.2.64", 443)])]),
|
||||
sockets=sockets_section(
|
||||
established=[conn(("10.1.1.27", 40000), ("10.1.2.64", 443)),
|
||||
conn(("10.1.1.27", 40002), ("8.1.2.64", 443))]),
|
||||
resolver=resolver_section([resolved("010.1.2.64", 443, ["10.1.2.64"])]))
|
||||
fixed = routes_to(gateway.parse(placed, "gateway"), "010.1.2.64")[0]
|
||||
assert fixed["out"]["resolved"] == ["10.1.2.64"]
|
||||
assert fixed["connections"] == 1, (
|
||||
"the one socket to the address libc returned, not the octal-reading 8.1.2.64 and not both")
|
||||
|
||||
|
||||
def test_two_names_one_endpoint_is_one_count_shared() -> None:
|
||||
"""Two **names** the gateway placed on one address are one endpoint and one number.
|
||||
|
||||
Phase 30 knew this shape only for one address written twice; resolving makes it ordinary, and
|
||||
the rule did not change: the endpoint's whole count is repeated on both lines with
|
||||
`shared: true`, never halved and never attributed to one of them.
|
||||
|
||||
Two readers then differ, and both are right. **Every total** books the endpoint once
|
||||
(`distinct_endpoint_counts`, which is what `/api/summary`'s `live_out` reads), while the gauge
|
||||
is per line-family — one series per name the config wrote, each carrying that endpoint's number,
|
||||
because a route's own history is asked for by the name on the route. Two names therefore store
|
||||
two series of one figure; lines that share a **name** (`:4443` and `:8443` reaching one target)
|
||||
still store exactly one.
|
||||
"""
|
||||
snap = snapshot(
|
||||
nginx=nginx_section([nginx_server([4443], ["a.example.test"],
|
||||
[proxy_pass(("one.example.test", 8080))]),
|
||||
nginx_server([8443], ["b.example.test"],
|
||||
[proxy_pass(("two.example.test", 8080))])]),
|
||||
sockets=sockets_section(
|
||||
established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 8080)),
|
||||
conn(("10.1.1.27", 40002), ("10.5.5.5", 8080))]),
|
||||
resolver=resolver_section([resolved("one.example.test", 8080, ["10.5.5.5"]),
|
||||
resolved("two.example.test", 8080, ["10.5.5.5"])]))
|
||||
payload = gateway.parse(snap, "gateway")
|
||||
rows = [row for row in payload["connections"] if row["in"]["port"] in (4443, 8443)]
|
||||
assert [row["connections"] for row in rows] == [2, 2], "the endpoint's total, twice"
|
||||
assert [row["shared"] for row in rows] == [True, True]
|
||||
assert gateway.distinct_endpoint_counts(payload["connections"]) == [2], (
|
||||
"and one figure for anything that adds the outbound side up")
|
||||
assert out_metrics(payload) == ["gateway.out_conns@one.example.test:8080",
|
||||
"gateway.out_conns@two.example.test:8080"]
|
||||
|
||||
|
||||
def test_two_spellings_of_one_multi_homed_name_are_one_endpoint() -> None:
|
||||
"""A name with three addresses is **one** route, not three, and its sockets are one count.
|
||||
|
||||
Every key the resolver published points at the same counter row, so a client round-robin across
|
||||
`10.5.5.5`/`10.5.5.6` reads as three connections on one pipe. The case union-find exists for is
|
||||
two names whose address sets *overlap* — a multi-homed host and a single-homed alias for one of
|
||||
its addresses — which would otherwise count the same endpoint twice in one total.
|
||||
"""
|
||||
snap = snapshot(
|
||||
nginx=nginx_section([nginx_server([4443], ["a.example.test"],
|
||||
[proxy_pass(("multi.example.test", 443))]),
|
||||
nginx_server([8443], ["b.example.test"],
|
||||
[proxy_pass(("alias.example.test", 443))])]),
|
||||
sockets=sockets_section(
|
||||
established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 443)),
|
||||
conn(("10.1.1.27", 40002), ("10.5.5.6", 443))]),
|
||||
resolver=resolver_section([resolved("multi.example.test", 443,
|
||||
["10.5.5.5", "10.5.5.6"]),
|
||||
resolved("alias.example.test", 443, ["10.5.5.6"])]))
|
||||
payload = gateway.parse(snap, "gateway")
|
||||
rows = [row for row in payload["connections"] if row["in"]["port"] in (4443, 8443)]
|
||||
assert [row["connections"] for row in rows] == [2, 2], (
|
||||
"the shared endpoint's two sockets, on both lines — not one socket each")
|
||||
assert [row["shared"] for row in rows] == [True, True]
|
||||
assert gateway.distinct_endpoint_counts(payload["connections"]) == [2], (
|
||||
"overlapping address sets are one endpoint, so a total books them once")
|
||||
|
||||
|
||||
def test_a_name_that_bridges_two_others_merges_all_three_lines_into_one_endpoint() -> None:
|
||||
"""A→{x}, B→{x,y}, C→{y}: three routes, one endpoint, one figure — transitivity included.
|
||||
|
||||
The bridging name is what makes this a *merge* rather than three pairwise comparisons: the first
|
||||
and third share no address, and yet the same kernel endpoint sits behind both, so counting them
|
||||
separately would book two endpoints where `/proc` has one. **The config order is load-bearing**
|
||||
beyond the transitivity: the bridge arrives second and its lowest address (`10.5.5.5`) is lower
|
||||
than the root the first route already built, so that whole group is re-parented, and the third
|
||||
route's lookup then has to walk through two links to find it — the path the union-find compresses
|
||||
so a fan-out of this size stays cheap on the next poll.
|
||||
"""
|
||||
snap = snapshot(
|
||||
nginx=nginx_section([nginx_server([4443], ["a.example.test"],
|
||||
[proxy_pass(("multi.example.test", 443))]),
|
||||
nginx_server([8443], ["b.example.test"],
|
||||
[proxy_pass(("bridge.example.test", 443))]),
|
||||
nginx_server([8444], ["c.example.test"],
|
||||
[proxy_pass(("alias.example.test", 443))])]),
|
||||
sockets=sockets_section(
|
||||
established=[conn(("10.1.1.27", 40000), ("10.5.5.6", 443)),
|
||||
conn(("10.1.1.27", 40002), ("10.5.5.7", 443))]),
|
||||
resolver=resolver_section([resolved("multi.example.test", 443,
|
||||
["10.5.5.6", "10.5.5.7"]),
|
||||
resolved("bridge.example.test", 443,
|
||||
["10.5.5.5", "10.5.5.6"]),
|
||||
resolved("alias.example.test", 443, ["10.5.5.7"])]))
|
||||
payload = gateway.parse(snap, "gateway")
|
||||
rows = [row for row in payload["connections"] if row["in"]["port"] in (4443, 8443, 8444)]
|
||||
assert len(rows) == 3
|
||||
assert [row["connections"] for row in rows] == [2, 2, 2], (
|
||||
"one endpoint's two sockets, repeated on all three lines")
|
||||
assert [row["shared"] for row in rows] == [True, True, True]
|
||||
assert gateway.distinct_endpoint_counts(payload["connections"]) == [2], (
|
||||
"and one figure for any total, however many names bridged their way to it")
|
||||
|
||||
|
||||
def test_a_resolver_entry_that_publishes_nothing_address_like_places_nothing() -> None:
|
||||
"""A malformed answer is a name with no address, not an address that matches by luck.
|
||||
|
||||
The agent publishes strings it got from `getaddrinfo`, and this module has no reason to believe
|
||||
them: an entry whose `addresses` are not addresses yields no endpoint key, so the route keeps its
|
||||
dash and its missing gauge exactly as an `nxdomain` does. The alternative — comparing the junk
|
||||
text against socket addresses — is how a typo in one file could credit a socket to a route.
|
||||
"""
|
||||
snap = snapshot(
|
||||
nginx=nginx_section([nginx_server([4443], ["odd.example.test"],
|
||||
[proxy_pass(("odd.example.test", 443))])]),
|
||||
sockets=sockets_section(established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 443))]),
|
||||
resolver=resolver_section([
|
||||
resolved("odd.example.test", 443, ["localhost", "10.5.5.5.6", ""])]))
|
||||
row = routes_to(gateway.parse(snap, "gateway"), "odd.example.test")[0]
|
||||
assert row["out"]["resolved"] == ["localhost", "10.5.5.5.6"], (
|
||||
"the agent's bytes verbatim, minus the empty string — the payload is not where this is fixed")
|
||||
assert row["connections"] is None and row["metric"] is None, (
|
||||
"and not one of them is an address to match a socket with")
|
||||
|
||||
|
||||
def test_a_line_with_no_endpoint_contributes_nothing_to_a_total() -> None:
|
||||
"""`distinct_endpoint_counts` skips the unplaced line instead of offering it as a zero.
|
||||
|
||||
The caller turns "no figures at all" into `null` (`api._live_total`), which is the arm an
|
||||
ingress of nothing-but-unresolved-names lands in. A `null` inside the list instead would make
|
||||
every total on the page `null` because one route has a name nobody could place — and rule 2's
|
||||
complaint about that shape is precisely that it invents a fault in the healthy routes beside it.
|
||||
"""
|
||||
snap = snapshot(
|
||||
nginx=nginx_section([nginx_server([4443], ["up.example.test"],
|
||||
[proxy_pass(("up.example.test", 443))]),
|
||||
nginx_server([8443], ["gone.example.test"],
|
||||
[proxy_pass(("gone.example.test", 443))])]),
|
||||
sockets=sockets_section(established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 443))]),
|
||||
resolver=resolver_section([resolved("up.example.test", 443, ["10.5.5.5"]),
|
||||
resolved("gone.example.test", 443, [], status="nxdomain")]))
|
||||
payload = gateway.parse(snap, "gateway")
|
||||
placed, unplaced = (routes_to(payload, "up.example.test")[0],
|
||||
routes_to(payload, "gone.example.test")[0])
|
||||
assert (placed["connections"], unplaced["connections"]) == (1, None)
|
||||
assert gateway.distinct_endpoint_counts(payload["connections"]) == [1], (
|
||||
"the placed endpoint alone: no second entry, no None, no zero")
|
||||
assert gateway.distinct_endpoint_counts([unplaced]) == [], (
|
||||
"and a payload of nothing-but-dashes is no figures at all, which the API renders as null")
|
||||
|
||||
|
||||
def test_an_unresolved_name_publishes_no_metric_and_no_sample() -> None:
|
||||
"""`nxdomain`, an error, and no entry at all are one answer here: drawn, never counted.
|
||||
|
||||
Three arms in one snapshot — a name the resolver placed, a name it looked up and got nothing
|
||||
for, and a name it never mentioned — so the difference between "no record" and "nobody asked"
|
||||
stays visible in the resolver block while the **route rows** behave identically: `connections`
|
||||
`null`, `metric` `null`, no sample, and never a `0` that reads as an idle route.
|
||||
"""
|
||||
snap = snapshot(
|
||||
nginx=nginx_section([
|
||||
nginx_server([4443], ["up.example.test"], [proxy_pass(("up.example.test", 443))]),
|
||||
nginx_server([8443], ["gone.example.test"], [proxy_pass(("gone.example.test", 443))]),
|
||||
nginx_server([9443], ["silent.example.test"],
|
||||
[proxy_pass(("silent.example.test", 443))])]),
|
||||
sockets=sockets_section(
|
||||
established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 443)),
|
||||
conn(("10.1.1.27", 40002), ("10.5.5.9", 443))]),
|
||||
resolver=resolver_section([
|
||||
resolved("up.example.test", 443, ["10.5.5.5"]),
|
||||
resolved("gone.example.test", 443, [], status="nxdomain")]))
|
||||
payload = gateway.parse(snap, "gateway")
|
||||
assert routes_to(payload, "up.example.test")[0]["connections"] == 1
|
||||
for name in ("gone.example.test", "silent.example.test"):
|
||||
row = routes_to(payload, name)[0]
|
||||
assert row["out"]["resolved"] is None, "never `[]`: unknown is one test, not a length"
|
||||
assert row["out"]["via"] is None and row["out"]["resolved_ts"] is None
|
||||
assert row["connections"] is None, "no count, and no 0 standing in for one"
|
||||
assert row["metric"] is None, "no series for a number that cannot exist"
|
||||
stored = [g.metric for g in gateway.samples(payload, 5.0)
|
||||
if g.metric.startswith("gateway.out_conns@")]
|
||||
assert stored == ["gateway.out_conns@up.example.test:443"]
|
||||
assert sum(row["connections"] or 0 for row in payload["connections"]) == 1, (
|
||||
"the socket to the unresolved target is credited to nothing at all")
|
||||
|
||||
|
||||
def test_a_resolver_section_that_did_not_answer_places_nothing(gateway_snapshot: Any) -> None:
|
||||
"""`available: false` and `available: null` are the agent's answers, and neither joins.
|
||||
|
||||
`false` is "there was nothing to resolve" (no proxy config was read) and `null` is "a pass that
|
||||
could not run"; both arrive with no usable entries, and the payload's `resolver.available` says
|
||||
which one it was rather than leaving a page to infer it from an empty list.
|
||||
"""
|
||||
for available in (False, None):
|
||||
snap = snapshot(
|
||||
nginx=nginx_section([nginx_server([4443], ["a.example.test"],
|
||||
[proxy_pass(("x.example.test", 443))])]),
|
||||
sockets=sockets_section(
|
||||
established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 443))]),
|
||||
resolver=resolver_section([resolved("x.example.test", 443, ["10.5.5.5"])],
|
||||
available=available, via=None))
|
||||
payload = gateway.parse(snap, "gateway")
|
||||
assert payload["resolver"]["available"] is available
|
||||
assert payload["resolver"]["entries"] == [], "nothing was indexed from a dead pass"
|
||||
assert routes_to(payload, "x.example.test")[0]["connections"] is None, (
|
||||
"an entry beside `available: null` is a stale answer, not a joinable address")
|
||||
|
||||
|
||||
def test_the_resolver_block_names_the_entries_the_index_was_built_from(
|
||||
gateway_snapshot: Any) -> None:
|
||||
"""The payload carries the document it was drawn from, so provenance is one read.
|
||||
|
||||
`entries` is every entry the section published — the index filters **within** it, so a page can
|
||||
still say *this name has no record* — and each is projected to the five keys the agent owns.
|
||||
`via` is the section's string. The `stub` arm keeps the same key set with nulls, which is what
|
||||
the parse/stub parity pin in `tests/test_api.py` compares.
|
||||
"""
|
||||
payload = gateway.parse(gateway_snapshot("resolved"), "gateway")
|
||||
block = payload["resolver"]
|
||||
assert set(block) == {"available", "via", "entries"}
|
||||
assert block["available"] is True and block["via"] == "libc"
|
||||
document = json.loads((FIXTURES / "gateway_resolver.json").read_text())
|
||||
assert [row["name"] for row in block["entries"]] == [row["name"] for row in document["entries"]]
|
||||
assert [row["port"] for row in block["entries"]] == [row["port"] for row in document["entries"]]
|
||||
assert all(set(row) == {"name", "port", "status", "addresses", "ts"}
|
||||
for row in block["entries"]), "no agent field reaches a page unreviewed"
|
||||
assert {row["status"] for row in block["entries"]} == {"ok", "literal"}, (
|
||||
"this capture resolves every name it was asked about, plus the two `127.0.0.1` targets it "
|
||||
"recognised as addresses without asking at all")
|
||||
|
||||
empty = gateway.stub("gateway")
|
||||
assert empty["resolver"] == {"available": None, "via": None, "entries": []}, (
|
||||
"a host that has not answered has not said resolution is unavailable")
|
||||
|
||||
|
||||
def test_the_dashboard_still_never_resolves(gateway_snapshot: Any) -> None:
|
||||
"""The spy pin, restated for phase 32 rather than relaxed by it.
|
||||
|
||||
`parse()` may **read** a resolver document and must not produce one, so the spy now runs over
|
||||
three paths: a synthetic snapshot whose every target is a name, the same with a resolver section
|
||||
full of answers, and the captured `resolved` ingress — then `samples()` over the result, since
|
||||
a gauge name is built from a host string and is the other place a resolver could creep in.
|
||||
|
||||
Why the split is the design and not a limitation: the agent resolves with `getaddrinfo` on the
|
||||
host that **opens** these connections (haproxy's `init-addr last,libc,none`), so its answer is
|
||||
the one that decides what `/proc` then says. The dashboard asking the same name through its own
|
||||
`resolv.conf` would answer a different question — a different server, a different view, and on
|
||||
this fleet a name that resolves back through this same router — and would then attribute a
|
||||
socket to a route on the strength of a lookup the router never made.
|
||||
"""
|
||||
calls: list[tuple[Any, ...]] = []
|
||||
|
||||
def spy(*args: Any, **kwargs: Any) -> Any: # pragma: no cover - must never run
|
||||
calls.append((args, kwargs))
|
||||
raise AssertionError("the gateway join must never resolve a name")
|
||||
|
||||
named = snapshot(
|
||||
haproxy=haproxy_section([frontend("mc", [25565])],
|
||||
[backend("mc-be", [server("mc1", "minecraft.example.test",
|
||||
25565)])]),
|
||||
nginx=nginx_section([nginx_server([443], ["a.example.test"],
|
||||
[proxy_pass(("app.example.test", 8080))])]),
|
||||
sockets=sockets_section(listens=[listen(25565), listen(443)],
|
||||
established=[conn(("10.1.1.27", 443), ("203.0.113.5", 900)),
|
||||
conn(("10.1.1.27", 1), ("10.2.2.2", 8080))]))
|
||||
answered = snapshot(
|
||||
haproxy=named["haproxy"], nginx=named["nginx"], sockets=named["sockets"],
|
||||
resolver=resolver_section([resolved("minecraft.example.test", 25565, ["10.9.9.9"]),
|
||||
resolved("app.example.test", 8080, ["10.2.2.2"])]))
|
||||
|
||||
original = socket_module.getaddrinfo
|
||||
socket_module.getaddrinfo = spy
|
||||
try:
|
||||
for document in (named, answered, gateway_snapshot("resolved")):
|
||||
payload = gateway.parse(document, "gateway")
|
||||
gateway.samples(payload, 5.0)
|
||||
gateway.warnings(gateway._section(document, "haproxy"),
|
||||
gateway._section(document, "nginx"))
|
||||
finally:
|
||||
socket_module.getaddrinfo = original
|
||||
assert calls == []
|
||||
|
||||
|
||||
def test_a_snapshot_without_a_resolver_section_is_phase_30_verbatim(
|
||||
gateway_snapshot: Any) -> None:
|
||||
"""A 1.2.0 agent, or a 1.3.0 one run with `--no-resolve`, is **not** a changed page.
|
||||
|
||||
The topology, the sections, the warnings and the two loopback counts are byte-for-byte what
|
||||
phase 30 produced; what differs is three `null`s on every `out` and an empty `resolver` block,
|
||||
plus the routes phase 32 makes countable. `supported` stays `true` — a missing resolver is a
|
||||
missing measurement, not an unsupported agent, and a fleet running two versions (which this one
|
||||
did until task 05) must not read half-red.
|
||||
"""
|
||||
without = gateway.parse(gateway_snapshot("full"), "gateway")
|
||||
with_resolver = gateway.parse(gateway_snapshot("resolved"), "gateway")
|
||||
assert without["supported"] is True and with_resolver["supported"] is True
|
||||
assert without["resolver"] == {"available": None, "via": None, "entries": []}
|
||||
assert all(row["out"]["resolved"] is None and row["out"]["via"] is None
|
||||
and row["out"]["resolved_ts"] is None for row in without["connections"])
|
||||
|
||||
def shape(payload: Mapping[str, Any]) -> Any:
|
||||
"""Everything except the counts, which is the one thing the section may move."""
|
||||
return {
|
||||
"routes": [row["id"] for row in payload["connections"]],
|
||||
"out": [(row["out"]["host"], row["out"]["port"], row["out"]["name"],
|
||||
row["out"]["kind"], tuple(row["out"]["selectors"]), row["out"]["address"])
|
||||
for row in payload["connections"]],
|
||||
"trunks": [(row["port"], row["source"], tuple(row["domains"]))
|
||||
for row in payload["listeners"]],
|
||||
"sections": payload["sections"],
|
||||
"why": payload["why"],
|
||||
"warnings": payload["warnings"],
|
||||
"sockets": payload["sockets"],
|
||||
}
|
||||
|
||||
assert shape(without) == shape(with_resolver), (
|
||||
"the resolver section counts through an address; it moves no route, trunk or warning")
|
||||
counted = {row["id"] for row in without["connections"] if row["connections"] is not None}
|
||||
assert counted == {"nginx:443->127.0.0.1:4443", "nginx:8443->127.0.0.1:8081"}, (
|
||||
"phase 30 counted exactly the two endpoints the config wrote as addresses")
|
||||
|
||||
|
||||
def test_resolution_raises_the_counted_route_count_from_two_to_every_name(
|
||||
gateway_snapshot: Any) -> None:
|
||||
"""The change, as one printed number rather than as an implied one.
|
||||
|
||||
Phase 30 could count the two routes whose target the configs wrote as an address. The recorded
|
||||
resolver document places the other thirty-one, so against the captures every route on the
|
||||
ingress now carries a figure — which is what the operator's "the only animated line is `:443 ->
|
||||
127.0.0.1:4443`" was really asking for.
|
||||
"""
|
||||
before = gateway.parse(gateway_snapshot("full"), "gateway")
|
||||
after = gateway.parse(gateway_snapshot("resolved"), "gateway")
|
||||
counted_before = [row for row in before["connections"] if row["connections"] is not None]
|
||||
counted_after = [row for row in after["connections"] if row["connections"] is not None]
|
||||
written_as_name = [row for row in after["connections"] if row["out"]["address"] is not True]
|
||||
unplaced = sorted(row["out"]["host"] for row in written_as_name
|
||||
if row["connections"] is None)
|
||||
report = (f"/gateway counted routes: {len(counted_before)} "
|
||||
f"(phase 30, addresses only) → {len(counted_after)} of {len(after['connections'])}; "
|
||||
f"the resolver placed {len(written_as_name)} name-written routes")
|
||||
print(report)
|
||||
assert len(counted_before) == 2, "phase 30's figure, so the delta is measurable"
|
||||
assert not unplaced, f"{report} — unplaced: {unplaced}"
|
||||
assert len(counted_after) == len(after["connections"]), (
|
||||
f"{report} — a route the document places must carry a count")
|
||||
|
||||
|
||||
def test_two_ports_on_one_host_are_two_gauges_even_when_both_names_resolve() -> None:
|
||||
"""`minecraft:25565` and `minecraft:25566` stay two series — the port earns its place in the name.
|
||||
|
||||
This is why phase 29 put the port in `gateway.out_conns@<host>:<port>` in the first place, and
|
||||
resolution does not remove the reason: this fleet's two haproxy Minecraft backends name the *same*
|
||||
domain on two ports, libc answers one address for both, and a gauge keyed on the name alone would
|
||||
have the second poll overwrite the first in silence. Two endpoints, two counters, two series — and
|
||||
the fact that they share an address is invisible in the names and visible in the traffic.
|
||||
"""
|
||||
snap = snapshot(
|
||||
haproxy=haproxy_section(
|
||||
[frontend("mc", [25565], default="mc-a"), frontend("mc2", [25566], default="mc-b")],
|
||||
[backend("mc-a", [server("mc1", "minecraft.example.test", 25565)]),
|
||||
backend("mc-b", [server("mc2", "minecraft.example.test", 25566)])]),
|
||||
sockets=sockets_section(established=[
|
||||
conn(("10.1.1.27", 40000), ("10.9.9.9", 25565)),
|
||||
conn(("10.1.1.27", 40002), ("10.9.9.9", 25565)),
|
||||
conn(("10.1.1.27", 40004), ("10.9.9.9", 25566))]),
|
||||
resolver=resolver_section([resolved("minecraft.example.test", 25565, ["10.9.9.9"]),
|
||||
resolved("minecraft.example.test", 25566, ["10.9.9.9"])]))
|
||||
payload = gateway.parse(snap, "gateway")
|
||||
by_port = {row["in"]["port"]: row for row in payload["connections"]}
|
||||
assert by_port[25565]["metric"] == "gateway.out_conns@minecraft.example.test:25565"
|
||||
assert by_port[25566]["metric"] == "gateway.out_conns@minecraft.example.test:25566"
|
||||
assert [by_port[25565]["connections"], by_port[25566]["connections"]] == [2, 1], (
|
||||
"two endpoints, two counts — one address does not merge two ports")
|
||||
assert by_port[25565]["shared"] is False and by_port[25566]["shared"] is False, (
|
||||
"sharing an *address* is not sharing an *endpoint*, which is what `shared` means")
|
||||
stored = gateway.samples(payload)
|
||||
assert [(s.metric, s.value) for s in stored if "out_conns" in s.metric] == [
|
||||
("gateway.out_conns@minecraft.example.test:25565", 2),
|
||||
("gateway.out_conns@minecraft.example.test:25566", 1)], (
|
||||
"neither gauge overwrites the other on the way to the store")
|
||||
assert gateway.distinct_endpoint_counts(payload["connections"]) == [2, 1], (
|
||||
"and the total books both, because the kernel has both")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# the map: which maps route, and which are just header words
|
||||
# --------------------------------------------------------------------------- #
|
||||
@@ -1336,9 +2051,9 @@ def test_the_captured_configs_produce_no_warnings(gateway_snapshot: Any) -> None
|
||||
assert used == {"targetBackend"} and used <= defined, (
|
||||
"the one variable a `proxy_pass` passes IS defined by a map")
|
||||
referenced = [block for block in nginx["maps"] if block["variable"].lstrip("$") in used]
|
||||
assert len(referenced) == 1 and len(referenced[0]["entries"]) == 17
|
||||
assert len(referenced) == 1 and len(referenced[0]["entries"]) == 18
|
||||
assert all(entry["target"] for entry in referenced[0]["entries"]), (
|
||||
"all 17 stream routes name an address, `default` included")
|
||||
"all 18 stream routes name an address, `default` included")
|
||||
unreferenced = [block for block in nginx["maps"] if block not in referenced]
|
||||
assert any(entry["target"] is None for block in unreferenced for entry in block["entries"]), (
|
||||
"the capture must still hold null-target entries on the UNREFERENCED map, or this pin "
|
||||
|
||||
@@ -21,6 +21,7 @@ a unit test.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import json
|
||||
import os
|
||||
import sqlite3
|
||||
@@ -44,6 +45,31 @@ from ups_contract import assert_ups_hero_contract
|
||||
pytestmark = pytest.mark.live
|
||||
|
||||
|
||||
def endpoint_identity(out: Any) -> tuple[Any, ...] | None:
|
||||
"""One endpoint in the kernel's currency, derived **here** rather than by the source.
|
||||
|
||||
An address endpoint is its own address; a name endpoint is the addresses the agent's resolver
|
||||
published for it (`out.resolved`), normalised so `::ffff:10.1.2.64` and `10.1.2.64` are one
|
||||
key. `None` is "nothing can name this far end", which is counted nowhere. Phase 32 made this
|
||||
helper necessary: a live router can have two route lines reaching one address through two
|
||||
different names, and a total keyed on the config's text would book it twice.
|
||||
"""
|
||||
port = out.get("port")
|
||||
if not isinstance(port, int):
|
||||
return None
|
||||
spellings = ([out["host"]] if out.get("address") is True
|
||||
else list(out.get("resolved") or []))
|
||||
keys: list[str] = []
|
||||
for spelling in spellings:
|
||||
try:
|
||||
address = ipaddress.ip_address(str(spelling))
|
||||
except ValueError:
|
||||
continue
|
||||
mapped = getattr(address, "ipv4_mapped", None)
|
||||
keys.append(str(mapped if mapped is not None else address))
|
||||
return (min(keys), port) if keys else None
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def settings() -> Settings:
|
||||
"""Live env, not the import-time SETTINGS singleton."""
|
||||
@@ -270,6 +296,11 @@ async def test_the_gateway_source_joins_the_live_ingress(settings: Settings,
|
||||
against the kernel rather than against a fixture. The totals are an inequality by design
|
||||
(`sum(listeners) + sum(endpoints) <= established`): a socket neither config file describes
|
||||
is counted nowhere, and on this router that is most of the table.
|
||||
|
||||
Phase 32 widened **which** routes can be counted — an endpoint the config wrote as an address,
|
||||
or one the gateway's own resolver published for the name it wrote — and the two assertions near
|
||||
the end are stated in those terms, with the endpoint identity derived here so the outbound total
|
||||
is not simply the source's own arithmetic re-added.
|
||||
"""
|
||||
target = next((item for item in settings.host_targets if item.name == settings.gateway_host),
|
||||
None)
|
||||
@@ -307,11 +338,14 @@ async def test_the_gateway_source_joins_the_live_ingress(settings: Settings,
|
||||
for route in payload["connections"]:
|
||||
assert route["in"]["port"] in configured, "a route hangs off no trunk the page draws"
|
||||
out = route["out"]
|
||||
assert (route["connections"] is None) is (out["address"] is not True), (
|
||||
f"{route['id']}: only an endpoint the config wrote as an address can be counted")
|
||||
if out["address"] is True and isinstance(out["port"], int):
|
||||
placed = out["address"] is True or bool(out.get("resolved"))
|
||||
assert (route["connections"] is None) is (not placed), (
|
||||
f"{route['id']}: only an endpoint whose address is known — written, or resolved by the "
|
||||
"gateway itself — can carry a count")
|
||||
key = endpoint_identity(out)
|
||||
if key is not None:
|
||||
joinable.add(str(route["metric"]))
|
||||
endpoints.setdefault((out["host"], out["port"]), route["connections"])
|
||||
endpoints.setdefault(key, route["connections"])
|
||||
else:
|
||||
assert route["metric"] is None, "a route nobody can count publishes no series"
|
||||
live_out = sum(value for value in endpoints.values() if isinstance(value, int))
|
||||
@@ -325,8 +359,61 @@ async def test_the_gateway_source_joins_the_live_ingress(settings: Settings,
|
||||
stored = gateway.samples(payload, 0.0)
|
||||
assert all(sample.source == gateway.SOURCE for sample in stored)
|
||||
assert {sample.metric for sample in stored if "out_conns" in sample.metric} <= joinable, (
|
||||
"only an address endpoint publishes an outbound gauge — a domain-keyed series is the one "
|
||||
"phase 30 stopped writing, because the figure it carried belonged to the port")
|
||||
"only an endpoint whose address the join knows publishes an outbound gauge — written as an "
|
||||
"address, or resolved by the gateway itself (phase 32); an unplaced name publishes nothing")
|
||||
|
||||
|
||||
async def test_a_resolved_target_on_the_live_ingress_is_never_a_dash(
|
||||
settings: Settings, agent_tls: dict[str, collectors.AgentTls]) -> None:
|
||||
"""Phase 32 task 03 against the real ingress: `ok` in the resolver means a number on the page.
|
||||
|
||||
The operator's complaint was that one line out of thirty-three moved, and the acceptance line for
|
||||
removing the dash is that it goes because the gateway published the address — not because a
|
||||
column was hidden. So every route the resolver placed carries a **finite** `connections` (`0`
|
||||
allowed: resolved and idle is a reading), a non-empty `out.resolved`, the agent's own `via`, the
|
||||
timestamp of the lookup, and the gauge that goes with it. Skips when the ingress runs an agent
|
||||
with no `resolver` section yet — which is every host until task 05 re-stages it.
|
||||
"""
|
||||
target = next((item for item in settings.host_targets if item.name == settings.gateway_host),
|
||||
None)
|
||||
if target is None:
|
||||
pytest.skip(f"GATEWAY_HOST={settings.gateway_host!r} is not in HOST_TARGETS")
|
||||
async with httpx.AsyncClient(timeout=settings.gateway_timeout) as client:
|
||||
try:
|
||||
snapshot = await collectors.fetch_snapshot(client, target,
|
||||
timeout=settings.gateway_timeout,
|
||||
tls=agent_tls.get(target.name))
|
||||
except collectors.UpstreamError as exc:
|
||||
pytest.skip(f"{target.name} did not answer: {exc}")
|
||||
section = snapshot.get("resolver")
|
||||
if not isinstance(section, dict):
|
||||
pytest.skip(f"{target.name} publishes no resolver section — agent 1.3.0 is task 05's "
|
||||
"re-stage; until then every name-written route is a dash by design")
|
||||
payload = gateway.parse(snapshot, target.name)
|
||||
if section.get("available") is not True:
|
||||
pytest.skip(f"{target.name}'s resolver did not answer "
|
||||
f"(available={section.get('available')!r}): nothing is placed, so nothing "
|
||||
"is expected to be counted")
|
||||
|
||||
placed = [route for route in payload["connections"] if route["out"].get("resolved")]
|
||||
assert placed, (f"{target.name} published {len(section.get('entries') or [])} resolver "
|
||||
"entries and the join placed none of them")
|
||||
for route in placed:
|
||||
out = route["out"]
|
||||
# The count lives on the **route** (`connections[]`), not on its `out` endpoint — the same
|
||||
# shape `test_the_gateway_source_joins_the_live_ingress` reads. This path only ever ran once
|
||||
# agent 1.3.0 was installed (before that the whole test skipped for want of a `resolver`
|
||||
# section), which is how a wrong key survived task 03: a KeyError is not a dash.
|
||||
assert isinstance(route["connections"], int) and not isinstance(route["connections"], bool), (
|
||||
f"{route['id']}: resolved to {out['resolved']} and the line is still a dash")
|
||||
assert out["via"] == section.get("via") == "libc", (
|
||||
"the vantage is the agent's string, copied, never one this module invented")
|
||||
assert isinstance(out["resolved_ts"], int | float), (
|
||||
f"{route['id']}: a count obtained through a lookup must say when the lookup happened")
|
||||
assert route["metric"] is not None, "a counted route always has its gauge"
|
||||
stored = {sample.metric for sample in gateway.samples(payload, 0.0)}
|
||||
assert {route["metric"] for route in placed} <= stored, (
|
||||
"a resolved route's count must reach the store, or its chart stays empty forever")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("engine", ["gufo", "vllm"])
|
||||
|
||||
@@ -297,7 +297,7 @@ dev = [
|
||||
{ name = "debugpy", specifier = ">=1.8.22" },
|
||||
{ name = "pyright", specifier = ">=1.1.414" },
|
||||
{ name = "pytest", specifier = ">=9.1.1" },
|
||||
{ name = "pytest-asyncio", specifier = ">=1.4.0" },
|
||||
{ name = "pytest-asyncio", specifier = ">=1.4.1" },
|
||||
{ name = "pytest-cov", specifier = ">=7.1.0" },
|
||||
{ name = "ruff", specifier = ">=0.17.1" },
|
||||
]
|
||||
@@ -626,14 +626,14 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "pytest-asyncio"
|
||||
version = "1.4.0"
|
||||
version = "1.4.1"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "pytest" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/43/7c/d36d04db312ecf4298932ef77e6e4a9e8ad017906e24e34f0b0c361a2473/pytest_asyncio-1.4.0.tar.gz", hash = "sha256:c6c0d2259945122819f171a32ecea2c349ead889ee28176caaf492143424be42", size = 58514, upload-time = "2026-05-26T09:56:04.083Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/f8/9a/887535e477847f5159377ab35303923644840b13c0ca03c0a467f84d4680/pytest_asyncio-1.4.1.tar.gz", hash = "sha256:e8266eb094e5ee9ffdb2068c9836f33e209ea77a970eb846933d416b5602669b", size = 145237, upload-time = "2026-10-25T21:05:37.135Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/03/e2/08a497ef684b88559c9cc5f4ad53a37e7b99e727094a86d6ea32536d5d3c/pytest_asyncio-1.4.0-py3-none-any.whl", hash = "sha256:933ca923a23075a87fb7070c0ec272a6848489824d887c85c812670932835aa1", size = 16930, upload-time = "2026-05-26T09:56:02.576Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/46/d7/cd9e2708afdbf059658adaf08bdb0792ca52489871051d6ce7d498b51277/pytest_asyncio-1.4.1-py3-none-any.whl", hash = "sha256:507afa78ec77c6661164ef1a2368bc7820383fd1a14b2a7f813de8b5a92949cc", size = 17061, upload-time = "2026-10-25T21:05:35.473Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -712,14 +712,14 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "starlette"
|
||||
version = "1.9.0"
|
||||
version = "1.10.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
dependencies = [
|
||||
{ name = "anyio" },
|
||||
]
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/ad/eb/4e197bf26b023abafc5e9043ec8710d7de0fcb032d0969045236d72aa0b1/starlette-1.9.0.tar.gz", hash = "sha256:3d931de246ed870e35ce605974440816ef895e592f255d1396cfdf191d4c4318", size = 2732466, upload-time = "2026-10-18T15:50:33.189Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/48/bb/3d8757f21e9a53d9035c9eb039a777047139a16bfdbedfc7a58241c3d914/starlette-1.10.0.tar.gz", hash = "sha256:b8399098d4c1e3522734f7235dc79b1fc967f96cf139c81e0fa30479e0685f9b", size = 2734356, upload-time = "2026-10-24T15:04:12.342Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/f3/93/181df62699d9b5aca1d87e5cfda131dac87e3cdaebf668662f7206ca3470/starlette-1.9.0-py3-none-any.whl", hash = "sha256:fd3f159936e882d06c1108f84688f7cda2cf85fc6a592a50edbd175e882c2ddc", size = 80335, upload-time = "2026-10-18T15:50:31.526Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/88/2e/947cfe02def9371c07cd5bf5b6b61c829e9aecb9904db02a66cfafbc1eec/starlette-1.10.0-py3-none-any.whl", hash = "sha256:478c0ab3b4ced6e82719849c118fa406f0278a3499c1a898cd15cd31bdc38ce4", size = 80571, upload-time = "2026-10-24T15:04:10.371Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
Reference in New Issue
Block a user