diff --git a/.agents/phases/todo/29_gateway_dashboard_page/00_phase.md b/.agents/phases/complete/29_gateway_dashboard_page/00_phase.md similarity index 100% rename from .agents/phases/todo/29_gateway_dashboard_page/00_phase.md rename to .agents/phases/complete/29_gateway_dashboard_page/00_phase.md diff --git a/.agents/phases/todo/29_gateway_dashboard_page/01_config_and_source.md b/.agents/phases/complete/29_gateway_dashboard_page/01_config_and_source.md similarity index 100% rename from .agents/phases/todo/29_gateway_dashboard_page/01_config_and_source.md rename to .agents/phases/complete/29_gateway_dashboard_page/01_config_and_source.md diff --git a/.agents/phases/todo/29_gateway_dashboard_page/02_wiring.md b/.agents/phases/complete/29_gateway_dashboard_page/02_wiring.md similarity index 100% rename from .agents/phases/todo/29_gateway_dashboard_page/02_wiring.md rename to .agents/phases/complete/29_gateway_dashboard_page/02_wiring.md diff --git a/.agents/phases/todo/29_gateway_dashboard_page/03_api_routes.md b/.agents/phases/complete/29_gateway_dashboard_page/03_api_routes.md similarity index 100% rename from .agents/phases/todo/29_gateway_dashboard_page/03_api_routes.md rename to .agents/phases/complete/29_gateway_dashboard_page/03_api_routes.md diff --git a/.agents/phases/todo/29_gateway_dashboard_page/04_page.md b/.agents/phases/complete/29_gateway_dashboard_page/04_page.md similarity index 100% rename from .agents/phases/todo/29_gateway_dashboard_page/04_page.md rename to .agents/phases/complete/29_gateway_dashboard_page/04_page.md diff --git a/.agents/phases/todo/29_gateway_dashboard_page/05_nav_and_summary.md b/.agents/phases/complete/29_gateway_dashboard_page/05_nav_and_summary.md similarity index 100% rename from .agents/phases/todo/29_gateway_dashboard_page/05_nav_and_summary.md rename to .agents/phases/complete/29_gateway_dashboard_page/05_nav_and_summary.md diff --git a/.agents/phases/todo/29_gateway_dashboard_page/06_docs_and_verify.md b/.agents/phases/complete/29_gateway_dashboard_page/06_docs_and_verify.md similarity index 100% rename from .agents/phases/todo/29_gateway_dashboard_page/06_docs_and_verify.md rename to .agents/phases/complete/29_gateway_dashboard_page/06_docs_and_verify.md diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__00_phase.a1.err b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__00_phase.a1.md b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__00_phase.a1.md new file mode 100644 index 0000000..c1152fa --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__00_phase.a1.md @@ -0,0 +1,19 @@ +Phase 29 completion pass is done. Final state: + +**Verification (all re-run this pass, no task files touched)** +- `uv run pytest` → **1746 passed, 16 skipped**; `uv run pytest --cov` → **99.05 %** (gate 94 %); `bash .agents/validate.sh` → **rc=0** (ruff clean, pyright `0 errors, 0 warnings`, `node --check` clean); `uv run pytest -m live --live` → **12 passed**. +- Live, read-only: dev server **10/10 sources `ok`** (incl. `gateway`); `data/logs/29-task06-live.cjs` → **49 PASS / 0 FAIL** with per-port inbound counts reconciled against a direct `/proc/net/tcp{,6}` read (divergence 0); `data/logs/verify-29-task06-predates.py` → 0 failed (1.1.0 arm renders `WHY_PREDATES`, no pipes, every tally `n/a`, job still `ok`); phase-27 mobile audit re-run over all nine served pages → **371 PASS / 0 FAIL / 0 FINDING** (`data/logs/29-final-mobile-audit.out`). +- Criteria: (1) unconfigured `GATEWAY_HOST` → `404 {configured:false}` with dial spies + `#gate-empty` sentence ✅; (2) 1.1.0/`--no-proxy` → `supported:false`, no `0`s, never red ✅; (3) live gateway 1.2.0 → pipes animate, totals reconcile ✅; (4) no `severity`/verdict colour anywhere, validate green ✅; (5) docs updated + gap 13 logged ✅; (6) the five existing dashboards, jobs and `/api/summary` unchanged (live control leg) ✅. + +**Docs (consistency pass, incl. pre-existing drift)** +- `README.md`: architecture diagram now lists `sources/gateway.py`, the 30 s gateway job and the third `/snapshot` reader; `GATEWAY_HOST` row says *a name, never a URL*; the `POLL_MS_*` row states `/gateway`'s fixed 15 s browser poll; "nine pollers" → per-source cadences; phase-28 "until phase 29" wording → current state; new Testing paragraph for `tests/test_gateway_source.py` (join pinned from the captures, reconciliation identity, `getaddrinfo` spy). +- `AGENTS.md`: phase-28 stop-point wording replaced with the shipped phase-29 state; suite/coverage figures → 1746 / 16 skipped / 99.08 %→99.05 % band and `~99 %`; window-chips paragraph records that `/gateway` has **no** chips (fixed 24 h); new testing-policy bullet for the join; stale page counts fixed (seven → eight tabbed pages / `MARKUP_PAGES`, clock-order and badge-count transcripts annotated); gap 12 now records the phase-29 nine-page audit re-run. +- `tests/test_api.py`: comment/docstring-only fixes for the "seven pages" wording that phase 29 made eight. + +**Deviations / notes** +- Nothing implemented: all six task files were already in `complete/`, so this run was the final verification + docs pass; no git index or phase file was touched. +- Coverage wobbles 99.05–99.08 % between runs on a pre-existing timing-dependent line (`collectors.py` `/slots`-alias `except`/`break`), not a phase-29 path (`sources/gateway.py` is 100 %). +- `uv pip list --outdated` still lists upgrades; dependencies did not move in this phase, so they were left alone rather than upgraded mid-verification. +- Historical browser-transcript counts (phase 18/20/25 script runs) were left as recorded history and annotated instead of rewritten. + +**Next phase:** none — `.agents/phases/todo/` is empty after 29; further work needs a new phase authored first. diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__00_phase.a1.md.status b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__00_phase.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__00_phase.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__00_phase.a1.validate b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__00_phase.a1.validate new file mode 100644 index 0000000..a1f5d79 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__00_phase.a1.validate @@ -0,0 +1,74 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 4%] +........................................................................ [ 8%] +........................................................................ [ 12%] +........................................................................ [ 16%] +........................................................................ [ 20%] +........................................................................ [ 24%] +........................................................................ [ 28%] +........................................................s............... [ 32%] +........................................................................ [ 36%] +........................................................................ [ 40%] +........................................................................ [ 44%] +........................................................................ [ 49%] +........................................................................ [ 53%] +........................................................................ [ 57%] +........................................................................ [ 61%] +........................................................................ [ 65%] +........................................................................ [ 69%] +............................................................ssssssssssss [ 73%] +........................................................................ [ 77%] +........................................................................ [ 81%] +........................................................................ [ 85%] +........................................................................ [ 89%] +........................................................................ [ 93%] +................................................sss..................... [ 98%] +.................................. [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 182 0 100% +src/dashboards/api.py 491 0 100% +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 216 5 98% 323-324, 328-329, 346 +src/dashboards/config.py 126 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 321 0 100% +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 3894 37 99% +Required test coverage of 94% reached. Total coverage: 99.05% +1746 passed, 16 skipped, 1 warning in 167.20s (0:02:47) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == +advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose): + certifi 2026.10.19 2026.10.23 wheel +coverage 7.16.2 7.17.0 wheel +fastapi 0.153.0 0.161.1 wheel +pycparser 3.11 3.12 wheel +ruff 0.17.1 0.17.2 wheel diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__01_config_and_source.a1.err b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__01_config_and_source.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__01_config_and_source.a1.md b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__01_config_and_source.a1.md new file mode 100644 index 0000000..d5ce846 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__01_config_and_source.a1.md @@ -0,0 +1,15 @@ +**Implemented** +- `config.py`: `gateway_host` (`GATEWAY_HOST`, default `gateway`), `gateway_interval` (30 s), `gateway_timeout` (10 s), `gateway_enabled` (name must be a `HOST_TARGETS` entry, `nut_enabled` shape) and it now counts in `any_source_enabled`. +- New `src/dashboards/sources/gateway.py`: `parse(snapshot, host)` joins agent 1.2.0's `haproxy`/`nginx`/`sockets` into inbound pipes (per listen port: domains, live count, distinct clients) and outbound pipes (haproxy `server`, concrete `proxy_pass`, `$var`→`map` fan-out), plus `unmatched` tallies, `samples()` gauges (`gateway.in_conns@`, `gateway.out_conns@:`) and `parse_gateway_snapshot() -> Result`. Gauges only: no estimator/`Δ` window, no `severity`, `null` (never `0`) whenever the socket table or the join cannot answer; `supported:false` (1.1.0/`--no-proxy`) + per-section `why` + verbatim `section_errors`. +- Join never resolves a name: `socket.getaddrinfo` spy pins zero calls; exact (ip,port) match `by_port:false`, name matched on port only when that port has one configured host (`by_port:true`), several names on one port → nobody chosen (`connections:null`, sockets in `unmatched.outbound`, one `unmatched.groups` entry naming port/count/candidates). `join_sockets` attributes each socket once, so inbound+outbound+unmatched == the socket table. +- Tests: `tests/test_gateway_source.py` (35, incl. a fixture built by running the **real agent `Collector`** over the captured configs via new `gateway_snapshot` conftest fixture), 3 new `tests/test_config.py` cases, and a read-only `--live` test against the real gateway agent. Probe: `data/logs/verify-29-task01.py` → `29-task01-live.out`. +- Docs: `.env.example` + `.env.dev.example` gateway block, README Configuration rows, AGENTS.md (config bullet, new `sources/gateway.py` responsibility bullet, metric-naming/gauge family), `store.py` family docstring (`disks`/`gateway` added). + +**Results** — `uv run pytest`: 1691 passed, 16 skipped; `uv run pytest --cov`: 99.05% total, `sources/gateway.py` and `config.py` 100%; `uv run ruff check src tests agent` + `uv run pyright`: clean; `bash .agents/validate.sh`: green; `uv run pytest -m live --live`: 12 passed; dev server restarted, all nine sources `ok`. Live join: 10 inbound / 31 outbound / 17 map entries, partition `19 + 1 + 8 = 28 == established` OK, zero `severity` keys. + +**Deviations (documented in code + AGENTS.md)** +- Outbound series carry the port (`@host:port`, vs the phase's host-only sketch): this gateway's two haproxy Minecraft backends share `minecraft.reeselink.com` on 25565/25566, and one name would `INSERT OR REPLACE` one pipe's gauge over the other in silence (rule 2). +- The shared-port fan-out (16 `*.reeselink.com:443` map entries) yields `null` counts + `unmatched.groups` rather than a guessed or repeated attribution; `unmatched.other` is defined as live listeners no config claims (phase 28's "unclaimed socket"). +- No live infra changes were made (read-only dial only). Note: `NOTES.md` is empty in the working tree (pre-existing, not this task); its two items are already shipped (phase 24 mTLS, phase 20 task 04 admin links), so I left it as-is. + +**Next pending task:** `.agents/phases/todo/29_gateway_dashboard_page/02_wiring.md` (the `gateway` poller job, alerts rule, `/api/summary` block, `/api/config` entry). diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__01_config_and_source.a1.md.status b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__01_config_and_source.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__01_config_and_source.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__01_config_and_source.a1.validate b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__01_config_and_source.a1.validate new file mode 100644 index 0000000..35e2d6a --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__01_config_and_source.a1.validate @@ -0,0 +1,72 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 4%] +........................................................................ [ 8%] +........................................................................ [ 12%] +........................................................................ [ 16%] +........................................................................ [ 21%] +........................................................................ [ 25%] +........................................................................ [ 29%] +.......s................................................................ [ 33%] +........................................................................ [ 37%] +........................................................................ [ 42%] +........................................................................ [ 46%] +........................................................................ [ 50%] +........................................................................ [ 54%] +........................................................................ [ 59%] +........................................................................ [ 63%] +........................................................................ [ 67%] +........................................................................ [ 71%] +.....ssssssssssss....................................................... [ 75%] +........................................................................ [ 80%] +........................................................................ [ 84%] +........................................................................ [ 88%] +........................................................................ [ 92%] +.................................................................sss.... [ 97%] +................................................... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 176 0 100% +src/dashboards/api.py 440 0 100% +src/dashboards/app.py 186 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 206 4 98% 304, 308-309, 326 +src/dashboards/config.py 126 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 307 0 100% +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 3809 36 99% +Required test coverage of 94% reached. Total coverage: 99.05% +1691 passed, 16 skipped, 1 warning in 163.88s (0:02:43) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == +advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose): + coverage 7.16.2 7.17.0 wheel +fastapi 0.153.0 0.161.1 wheel +pycparser 3.11 3.12 wheel +ruff 0.17.1 0.17.2 wheel diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__02_wiring.a1.err b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__02_wiring.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__02_wiring.a1.md b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__02_wiring.a1.md new file mode 100644 index 0000000..8efe26b --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__02_wiring.a1.md @@ -0,0 +1,18 @@ +Everything is green. Final state: + +**Implementation** (`src/dashboards/`) +- `collectors.py`: `fetch_gateway_snapshot()` (third reader of the same agent `/snapshot`) + `gateway_job()` — **one** job (the `nut` shape), key/dashboard `gateway`, `label` = `GATEWAY_HOST`'s name, `GATEWAY_INTERVAL_SECONDS`/`GATEWAY_TIMEOUT_SECONDS`, **no** `RateEstimator`, `None` when the name isn't a target (invariant 3). +- `app.build_hub`: adds that job between `disks` and `llm` on the **same** `AgentTls` entry as `host:`/`disks:`, so one revoke fails all three without a dial. +- `alerts.py`: `HREF["/gateway"]`, staleness in the gateway's own 30 s interval, `AGENT_ECHO_DASHBOARDS = ("disks", "gateway")` with `_partner_host()` (gateway finds its host by **label**, not key) — dead agent = one line, wedged sections while the host answers = the only voice; **no** gateway threshold or payload argument. +- `api.py`: additive `/api/summary.gateway` (`configured, host, supported, unreachable, inbound, outbound, map_targets, unmatched, unmatched_listeners, partial, states`; `null` never `0` in the arms that didn't measure) and the `/api/config.gateway` row with redacted `display_url`. + +**Tests/lint/coverage** +- 19 new tests (5 alerts, 11 api, 3 collectors); full suite **1712 passed, 16 skipped**, coverage **99.07%** (was 99.05%), `alerts.py`/`api.py`/`app.py` back to **100%**; `ruff` + `pyright` clean; `bash .agents/validate.sh` exit 0 (dep list is the pre-existing advisory). +- Mutation-checked: breaking the gateway label lookup fails 3 pins, so the echo rule isn't vacuous. +- Live (`--live`, `.env.dev`): **12 passed**; dev server restarted — `/api/health` **ten sources ok** incl. `gateway`, `/api/summary.gateway` live (in 12 / out 4 / map 17 / unmatched 10+2 / partial), `/api/config.gateway` `{enabled, gateway, https://10.1.1.27:9120, 30, 10}`. + +**Docs**: AGENTS.md (poller job-of-its-own, collectors, alerts, api block, app, "ten sources" fleet fact) and README (alerts echo/interval rule, summary+config blocks). + +**Decisions**: `states` as the repo-standard list of health records (not a `{gateway: str}` map); added `partial` so a sum that omitted an unattributable port isn't read as the total; no `severity` anywhere. + +**Next pending task**: `29_gateway_dashboard_page/03_api_routes.md` — `/api/gateway/snapshot` + `/api/gateway/history`. diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__02_wiring.a1.md.status b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__02_wiring.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__02_wiring.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__02_wiring.a1.validate b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__02_wiring.a1.validate new file mode 100644 index 0000000..50cf107 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__02_wiring.a1.validate @@ -0,0 +1,72 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 4%] +........................................................................ [ 8%] +........................................................................ [ 12%] +........................................................................ [ 16%] +........................................................................ [ 20%] +........................................................................ [ 25%] +........................................................................ [ 29%] +.........................s.............................................. [ 33%] +........................................................................ [ 37%] +........................................................................ [ 41%] +........................................................................ [ 45%] +........................................................................ [ 50%] +........................................................................ [ 54%] +........................................................................ [ 58%] +........................................................................ [ 62%] +........................................................................ [ 66%] +........................................................................ [ 70%] +..........................ssssssssssss.................................. [ 75%] +........................................................................ [ 79%] +........................................................................ [ 83%] +........................................................................ [ 87%] +........................................................................ [ 91%] +........................................................................ [ 95%] +..............sss....................................................... [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 182 0 100% +src/dashboards/api.py 473 0 100% +src/dashboards/app.py 189 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 216 4 98% 324, 328-329, 346 +src/dashboards/config.py 126 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 307 0 100% +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 3861 36 99% +Required test coverage of 94% reached. Total coverage: 99.07% +1712 passed, 16 skipped, 1 warning in 164.73s (0:02:44) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == +advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose): + coverage 7.16.2 7.17.0 wheel +fastapi 0.153.0 0.161.1 wheel +pycparser 3.11 3.12 wheel +ruff 0.17.1 0.17.2 wheel diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__03_api_routes.a1.err b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__03_api_routes.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__03_api_routes.a1.md b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__03_api_routes.a1.md new file mode 100644 index 0000000..857a42d --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__03_api_routes.a1.md @@ -0,0 +1,13 @@ +**Task 03 complete — `/api/gateway/snapshot` + `/api/gateway/history`.** + +**Implementation** — `api.gateway_router()` (both GETs, included after `disks_router()` in `app.create_app`): `/snapshot` tests `settings.gateway_enabled` in its first line and raises `404 {"configured": false}` naming `GATEWAY_HOST` before any client/fetch/store read; answering → `sources/gateway.parse`'s payload verbatim + `states` (payload `ts` untouched); never-answered → new `gateway_source.stub(host, ts)` (`sources/gateway.py`: `_skeleton()` shared with `parse`, `WHY_NO_ANSWER`, `unreachable: true`, `supported: null`, every count `null`). `/history` goes through the one `_history_batch()` → `store.history_many()` door, `hosts=` defaulting to the ingress alone (no configured gate, since history dials nothing — the `/api/disks/history` rule). + +**Tests** — 16 new tests in `tests/test_api.py` (both unconfigured arms — unset name and a name that is not a target — spy-pinned on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`, with a control leg that names a real target; the stub's keys/counts/`states`; the payload↔`parse` key-set equality; inbound+outbound+unmatched == `sockets.established`; a recursive no-`severity`/no-`window_seconds` walk; history points re-read through `store.query`; unknown names → empty series; host default; family isolation in `/api/metrics?source=`; the two GETs, 405 on the rest, and the app's one upstream POST still `/api/ups/command`), plus the two paths added to `test_auth_api.py::PROTECTED_API`. Both controls were verified by mutation (gate moved after the fetch → refusal tests fail; batch source changed `gateway`→`host` → history tests fail). + +**Gate** — `uv run pytest --cov`: **1728 passed, 16 skipped**, coverage **99.05%** (`api.py`, `app.py`, `sources/gateway.py` each 100%); `ruff` and `pyright` clean; `bash .agents/validate.sh` **exit 0** (only the pre-existing dependency advisory; no deps moved); `-m live --live` 12 passed. Dev server restarted: all ten sources `ok`, and live `/api/gateway/snapshot` against agent 1.2.0 answered 6 frontends / 17 map targets / 31 outbound pipes with the identity reconciling exactly (39 == 39), while `/history` served `gateway.in_conns@443` and empty for a name that never measured. + +**Docs** — README and AGENTS `api.py` bullets now describe the pair (three arms, the gate-before-dial, the stub, the port-in-the-name gauges, no write), AGENTS records `gateway_source.stub()` and that all five `history` routes share the batched door. + +**Deviations** — none. `tests/card_contract.py` untouched (task 04's `COLUMNS` device); task file left in `todo/`, no git commands run. + +**Next task** — `.agents/phases/todo/29_gateway_dashboard_page/04_page.md`. diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__03_api_routes.a1.md.status b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__03_api_routes.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__03_api_routes.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__03_api_routes.a1.validate b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__03_api_routes.a1.validate new file mode 100644 index 0000000..0f78778 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__03_api_routes.a1.validate @@ -0,0 +1,73 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 4%] +........................................................................ [ 8%] +........................................................................ [ 12%] +........................................................................ [ 16%] +........................................................................ [ 20%] +........................................................................ [ 24%] +........................................................................ [ 28%] +.......................................s................................ [ 33%] +........................................................................ [ 37%] +........................................................................ [ 41%] +........................................................................ [ 45%] +........................................................................ [ 49%] +........................................................................ [ 53%] +........................................................................ [ 57%] +........................................................................ [ 61%] +........................................................................ [ 66%] +........................................................................ [ 70%] +..........................................ssssssssssss.................. [ 74%] +........................................................................ [ 78%] +........................................................................ [ 82%] +........................................................................ [ 86%] +........................................................................ [ 90%] +........................................................................ [ 94%] +..............................sss....................................... [ 99%] +................ [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 182 0 100% +src/dashboards/api.py 491 0 100% +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 216 5 98% 323-324, 328-329, 346 +src/dashboards/config.py 126 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 321 0 100% +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 3894 37 99% +Required test coverage of 94% reached. Total coverage: 99.05% +1728 passed, 16 skipped, 1 warning in 167.39s (0:02:47) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == +advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose): + coverage 7.16.2 7.17.0 wheel +fastapi 0.153.0 0.161.1 wheel +pycparser 3.11 3.12 wheel +ruff 0.17.1 0.17.2 wheel diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__04_page.a1.err b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__04_page.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__04_page.a1.md b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__04_page.a1.md new file mode 100644 index 0000000..984ba10 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__04_page.a1.md @@ -0,0 +1,10 @@ +**Task 04 complete — `/gateway` page with animated pipes. Everything green.** + +- **Page**: `web/pages/gateway.html` + `web/static/page-gateway.js` + a `/* ---- gateway pipes */` CSS section; route added in `app.py`. Four cards (`haproxy`, `nginx`, `map`, `unmatched`) keyed by each pipe's own `source`; hand-rolled SVG pipes (``, `vector-effect`), node label = `snapshot.host`, solo-side layouts for the stream map. +- **Honesty pins**: labels/counts/clients are `textContent` only; `null` → `n/a` (no `|| 0` on any count); a pipe animates **only** when `connections > 0`, `.still` otherwise; `prefers-reduced-motion` names **both** states; no `ok|warn|fail` token anywhere in the gateway CSS section; every plot carries `ariaLabel`; one `/api/gateway/history` request per card, plot built only while its `Charts` disclosure is open. +- **Degradation** (browser-proved, all from payload/route text): unconfigured → setup block naming `GATEWAY_HOST` + agent 1.2.0, zero red, all `n/a`; predates / never-answered → banner prints the payload's own `why`, nothing drawn, dimmed; silent socket table → config map stays, every count `n/a`. +- **Two defects found and fixed**: `LineChart._yExtent()` padded a flat series **below** its declared `yMin` (an idle pipe drew a `-1` connection axis) — shared fix, measured on `/system` `/disks` `/llm` `/ups` (no axis moved); and an all-unknown remainder/total rendered a card of `n/a`s or a `0` — now hidden/`n/a`. +- **Tests**: 14 new served-byte pins in `tests/test_api.py` (+3 existing audits widened to nine pages/twelve assets), `/gateway` added to `tests/test_auth_api.py`'s protected pages and served route table. +- **Browser (live fleet)**: `data/logs/29-task04-{pipes,degradation,unconfigured,charts-floor}.cjs` → all pass; 41 live pipes, `15 + 4 + 11 = 30 = established`; screenshots `29-task04-{gateway,charts-open,phone,predates,sockets_silent,unanswered,unconfigured}.png`, transcripts `.out`. +- **Gate**: `pytest` 1742 passed / 16 skipped, coverage 99.08 % (floor 94), `ruff` + `pyright` clean, `node --check` clean; dev server up with all ten sources `ok`. +- Docs (`AGENTS.md`/`README.md`) and the sixth masthead tab are task 05/06's deliverables and were deliberately left untouched; `PAGES` in `test_api.py` stays the five-tab table so task 05 moves it as one contract. diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__04_page.a1.md.status b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__04_page.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__04_page.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__04_page.a1.validate b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__04_page.a1.validate new file mode 100644 index 0000000..15ced53 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__04_page.a1.validate @@ -0,0 +1,73 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 4%] +........................................................................ [ 8%] +........................................................................ [ 12%] +........................................................................ [ 16%] +........................................................................ [ 20%] +........................................................................ [ 24%] +........................................................................ [ 28%] +....................................................s................... [ 32%] +........................................................................ [ 36%] +........................................................................ [ 40%] +........................................................................ [ 45%] +........................................................................ [ 49%] +........................................................................ [ 53%] +........................................................................ [ 57%] +........................................................................ [ 61%] +........................................................................ [ 65%] +........................................................................ [ 69%] +........................................................ssssssssssss.... [ 73%] +........................................................................ [ 77%] +........................................................................ [ 81%] +........................................................................ [ 86%] +........................................................................ [ 90%] +........................................................................ [ 94%] +............................................sss......................... [ 98%] +.............................. [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 182 0 100% +src/dashboards/api.py 491 0 100% +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 216 5 98% 323-324, 328-329, 346 +src/dashboards/config.py 126 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 321 0 100% +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 3894 37 99% +Required test coverage of 94% reached. Total coverage: 99.05% +1742 passed, 16 skipped, 1 warning in 161.90s (0:02:41) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == +advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose): + coverage 7.16.2 7.17.0 wheel +fastapi 0.153.0 0.161.1 wheel +pycparser 3.11 3.12 wheel +ruff 0.17.1 0.17.2 wheel diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__05_nav_and_summary.a1.err b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__05_nav_and_summary.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__05_nav_and_summary.a1.md b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__05_nav_and_summary.a1.md new file mode 100644 index 0000000..b4b66cd --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__05_nav_and_summary.a1.md @@ -0,0 +1,18 @@ +All gates green. Final state: + +**Done (task 05)** +- `Gateway` tab added to all **eight** signed-in pages (six dashboards + `/settings` + `/certs`), `aria-current` on `/gateway` only; tab audit rewritten as `nav hrefs == TAB_ORDER` (`PAGES` is now the six dashboards) with per-page `aria-current` asserted in both directions; `test_the_certs_page_is_a_page_and_not_a_sixth_dashboard` renamed `…_not_a_dashboard` now that `/gateway` *is* the sixth. +- Overview ingress line: `#fleet-gate` in the fleet panel, rendered by `renderGateway()` from `/api/summary`'s block (`GATEWAY_FIELDS` resolved against the live payload — the `SLOT_FIELDS` device), `.stale` when unreachable, **hidden when `configured: false`**. `/api/summary`'s `gateway` block gained an additive `href` derived from `alerts.HREF`, so the row links with data instead of a page-side route literal (same rule the alert rows keep). +- Pins: `SUMMARY_CONTRACT` now names every gateway key; `NAV_STATUS_BADGES` gained `gateway`; new `test_the_overview_line_reads_the_gateway_block_it_claims` and `test_the_gateway_block_links_to_the_page_the_alert_row_links_to`. +- Docs: AGENTS.md (six-page/nav/badge counts, the ingress line, the `href` derivation, the wrap rule, the nine-page audit) and README.md updated. + +**Deviations / defects found** +- Task assumption "the fifth `#nav-status` badge with **no JS change**" was false — `initChrome()`'s badge table is a literal, so app.js gained `{ key: 'gateway', label: 'gateway', href: '/gateway' }` (pinned). +- The fifth badge **broke phase 18's guarantee**: `#nav-status` was `nowrap` at ≤640px, so five chips (352px) widened the document to 372px — a sideways scroll on a phone. Fixed by letting the badge row wrap inside its own row (no label shrunk, nothing hidden); the served-CSS pin flipped to demand `wrap`, and the phase-27 mobile audit (extended to include `/gateway`) re-runs clean. + +**Verification** +- `bash .agents/validate.sh`: **1744 passed, 16 skipped**, coverage 99.08%, ruff + pyright + `node --check` clean. +- `data/logs/29-task05-tabs.cjs` → `29-task05-tabs.out`: **97 PASS / 0 FAIL** (8 pages × 1280/360 six tabs + one-own-current + five populated badges + no sideways scroll; 360px dropdown lists all six; the tab navigates and draws the tallies; the `/` line reads the block's counts/href, and a wire-simulated `configured:false` renders nothing). Screenshots `data/logs/masthead-29-*.png`. +- `data/logs/27-task02-mobile-audit.cjs` (now including `/gateway`): **0 failures, 0 findings**. +- Dev server up; `/api/health` and `dev-server.sh status` report all ten sources `ok`. +- validate.sh's dependency advisory (coverage/fastapi/pycparser/ruff newer) is pre-existing; this task touched no dependencies. diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__05_nav_and_summary.a1.md.status b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__05_nav_and_summary.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__05_nav_and_summary.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__05_nav_and_summary.a1.validate b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__05_nav_and_summary.a1.validate new file mode 100644 index 0000000..e16ca03 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__05_nav_and_summary.a1.validate @@ -0,0 +1,73 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 4%] +........................................................................ [ 8%] +........................................................................ [ 12%] +........................................................................ [ 16%] +........................................................................ [ 20%] +........................................................................ [ 24%] +........................................................................ [ 28%] +......................................................s................. [ 32%] +........................................................................ [ 36%] +........................................................................ [ 40%] +........................................................................ [ 45%] +........................................................................ [ 49%] +........................................................................ [ 53%] +........................................................................ [ 57%] +........................................................................ [ 61%] +........................................................................ [ 65%] +........................................................................ [ 69%] +..........................................................ssssssssssss.. [ 73%] +........................................................................ [ 77%] +........................................................................ [ 81%] +........................................................................ [ 85%] +........................................................................ [ 90%] +........................................................................ [ 94%] +..............................................sss....................... [ 98%] +................................ [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 182 0 100% +src/dashboards/api.py 491 0 100% +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 216 5 98% 323-324, 328-329, 346 +src/dashboards/config.py 126 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 321 0 100% +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 3894 37 99% +Required test coverage of 94% reached. Total coverage: 99.05% +1744 passed, 16 skipped, 1 warning in 164.35s (0:02:44) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == +advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose): + coverage 7.16.2 7.17.0 wheel +fastapi 0.153.0 0.161.1 wheel +pycparser 3.11 3.12 wheel +ruff 0.17.1 0.17.2 wheel diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__06_docs_and_verify.a1.err b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__06_docs_and_verify.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__06_docs_and_verify.a1.md b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__06_docs_and_verify.a1.md new file mode 100644 index 0000000..60f43f5 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__06_docs_and_verify.a1.md @@ -0,0 +1,34 @@ +Task 06 is complete. Final state: + +## Done + +**Docs** +- `README.md` — intro now *five* dashboards; `/gateway` page-table row (knob + **agent 1.2.0** requirement + the unconfigured-rather-than-red arm), the `GATEWAY_HOST` / `GATEWAY_INTERVAL_SECONDS` / `GATEWAY_TIMEOUT_SECONDS` config rows, and the user-facing paragraph on the topology question and both degraded arms. +- `AGENTS.md` — route-table row, the `/gateway` UI-conventions paragraph (pipes, `n/a`-not-`0`, no verdict colour, the label-line rule, one history request per card), the metric-naming paragraph's `gateway.in_conns@` / `gateway.out_conns@:`, the config bullet's `default \`gateway\``, and **gap 13** (the residual label width, with the rejected alternative and three costed options). +- New pin `tests/test_api.py::test_the_docs_name_the_page_its_knob_and_its_gauges` so prose can't rot: both page rows name `/gateway` + `GATEWAY_HOST` + 1.2.0, the bullet names the default, the gauges and `/api/gateway/*` stay named. + +**Live read-only pass** — `data/logs/29-task06-live.cjs` → `29-task06-live.out`: **49 PASS / 0 FAIL** +- payload: `configured`/`supported`, agent `1.2.0`, **6 haproxy frontends** (each port→backend), nginx listening on **80/443/4443/8443**, **17 stream-map entries** (fan-out 443×16 + 4443×1), every entry's key/host/port complete. +- page: 4 cards, 41 aria-labelled pipes matching the payload's sentence shape, header tallies = payload figures, `in + out + unmatched == sockets.established` exact, `by port` chip on every `by_port` pipe, collapsed→open chart labelled with legend rows naming exactly that card's pipes. +- `/proc/net/tcp{,6}` via `ssh gateway-ops cat`: per-port inbound == ESTABLISHED-on-that-port (≤1 tolerance), `unmatched.inbound` == the port complement (`2222`, `9120`), `by_port` spot-check page 2 vs /proc 2, ambiguous `:443` → 16 null pipes + one `groups` sentence, and nulls never became `0`. +- control leg: nine pre-existing sources `ok`; `/` 3 fleet rows, `/system` 3 cards, `/disks` 2 hosts (gateway still hidden), `/llm` 2 engines, `/ups` 2 UPSes; `/`'s ingress line renders and disappears on `configured:false`; 360px has no sideways scroll. +- Screenshots: `29-task06-{gateway,card-haproxy,card-stream,charts-expanded,phone}.png`. Transcripts grepped clean of PEM/token/cookie. + +**Pre-1.2.0 arm** — `data/logs/verify-29-task06-predates.py` (+ `.cjs`) → **PASS, 0 failed checks**: throwaway loopback dashboard + scratch SQLite + a stub agent from `tests/fixtures/agent_gateway.json` stamped `1.1.0` with the three sections stripped → `supported: false` with `WHY_PREDATES`, page prints that sentence, draws no cards, every tally `n/a`, banner `info` not red, job `ok`. Screenshot `29-task06-predates.png`. + +## Defect found and fixed +The pass caught the page clipping its own answer: pipe labels showed **seven characters** of a 40-character target (`.pipe-label` was one `nowrap` flex row, so the name shrank to what the chip/footnote/count left). Wrapping the text inside that box was tried and measured worse (eleven lines of one word), so the fix gives the label the line: `flex-wrap: wrap` + `flex: 1 1 max-content` on `.pipe-name` → the name now gets the whole column and the chip/footnote/count wrap beneath. `test_a_clipped_pipe_label_keeps_its_full_text_on_the_element` pins both declarations plus the `title`/`aria-label` that keep the remaining ellipsis from being a lost fact; the residual (28 of 42 at 1440px, 16 of 42 at 360px) is gap 13 with the options ordered by cost. + +## Gates +- `bash .agents/validate.sh` → **rc=0**: 1746 passed, 16 skipped, coverage **99.05 %**, ruff clean, pyright 0 errors, `node --check` clean. +- `./scripts/dev-server.sh status`: all **ten** sources `ok`. +- Regression control for the CSS change: phase 27's nine-page audit re-ran — **371 PASS, 0 failures, 0 findings**. +- validate.sh's dependency advisory (coverage/fastapi/pycparser/ruff newer) is pre-existing; no dependency moved. + +## Notable decisions +- The kernel cross-check asserts the **payload's own partition** (`attributed + unmatched == established`) and reports the raw `/proc` total as a finding — two reads 0.8 s apart cannot be equal on a busy ingress, and pretending otherwise would make the pass flaky about a fact that isn't a bug. The per-port table is the real join proof. +- Payload and DOM are read payload → probe → payload, retrying if `ts` moved, so a 30 s poll can't compare a picture to a newer document. +- The 1.1.0 arm was proven on a loopback stub, never by rolling a fleet host back. + +## Next pending task +This was phase 29's last file — `.agents/phases/todo/29_gateway_dashboard_page/` now holds only `00_phase.md` and `06_docs_and_verify.md`, so the phase is ready to move to `complete/`. No other phase is in `todo/`; gap 13 (pipe-label column width) is the one open item this task surfaced. diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__06_docs_and_verify.a1.md.status b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__06_docs_and_verify.a1.md.status new file mode 100644 index 0000000..9766475 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__06_docs_and_verify.a1.md.status @@ -0,0 +1 @@ +ok diff --git a/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__06_docs_and_verify.a1.validate b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__06_docs_and_verify.a1.validate new file mode 100644 index 0000000..b688c80 --- /dev/null +++ b/.agents/reports/29_gateway_dashboard_page/29_gateway_dashboard_page__06_docs_and_verify.a1.validate @@ -0,0 +1,74 @@ +== uv run pytest --cov (hermetic: no network, live tests skip) == +........................................................................ [ 4%] +........................................................................ [ 8%] +........................................................................ [ 12%] +........................................................................ [ 16%] +........................................................................ [ 20%] +........................................................................ [ 24%] +........................................................................ [ 28%] +........................................................s............... [ 32%] +........................................................................ [ 36%] +........................................................................ [ 40%] +........................................................................ [ 44%] +........................................................................ [ 49%] +........................................................................ [ 53%] +........................................................................ [ 57%] +........................................................................ [ 61%] +........................................................................ [ 65%] +........................................................................ [ 69%] +............................................................ssssssssssss [ 73%] +........................................................................ [ 77%] +........................................................................ [ 81%] +........................................................................ [ 85%] +........................................................................ [ 89%] +........................................................................ [ 93%] +................................................sss..................... [ 98%] +.................................. [100%] +=============================== warnings summary =============================== +.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.14.4-final-0 ________________ + +Name Stmts Miss Cover Missing +------------------------------------------------------------------ +src/dashboards/__init__.py 3 0 100% +src/dashboards/__main__.py 14 1 93% 32 +src/dashboards/alerts.py 182 0 100% +src/dashboards/api.py 491 0 100% +src/dashboards/app.py 190 0 100% +src/dashboards/auth.py 265 0 100% +src/dashboards/certs.py 292 0 100% +src/dashboards/collectors.py 216 4 98% 324, 328-329, 346 +src/dashboards/config.py 126 0 100% +src/dashboards/debug.py 14 0 100% +src/dashboards/poller.py 165 3 98% 192-194, 279 +src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183 +src/dashboards/rates.py 73 0 100% +src/dashboards/severity.py 49 0 100% +src/dashboards/sources/__init__.py 0 0 100% +src/dashboards/sources/disks.py 134 0 100% +src/dashboards/sources/gateway.py 321 0 100% +src/dashboards/sources/host.py 209 0 100% +src/dashboards/sources/llm.py 178 1 99% 285 +src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736 +src/dashboards/sources/nut_sim.py 134 0 100% +src/dashboards/store.py 313 2 99% 553-554 +------------------------------------------------------------------ +TOTAL 3894 36 99% +Required test coverage of 94% reached. Total coverage: 99.08% +1746 passed, 16 skipped, 1 warning in 165.41s (0:02:45) +== uv run ruff check src tests agent == +All checks passed! +== uv run pyright == +0 errors, 0 warnings, 0 informations +== node --check src/dashboards/web/static/*.js == +advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose): + certifi 2026.10.19 2026.10.23 wheel +coverage 7.16.2 7.17.0 wheel +fastapi 0.153.0 0.161.1 wheel +pycparser 3.11 3.12 wheel +ruff 0.17.1 0.17.2 wheel diff --git a/.env.dev.example b/.env.dev.example index d1b465f..eddbaff 100644 --- a/.env.dev.example +++ b/.env.dev.example @@ -44,6 +44,13 @@ export HOST_TARGETS="driveripper=https://driveripper.example.net:9120 deskwork=h # export DISKS_INTERVAL_SECONDS=60 # export DISKS_TIMEOUT_SECONDS=30 +# The /gateway source (phase 29) reads the one HOST_TARGETS agent that runs haproxy + nginx — +# agent 1.2.0's haproxy/nginx/sockets sections — on its own cadence. The default name already +# matches the HOST_TARGETS line above, so set these only to watch an ingress called something +# else, or to poll faster while developing the page. +# export GATEWAY_HOST=gateway +# export GATEWAY_INTERVAL_SECONDS=10 + # The engines are dialled directly on their hostnames, like the agents. export GUFO_BASE_URL=http://deskwork.example.net:8731 export VLLM_BASE_URL=http://driveripper.example.net:8000 diff --git a/.env.example b/.env.example index 1d392b2..05f1813 100644 --- a/.env.example +++ b/.env.example @@ -73,6 +73,22 @@ DISKS_INTERVAL_SECONDS=300 # fetch has to outlast HOST_TIMEOUT_SECONDS rather than inherit it. DISKS_TIMEOUT_SECONDS=30 +# --------------------------------------------------------------------------- # +# Gateway dashboard (/gateway): one HOST_TARGETS name is the ingress +# The `gateway` source reads the SAME agent /snapshot the system page reads, projected onto +# the agent's haproxy/nginx config sections and its live /proc socket table (agent 1.2.0). +# There is no URL here: `GATEWAY_HOST` must be a name in HOST_TARGETS, and a name that is not +# there simply means "this deployment has no ingress to watch" — the source is unconfigured, +# never a red panel for a host nobody installed (same rule as /disks, which has no targets of +# its own either). +# --------------------------------------------------------------------------- # +GATEWAY_HOST=gateway +# 30 s: a connection count is the shape of half a minute, not of ten seconds. The timeout is +# the plain agent fetch's own — these sections cost the agent two config reads and two /proc +# tables — with headroom because the same handler may be mid-smartctl pass for /disks. +GATEWAY_INTERVAL_SECONDS=30 +GATEWAY_TIMEOUT_SECONDS=10 + # --------------------------------------------------------------------------- # # LLM dashboard: either engine is optional — unset means "not deployed here" # and its half of the page disappears instead of showing a dead panel. diff --git a/AGENTS.md b/AGENTS.md index 7d59dab..517f2dc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -19,16 +19,17 @@ keep its numbers honest, and the state of the fleet it watches. ## What this is -Four dashboards for one homelab, behind one domain, in one process, backed by one SQLite -file — plus an overview page that says which of the four you need: +Five dashboards for one homelab, behind one domain, in one process, backed by one SQLite +file — plus an overview page that says which of the five you need: -| Route | Question it answers | Source | -| --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- | -| `/` | Is anything wrong, and where do I look first? | all four, in one request | -| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent per host | -| `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) | -| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` | -| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) | +| Route | Question it answers | Source | +| ---------- | ------------------------------------------------------------------------------ | ------------------------------------------------- | +| `/` | Is anything wrong, and where do I look first? | all five, in one request | +| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent per host | +| `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) | +| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` | +| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) | +| `/gateway` | What is coming in through the router, and where does it go — frontends, `server_name`s, the SNI map, live pipes | the same agent (1.2.0) on the one host `GATEWAY_HOST` names: its `haproxy` + `nginx` config sections joined to its `/proc/net/tcp{,6}` sockets | History is a first-class feature, not a sparkline afterthought: every number worth graphing is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`. @@ -342,11 +343,12 @@ Responsibilities, one line each: publish: the same rule that scrubs a drive's serial (invariant 6). Those two tables are volatile by nature, so `tests/test_agent.py` asserts their **shape**, the ports the recorded configs claim, and the absence of a globally routable endpoint — never a particular row, which a re-run may lawfully - change. What phase 28 stops at is the wire: `sources/host.py` passes **none** of the three into - `/api/system/snapshot` and `store.py` records no series for a proxy or a socket, so the sections - live in the agent's own document and reach no page until phase 29's. What the dashboard does read - today is their **error lines**, through `collectors.ADDON_ERROR_PREFIXES`, so a host with no proxy - config stays out of `/system`'s `degraded` (invariant 3). + change. What phase 28 stopped at was the wire: `sources/host.py` still passes **none** of the + three into `/api/system/snapshot`, and `store.py` still records no series for a config — the join + and the two gauges it publishes are phase 29's `sources/gateway.py`, which reads the same + `/snapshot` through its own `gateway` job. What the *system* page does read today is their + **error lines**, through `collectors.ADDON_ERROR_PREFIXES`, so a host with no proxy config stays + out of `/system`'s `degraded` (invariant 3). - `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real env var always wins. Defaults are safe except the sources, which default to unset. The auth block (phase 11) is `DASH_ADMIN_PASSWORD` (the **seed**, and `_env()` strips, so `VAR=` means @@ -355,19 +357,32 @@ Responsibilities, one line each: `http://127.0.0.1`; task 03 also treats `X-Forwarded-Proto: https` as secure), and `DASH_PBKDF2_ITERATIONS` (default `auth.PBKDF2_ITERATIONS`, imported so the primitive and the setting cannot drift) — that knob exists so the hermetic suite hashes in microseconds instead - of 0.3 s per client and is not a production tuning parameter. + of 0.3 s per client and is not a production tuning parameter. The gateway block (phase 29) is + `GATEWAY_HOST` (default `gateway`, and **a name, never a URL**: `gateway_enabled` is true only + when a `HOST_TARGETS` entry has it, so the page is unconfigured rather than red for a + deployment with no ingress — the `nut_enabled` shape, and the reason `any_source_enabled` + includes it), `GATEWAY_INTERVAL_SECONDS` (30 — a connection count is the shape of half a + minute, not of ten seconds) and `GATEWAY_TIMEOUT_SECONDS` (10, the plain agent fetch with + headroom for a handler that may be mid-smartctl pass for `/disks`). - `poller.py` — job loop, health records, stale/last-good semantics, transition events. `/disks` is a **source of its own** (`disks:`, `DISKS_INTERVAL_SECONDS` 300 s, `DISKS_TIMEOUT_SECONDS` 30 s, events under the dashboard name `disks`) reading the same agents the `system` jobs read: the separate task is what keeps a SMART poll out of the 10-second utilisation loop and lets `/disks` age on its own instead of blinking `/system`. + `gateway` is a **job of its own** for the same reason (phase 29 task 02) — `GATEWAY_INTERVAL_SECONDS` + 30 s, `GATEWAY_TIMEOUT_SECONDS` 10 s, health and events under the dashboard name `gateway` — but + **one** job rather than one per host (the `nut` shape), dialling the `HOST_TARGETS` entry + `GATEWAY_HOST` names with its `label` set to that host's name, because the page is about the single + ingress and a record per machine would ask four hosts a question only one can answer. It exists + only when that name is a target: an unconfigured source has no task that could look red + (invariant 3), and it is handed no `RateEstimator` because a count is not a rate (rule 2). The job list is built once (`app.build_hub()`) because it is wired twice — production and the injected-store test path — and two hand-written copies drifting is how a source ends up polling in tests and not in the container. The success path records (payload → samples → events) and only *then* moves `last_ok`, so a refused SQLite write ages the source and alerts instead of leaving it current with charts that have silently stopped drawing. - Events are stored under the **dashboard** name (`system`/`disks`/`llm`/`ups`), not the sample + Events are stored under the **dashboard** name (`system`/`disks`/`gateway`/`llm`/`ups`), not the sample source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever (found on `/system` during phase 03 browser verification). - `collectors.py` — the fetch half. `AgentTls` (phase 15 task 03) is what makes an `https://` @@ -390,8 +405,16 @@ Responsibilities, one line each: problem. `agent_tls_map()` builds one entry per configured host (the CA and client pair are shared — the dashboard has one identity — only the revocation answer is per name, and an absent entry is *active*, never-issued being a different fact from revoked); `build_hub` takes that map - and hands the **same** entry to the `host:` and `disks:` jobs of one host, so a revocation - fails both sources with one sentence and neither of them dials. `fetch_snapshot()` keeps + and hands the **same** entry to the `host:`, `disks:` and `gateway` jobs of one host, so a + revocation fails every source reading that agent with one sentence and none of them dials + (`tests/test_api.py`'s revoked-gateway pin is the three-job version of the `host:`/`disks:` one). + `fetch_gateway_snapshot()` + `gateway_job()` (phase 29 task 02) are the third reader of one + `/snapshot`: the job is **one per deployment** (the `nut` shape, key `gateway`, `label` = the + `GATEWAY_HOST` name) on `GATEWAY_INTERVAL_SECONDS` with `GATEWAY_TIMEOUT_SECONDS`, it is handed + **no** `RateEstimator` (a connection count that goes through a Δ-window is a rate with a label and + no measurement behind it, rule 2), and `gateway_job` returns `None` when the name is not in + `HOST_TARGETS` — the `disks` configured gate, so an unconfigured source gets no task that could + look broken and `/api/health` says nothing about it. `fetch_snapshot()` keeps `http://` targets on the shared client with their bearer token, unchanged from before the phase, so a rollout is per host. `NutMonitor` owns per-UPS transition bookkeeping (its @@ -447,6 +470,41 @@ Responsibilities, one line each: counts for the page header and drops a host with no payload from the sums while still counting it in `hosts_total` — the `/llm` fleet-block rule, so the page never does arithmetic and a dead agent never shrinks a fleet total. +- `sources/gateway.py` — the same snapshot projected onto the **connection topology** of the one + host `GATEWAY_HOST` names (phase 29): `parse(snapshot, host)` joins agent 1.2.0's `haproxy` + frontends and `nginx` `listen`s into **inbound** pipes (one per listen port, carrying the + `server_name` domains and the live ESTABLISHED count + distinct client addresses) and haproxy + `server` lines, concrete `proxy_pass` targets and the `$var`→`map` entries into **outbound** + pipes, plus the `unmatched` tallies (`inbound` = connections on a listening port no config + names, `outbound` = connections to something no target claims, `other` = live listeners no + config claims — phase 28's "an unclaimed socket stays visible as one"). `count` semantics are + the whole honesty of the module: **no DNS** (the spy test pins `socket.getaddrinfo` is never + called — `ipaddress` decides "is this already an address", and it never asks), an exact + `(address, port)` match for a target written as an address (`by_port: false`), a port-only + match for a **name** and only when that port has exactly one configured host + (`by_port: true`), and **no attribution at all** when several names share a port (the real + `stream.conf` map's sixteen `*.reeselink.com:443`): those pipes carry `connections: null`, the + sockets go to `unmatched.outbound`, and `unmatched.groups` says once per port how many and + among which candidates — picking one would be rule 2, and repeating the port's total on + sixteen pipes reads as sixteen times the traffic. A pipe is `null` (never `0`) whenever the + socket table did not answer or the join could not decide, and publishes **no sample** then, so + a chart cannot draw a zero for an unknown. The classification is one exclusive pass + (`join_sockets`), which is what makes "inbound + outbound + unmatched = the socket table" + hold — the identity task 06 checks against a direct `/proc/net/tcp` read. `supported: false` + (a 1.1.0 agent, or `--no-proxy`) keeps every key with empty lists and null counts plus + `WHY_PREDATES`, `available: false`/`null` per section becomes a `why` sentence and the agent's + own line in `section_errors`, and nothing here carries a `severity` (`net.*` precedent: a busy + proxy is the job) or a `window_seconds` (a count is not a rate). Gauges: `gateway.in_conns@` + and `gateway.out_conns@:` — the port **is** in the outbound name, because this + fleet's two haproxy Minecraft backends share `minecraft.reeselink.com` on 25565 and 25566 and + two gauges on one name would overwrite each other in silence. `parse_gateway_snapshot()` is the + `Result` the poller consumes; there are **no events** (a connection count has no transition + worth a line — a proxy that stops routing shows up as its host's own poll failing). The + never-answered arm is `stub(host, ts)` — the **same skeleton** `parse` fills in, plus + `unreachable: true`, `supported: null` and `WHY_NO_ANSWER` — because the route that serves a host + with no payload yet must not hand the page a shape with keys missing, and `WHY_NO_ANSWER` is not + `WHY_PREDATES`: an agent that has not replied has not said it has no sections (invariant 3's two + facts staying two sentences). - `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`, `VLLM_HISTOGRAMS`), measured Δ rates **diffed per label set** (`rate_key()` keys the estimator on the resolved name plus its labels; the cell is the Σ of the per-series @@ -638,7 +696,18 @@ Responsibilities, one line each: `severity.py` rates no sector counter and this module may not have a threshold of its own, so grading a drive here would be the bug, not the fix (phase 09). Its `HREF` table does carry `/disks`, so the one alert a disk job *can* raise — SMART failing while its host answers — - links to the page that names the drive. + links to the page that names the drive, and since phase 29 `/gateway` for the same reason: a + `gateway` job failing **while its host answers** (the proxy configs or the socket table went + quiet inside a healthy agent) has no other voice. The echo suppression is now a table + (`AGENT_ECHO_DASHBOARDS` = `disks`, `gateway`) because three jobs read one agent, and it finds the + partner differently for each: `disks:` names the host in its key, while the single `gateway` + job names it in its **label** (its key is one word, and `GATEWAY_HOST` is what an alert prints). + A missing `host:` record suppresses nothing — "I cannot find the host's job" is not evidence + the host is fine. The gateway source has **no verdict rule and none is wanted**, the same shape as + the disk rule: its payload publishes no `severity`, `severity.THRESHOLDS` has no `gateway.*` row, + `alerts.build` takes no gateway payload at all, so a `gateway` alert can only ever be + reachability — stale past 3× its own 30 s interval or stopped answering — and 400 connections on + an ingress is the job, not a fault. - `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning. The `meta` table is the only durable key/value state in the app and is reached through three named helpers — `get_meta`/`set_meta`/`delete_meta` (the last returns its rowcount, because @@ -654,7 +723,7 @@ Responsibilities, one line each: cannot remove a name, so `_note_series_write()` ends the cache on a `(source, host, metric)` the store has never written and leaves a poll of an existing series alone — invalidating on every sample was task 02's defect, and it put the 572 ms scan back on nearly every `/api/metrics` call - on a store nine pollers write to, behind a cache that never served; `delete_series()`/`maintain()` + on a store every poller writes to, behind a cache that never served; `delete_series()`/`maintain()` invalidate unconditionally because a `DELETE` can take a name, `delete_series()` also forgetting the triples it emptied so a re-created series counts as new again), through a generation counter so a write racing a read cannot park a stale answer under a fresh timestamp, with the name cache @@ -756,7 +825,7 @@ Responsibilities, one line each: configured host **with drives to show** (disk rows + `smart` + `smartctl_available` + per-row `severity` + a `fleet` block summed by the API, with a configured-but-never-answered host as an `unreachable` stub whose counts are `None`, not `0`), and `/history` serves the - `disk.*@dev` series with unknown names answering as empty series like the other two + `disk.*@dev` series with unknown names answering as empty series like the other history endpoints. The configured gate runs **before** anything that could reach the network — `HOST_TARGETS` empty is `404 {"configured": false}` with no client, no fetch and no `disks` job in the hub at all (invariant 3), and `tests/test_api.py` proves it with @@ -778,11 +847,33 @@ Responsibilities, one line each: only POST that **dials an upstream** stays `/api/ups/command` (invariant 1, pinned by enumerating every served route's verbs and sorting the writes into fleet / session / certificate-authority lists — the third is phase 15's, and it writes `meta` rows and nothing else). - All four `history` routes reach the store through one door, `api._history_batch()` → + `/api/gateway/*` (phase 29 task 03) is the ingress pair, and it is the same two rules on a + single-host source. `/snapshot` answers `sources/gateway.parse`'s payload for the one + `GATEWAY_HOST` entry — the haproxy frontends and the nginx `listen`/`map` echo, the inbound pipes + (port, domains, live count, up to `MAX_CLIENTS` client addresses), the outbound pipes (`by_port` + naming which half of the match is a socket fact), each pipe's own `metric`, the `unmatched` + tallies, and `supported`/`sections`/`why` — with `states` appended and the payload's own `ts` + left alone, because that is when the agent took the reading and it is what the page ages. Three + arms reach it and none of them is red: `404 {"configured": false}` **in the route's first line** + when `GATEWAY_HOST` names no `HOST_TARGETS` entry (unset and wrong-name are both spy-pinned to + `httpx.AsyncClient.get` and `collectors.fetch_snapshot`, and neither arm has a `gateway` job to + age out either), `supported: false` for a 1.1.0/`--no-proxy agent, and `gateway_source.stub()` + for a configured host that has never answered — the payload's key set plus `unreachable: true` + and minus only the `agent_version` nobody published, every count `null` because an agent that has + not spoken has not reported an idle gateway (rule 2). Nothing in the payload carries a `severity` + or a `window_seconds`, at any depth (pinned by a recursive walk: a busy proxy is the job and no + Δ was ever measured). `/history` serves the `gateway.in_conns@` / + `gateway.out_conns@:` gauges with `hosts=` defaulting to the ingress **alone** (the + name the samples are stored under — defaulting to every target would draw empty plots for + machines nobody asked to route), and it carries no configured gate because it dials nothing: + like `/api/disks/history`, an unconfigured source answers no hosts and no series. The pair adds + no write (the verb enumeration pins the two GETs and that the app's one upstream POST is still + `/api/ups/command`), and `tests/test_auth_api.py` lists both paths as session-required. + All five `history` routes reach the store through one door, `api._history_batch()` → `store.history_many()` (phase 14 task 02). That is a cost change and nothing else: the series list is host-major and metric-minor in request order, each entry the same `{source, host, metric, range_seconds, points}` document the per-series loop built, with the - same empty-series answer for a name that has no rows. `tests/test_api.py` pins it for all four + same empty-series answer for a name that has no rows. `tests/test_api.py` pins it for all five routes against points read straight out of the store with `query()` — a third path, in neither the loop nor the batch — so a future "optimisation" that moves a point, an order, or a key fails a test instead of redrawing a chart wrong. @@ -804,6 +895,25 @@ Responsibilities, one line each: job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in `system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total` counts the machines there are and `hosts_live` the ones answering. + The **`gateway` block** (phase 29 task 02) is additive in the same way: `{configured, host, href, + supported, unreachable, inbound, outbound, map_targets, unmatched, unmatched_listeners, partial, + states}`. Task 02 wrote it for programmatic consumers (the page itself reads + `/api/gateway/snapshot`, task 03); task 05 gave it one page-side reader, the single line `/` + renders under the fleet table, so `href` joined the block — derived from `alerts.HREF`, the one + table that writes a dashboard's route, so the overview's link and an alert about the same source + cannot spell the page differently (`test_the_gateway_block_links_to_the_page_the_alert_row_links_to`, + and the page half reads only block keys: `test_the_overview_line_reads_the_gateway_block_it_claims` + resolves the served `GATEWAY_FIELDS` against a live document). `inbound`/`outbound` are the sums of the pipes' own counts and `map_targets` counts the stream + map's routing entries; every figure is `null` — never `0` — in the arms that did not measure + (unconfigured, configured-but-never-answered, a 1.1.0/`--no-proxy agent, a silent socket table), + and `partial: true` says a total had to leave an unknown pipe out (a port whose configured targets + cannot be told apart) rather than pretend to be the whole number. `unmatched` counts + *connections* and `unmatched_listeners` counts *ports*, so they are two fields and never one sum, + and nothing in the block carries a `severity`. `/api/config` grew the matching row (`enabled`, + `host`, `url`, `interval_seconds`, `timeout_seconds`), where `url` is the named entry's + `Target.display_url` and `null` when no target carries the name — invariant 6's boundary applies + to a name-only source exactly as it does to a URL-bearing one, because this body is what a browser + renders. - `app.py` — assembly, and the **session gate** (`session_gate` middleware, invariant 12) whose `PUBLIC_PATHS`/`PUBLIC_PREFIXES` are the whole anonymous surface. It answers in this order and the order is the design: no `AuthState` → `503` for everything; no session and no such route → @@ -829,14 +939,18 @@ Responsibilities, one line each: directory the lifespan's `finally` removes. What hands that material to the loops is `agent_tls_map(material, store, targets)` → `build_hub(..., tls=...)`, called in **both** branches the same way: no material (or no targets) means no TLS map, which is the plain - `http://`/token path rather than an error, and one map covers both of a host's jobs. A corrupt stored certificate raises out of the same + `http://`/token path rather than an error, and one map covers all of a host's jobs — including the + `gateway` job, which is why one revoke in `/certs` silences the pipes with no restart. A corrupt stored certificate raises out of the same place — a dashboard that cannot present its own client certificate would otherwise spend the fleet's TLS history reporting an upstream problem it caused itself. `build_hub` also builds **two** `RateEstimator`s, because a window is only a claim about the polls that fill it (phase 23, `TODO.md:8`'s "ramp up/ramp down"): the `host` jobs get `RateEstimator(span=settings.rate_window)` (10 s — what a 10 s cadence can fill) and the `llm` jobs `RateEstimator(span=settings.llm_interval)` (2 s, one poll interval, so a slot that starts - reads steady on the next poll instead of climbing one fifth of the true rate per poll), and + reads steady on the next poll instead of climbing one fifth of the true rate per poll), and the + `gateway` job (phase 29) is handed **neither** — it is added to the hub between the `disks` and + `llm` jobs, only when `settings.gateway_enabled`, because a gauge needs no window and one job + cannot be right about two cadences' worth of them., there is no third knob — the LLM window **is** `LLM_INTERVAL_SECONDS`, which is the only way the pair cannot drift. One estimator for both could only be right about one of them: a 2 s span over a 10 s poll sits the window-empty fallback against the `2 × span` ceiling and answers @@ -844,9 +958,14 @@ Responsibilities, one line each: The estimator's own gates (the `MIN_WINDOW_SECONDS` floor, the ceiling, the 8× jump confirmation with its independent anchor) are unchanged and apply to the smaller window, whose `Δ2.0s` label the payload carries with the value. -- `web/` — five dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`) plus the login page - (`/login`, `pages/login.html` + `static/page-login.js`), one stylesheet, `charts.js`. No - framework. `/login` renders the masthead's **brand link and theme toggle** and nothing else +- `web/` — six dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`, `/gateway`) plus the + login page (`/login`, `pages/login.html` + `static/page-login.js`), one stylesheet, `charts.js`. + No framework. The tab nav is those six and nothing else, in that order, on **all eight** + signed-in pages (phase 29 task 05): `tests/test_api.py::test_the_disks_tab_is_in_the_masthead_of_every_page` + compares the nav's hrefs to `TAB_ORDER` (which **is** `PAGES`) and asserts the `aria-current` + count per page — one, naming itself, on the six dashboards and zero on `/settings` and `/certs`. + `/gateway`'s own tab arrived with that task; until then the page existed behind a route with no + way to click to it. `/login` renders the masthead's **brand link and theme toggle** and nothing else from it: no tabs (every tab bounces straight back here, and a nav of links that all redirect is a lie about what is available), no status dots and no clock (there is nothing signed in for them to say). It posts through `Dash.fetchJSON` like every other script, which is only safe because @@ -864,8 +983,8 @@ Responsibilities, one line each: login page is audited for orphan `id`s and CDN references too (`tests/test_auth_api.py`), fetched unsigned because a signed-in `GET /login` is a `303` and would audit `/`'s markup under the wrong name. `/settings` (`pages/settings.html` + `static/page-settings.js`) is a **page and not a - sixth dashboard**: same `` and masthead, the five tabs with **no** `aria-current` (it is - not one of them), the same four `#nav-status` badges as every dashboard (phase 25 task 01 — + seventh dashboard**: same `` and masthead, the six tabs with **no** `aria-current` (it is + not one of them), the same five `#nav-status` badges as every dashboard (phase 25 task 01 — `TODO.md:3`'s "hosts disks engines UPS disappear when in the certs/account page" was the missing container, and `Dash.initChrome()` fills it from `/api/health` with no JS change), and it is reached from the `⚙ Account` link in @@ -884,19 +1003,19 @@ Responsibilities, one line each: `csrf` is now invalid — every later write, the UPS buttons included, 403s until a reload) and on any `401` bounce. `/certs` (`pages/certs.html` + `static/page-certs.js`) is the second of those admin pages, and it - is the same shape for the same reasons: five tabs with **no** `aria-current`, the four fleet + is the same shape for the same reasons: six tabs with **no** `aria-current`, the five fleet badges in `#nav-status` like every other masthead (phase 25 task 01, same reasoning as - `/settings` above), and both masthead links — `⚷ Certs` beside `⚙ Account` — exactly as the five + `/settings` above), and both masthead links — `⚷ Certs` beside `⚙ Account` — exactly as the six dashboards render them. **Both links are on both admin pages** (phase 20 task 04, `TODO.md:4`), which retires phase 15's "no self-link" precedent: chrome that changes shape depending on where you already are reads as a broken button, while a link to the page you are on is a no-op that answers with the page you were looking at (`data/logs/20-task04-admin-links.cjs` measures exactly that — one document - response, no redirect, the same cluster afterwards). The two links therefore sit on all seven - signed-in pages (the five dashboards plus `/settings` and `/certs`) in one order + response, no redirect, the same cluster afterwards). The two links therefore sit on all eight + signed-in pages (the six dashboards plus `/settings` and `/certs`) in one order (`⚷ Certs` → `⚙ Account` → theme toggle → clock — phase 25 task 03 moved the clock to the end so it is the farthest-right item on a `flex-end` row), which is what lets the pinned audit be one loop over `MARKUP_PAGES` instead of two over different tuples; `/login` - is the eighth page and the one with neither. Everything on it is + is the ninth page and the one with neither. Everything on it is `GET /api/certs`: the Root CA block, the display-only Dashboard client line (that certificate is minted with the root and has no route of its own), and one row per host with its status chip, serial, expiry (local timestamp *and* `Dash.duration` left), issued-vs-deployed `bundle_rev`, @@ -926,7 +1045,12 @@ Responsibilities, one line each: `data/logs/27-task01-scroll.cjs`). `app.js`'s masthead status dots are one per dashboard job — including `disks`, because the same agent can answer the 10-second utilisation poll and - have a wedged SMART pass, and one dot for both would be right about neither. `app.js` owns the + have a wedged SMART pass, and one dot for both would be right about neither, and including + `gateway` (phase 29 task 05), which is one dot rather than one per host because the ingress is + one job and a dot for a source that stays `idle` when `GATEWAY_HOST` names nothing is how an + unconfigured source reads grey rather than red. The badge table is a literal in `initChrome()`, + so a new dashboard's job means a new entry there — `NAV_STATUS_BADGES` in `tests/test_api.py` + is the pin that says so. `app.js` owns the colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an `ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and `Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a @@ -941,7 +1065,7 @@ Responsibilities, one line each: **The masthead is three rows at every width, the two control rows flush right** (phase 20 task 01 made the rows, `TODO.md:5-6`; phase 25 task 02 re-justified them, `TODO.md:4` — "I made a mistake earlier, I did want hosts disks engines UPS badges and certs/account/dark right - justified"): row 1 the brand, `#nav-toggle` and the five tabs; row 2 the `#nav-status` + justified"): row 1 the brand, `#nav-toggle` and the six tabs; row 2 the `#nav-status` per-source dots; row 3 the `.masthead-right` controls (`⚷ Certs`, `⚙ Account`, theme, then the clock — phase 25 task 03 moved the clock last for `TODO.md:5`'s "the time should show up as the farthest right item"; @@ -960,11 +1084,20 @@ Responsibilities, one line each: brand and the tabs on separate rows. `height: var(--header-h)` became **`min-height`** (three rows do not fit in 58 px and a fixed height clips the controls) and is declared **once** — the ≤640px block no longer restates `height`/`min-height`/`flex-wrap`, and that pair drifting is exactly what - would clip the bar on a phone only. Every page that renders the bar now renders three rows — + would clip the bar on a phone only — except that **`#nav-status` wraps at ≤640px since phase 29 + task 05**: five badges are 352 px of chips against this bar's 320 px content box, and a `nowrap` + item that may not shrink does not get smaller, it widens the *document* (measured 372 px — the + sideways scroll phase 18 forbids). The fifth chip takes a second line of the same right-aligned + row and the bar grows by one line box; no label shrinks and nothing is hidden. Phase 18's + `nowrap` was right for four (267 px) and the pin now demands the opposite, with + `data/logs/29-task05-tabs.cjs` measuring 360 px documents on all eight pages. + Every page that renders the bar now renders three rows — `/certs` and `/settings` grew the dots row in phase 25 task 01, where `#nav-status` is pinned as - the **first child** of `.masthead-right` on all seven signed-in pages (`app.js` is untouched: - the container's existence is what makes `initChrome` build the four badges, which is why the fix - was markup and not JS) — and `/login` remains the one page with no dots, rendering two rows + the **first child** of `.masthead-right` on all eight signed-in pages (`app.js`'s *badge list* is + a literal, and phase 25's fix was markup because the container was what was missing; phase 29 + task 05 added the fifth entry — `{ key: 'gateway', label: 'gateway', href: '/gateway' }` — because + the ingress has a job of its own and a job with no badge is a source that blinks in silence) — + and `/login` remains the one page with no dots, rendering two rows (brand, then the theme toggle). Pinned on the served bytes by phase 20's three tests in `tests/test_api.py` — the row declarations (task 02 renamed that one to `test_the_masthead_is_three_rows_right_justified_at_every_width`), the mobile block's silence @@ -980,12 +1113,14 @@ Responsibilities, one line each: (the row-break assertions are the same ones phase 20 wrote, deliberately: the re-justify must not cost the rows), and the measured half is `data/logs/25-task02-justify.cjs` — seven pages × two widths, each demanding both rows compute `flex-end` and sit flush with the bar's right - padding edge, the four badges present, three rows intact, the dropdown still under the whole + padding edge, the four badges present (a four-badge claim — `29-task05-tabs.cjs` is the pin that + the fifth, `gateway`, joined them), three rows intact, the dropdown still under the whole bar and no sideways scroll, plus `/login`'s two-row bar with no badges (`data/logs/25-task02-justify.out`, screenshots `data/logs/25-task02-*.png`). **Phase 25 task 03 then put the clock at the end of the control row** (`TODO.md:5` — "The time should show up as the farthest right item on the navbar"): the `` moved from the first - control to the last one on all seven templates, so the row reads `⚷ Certs` → `⚙ Account` → theme + control to the last one on all seven templates then (eight since `/gateway` joined the nav), so + the row reads `⚷ Certs` → `⚙ Account` → theme toggle → clock. It is a **markup** move and deliberately not a CSS `order:` declaration, because on a `flex-end` row the last child *is* the farthest-right item and an `order:` override would be something the served-byte pin cannot see — `test_every_page_links_to_both_admin_pages` therefore @@ -1015,7 +1150,8 @@ Responsibilities, one line each: off it after a scroll; screenshots `data/logs/masthead-unsticky*.png`). Note that a browser cannot tell `relative` from `relative; top: 0` — Chrome computes `top: 0px` for either — so the absence of the offset is a served-CSS pin, not a measurement. - **The tab nav collapses into a hamburger at ≤640px** (phase 18): the seven tabbed pages each + **The tab nav collapses into a hamburger at ≤640px** (phase 18): the eight tabbed pages (the six + dashboards plus `/settings` and `/certs`; `/gateway` joined in phase 29 task 05) each carry `#nav-toggle` (`class="ghost nav-toggle"`, `type="button"`, `aria-controls="nav-tabs"`, `aria-label="Toggle navigation"`, `☰`) between `.brand` and `
diff --git a/src/dashboards/web/pages/disks.html b/src/dashboards/web/pages/disks.html index 02171e7..416f4b6 100644 --- a/src/dashboards/web/pages/disks.html +++ b/src/dashboards/web/pages/disks.html @@ -18,6 +18,7 @@ Disks LLM UPS + Gateway
diff --git a/src/dashboards/web/pages/gateway.html b/src/dashboards/web/pages/gateway.html new file mode 100644 index 0000000..f474b99 --- /dev/null +++ b/src/dashboards/web/pages/gateway.html @@ -0,0 +1,222 @@ + + + + + +Homelab · gateway + + + + +
+ ◆ Homelab + + +
+ + + + + --:--:-- +
+
+ +
+
+
+

Gateway

+
reading the ingress…
+
+
+ +
+
+
listeners
+
n/a
+
+
+
established
+
n/a
+
+
+
pipes in
+
n/a
+
+
+
pipes out
+
n/a
+
+
+
unmatched
+
n/a
+
+
+
+ + + + + + + + +
+

Connection map

+
+ + + + + + + + + +
+
+ +
+ a pipe animates while it carries at least one live connection and rests + at zero, and is never coloured by a verdict — a busy ingress is the job. + a by port pipe was matched on its socket port + alone: the target is a name, the agent resolves nothing, and that port carries more than one + configured host — so the count is the port's, not this target's. +
+
+ + + + + + diff --git a/src/dashboards/web/pages/index.html b/src/dashboards/web/pages/index.html index 99109d0..a7e2bf6 100644 --- a/src/dashboards/web/pages/index.html +++ b/src/dashboards/web/pages/index.html @@ -18,6 +18,7 @@ Disks LLM UPS + Gateway
@@ -67,6 +68,10 @@

Run agent/host_agent.py on each machine and list it in HOST_TARGETS.

+ +
diff --git a/src/dashboards/web/pages/llm.html b/src/dashboards/web/pages/llm.html index e0cfb65..56732a3 100644 --- a/src/dashboards/web/pages/llm.html +++ b/src/dashboards/web/pages/llm.html @@ -18,6 +18,7 @@ Disks LLM UPS + Gateway
diff --git a/src/dashboards/web/pages/settings.html b/src/dashboards/web/pages/settings.html index 3fea10c..4f952a2 100644 --- a/src/dashboards/web/pages/settings.html +++ b/src/dashboards/web/pages/settings.html @@ -23,6 +23,7 @@ Disks LLM UPS + Gateway
diff --git a/src/dashboards/web/pages/system.html b/src/dashboards/web/pages/system.html index f74e92f..ae5bf2c 100644 --- a/src/dashboards/web/pages/system.html +++ b/src/dashboards/web/pages/system.html @@ -18,6 +18,7 @@ Disks LLM UPS + Gateway
diff --git a/src/dashboards/web/pages/ups.html b/src/dashboards/web/pages/ups.html index 3adad41..31dde95 100644 --- a/src/dashboards/web/pages/ups.html +++ b/src/dashboards/web/pages/ups.html @@ -18,6 +18,7 @@ Disks LLM UPS + Gateway
diff --git a/src/dashboards/web/static/app.js b/src/dashboards/web/static/app.js index 7cf6ff3..d3b82e7 100644 --- a/src/dashboards/web/static/app.js +++ b/src/dashboards/web/static/app.js @@ -658,6 +658,10 @@ const Dash = (() => { { key: 'disks', label: 'disks', href: '/disks' }, { key: 'llm', label: 'engines', href: '/llm' }, { key: 'ups', label: 'UPS', href: '/ups' }, + // The fifth dashboard's job is a single job for the one ingress, so this badge is one + // dot rather than one per host — and it stays grey when `GATEWAY_HOST` names nothing, + // because `refreshStatus` only counts sources the health record calls `configured`. + { key: 'gateway', label: 'gateway', href: '/gateway' }, ]; nav.innerHTML = jobs.map((job) => ` ${job.label}`).join(''); diff --git a/src/dashboards/web/static/charts.js b/src/dashboards/web/static/charts.js index a194c2f..b333e60 100644 --- a/src/dashboards/web/static/charts.js +++ b/src/dashboards/web/static/charts.js @@ -124,7 +124,15 @@ class LineChart { } } if (!isFinite(lo)) return [0, 1]; - if (lo === hi) { const pad = Math.abs(lo) * 0.1 || 1; return [lo - pad, hi + pad]; } + if (lo === hi) { + // A flat series still needs a band to draw in, but a declared floor is not negotiable: + // padding below `yMin: 0` put "-1 connections" on the axis of a gateway whose every pipe was + // idle, which is a scale describing a state that cannot happen (the same reason the page + // prints `n/a` instead of a made-up zero). + const pad = Math.abs(lo) * 0.1 || 1; + const floor = this.opts.yMin !== null ? Math.min(lo, this.opts.yMin) : lo - pad; + return [floor, hi + pad]; + } const margin = (hi - lo) * 0.12; lo = this.opts.yMin !== null ? this.opts.yMin : lo - margin; hi = this.opts.yMax !== null ? this.opts.yMax : hi + margin; diff --git a/src/dashboards/web/static/dash.css b/src/dashboards/web/static/dash.css index 69540bb..0ef74bd 100644 --- a/src/dashboards/web/static/dash.css +++ b/src/dashboards/web/static/dash.css @@ -284,20 +284,26 @@ button.ghost:disabled { opacity: .45; cursor: not-allowed; } padding-top: calc(.5rem + env(safe-area-inset-top, 0px)); } - /* The same treatment one level down, and these three rules are still what a 360px phone - * actually requires — phase 18 measured both sides of them: without them the four status - * dots fell into a single-file column (`#nav-status` carries the shared `.gap` class, which - * is `flex-wrap: wrap`) and `⚷ Certs` broke over two lines, because every item shrank to a - * fraction of its content. So the cluster may wrap, its items may neither shrink nor break a - * label, and the dots stay the one line that makes them a readout instead of a stack. Row 3 - * still wraps *within itself*: `⚷ Certs`, `⚙ Account` and the theme button do not fit in - * 320px once over on the narrowest phones, and nothing here hides a control to make - * the arithmetic work — a phone gets the same words every other width does. The row - * alignment is the base rule's `flex-end` (phase 25 task 02), restated nowhere: this block - * only tightens the gap, so the two rows cannot disagree about which edge they sit on. */ + /* The same treatment one level down, and these rules are still what a 360px phone actually + * requires — phase 18 measured both sides of them: the status dots fell into a single-file + * column and `⚷ Certs` broke over two lines, because every item shrank to a fraction of its + * content. So the cluster may wrap, its items may neither shrink nor break a label, and the + * dots keep their labels. Row 3 still wraps *within itself*: `⚷ Certs`, `⚙ Account` and the + * theme button do not fit in 320px once over on the narrowest phones, and nothing here hides a + * control to make the arithmetic work — a phone gets the same words every other width does. + * The row alignment is the base rule's `flex-end` (phase 25 task 02), restated nowhere: this + * block only tightens the gap, so the two rows cannot disagree about which edge they sit on. + * + * The dots wrap too, since phase 29 task 05 gave them a fifth member. Phase 18's `nowrap` was + * right for four (measured then at 267px) and wrong for five: `hosts disks engines UPS + * gateway` is 352px of chips against this bar's 320px content box, and an item that may not + * shrink and may not wrap does not become smaller — it widens the *document* (372px, so a + * phone got a sideways scroll, which phase 18 exists to forbid). Wrapping inside a row that is + * `flex-basis: 100%` is not the single-file column phase 18 saw either: the fifth chip moves to + * a second line of the same right-aligned row and the bar grows by one line box. */ .masthead-right { flex-wrap: wrap; gap: .4rem; } .masthead-right > * { flex: none; white-space: nowrap; } - #nav-status { flex-wrap: nowrap; } + #nav-status { flex-wrap: wrap; row-gap: .25rem; } /* The reveal, through task 01's token: `#nav-toggle` reads its `display` from * `--nav-toggle-display`, so declaring the token on the element here is the whole flip — @@ -1253,6 +1259,102 @@ input[type="text"].confirm { * and it is the one cell a table cannot nowrap: an address list. */ table.data td.san { white-space: normal; } +/* ---- gateway pipes --------------------------------------------------------- + * `/gateway` (phase 29 task 04): the connection map. One card per source family, and inside each + * card two columns of pipes joined by one node — the gateway itself. A pipe is one `` with + * a travelling dash pattern, so the whole vocabulary here is ink and motion: a connection count + * is not a graded number, which is why this section draws no coloured verdict on any of it — the + * `net.*` precedent, and the API sends no verdict for a gateway figure at any depth. + * + * Three layout facts are load-bearing, all of them measured in `data/logs/29-task04-pipes.cjs`: + * the node is one rounded rect that does **not** stretch to the taller column (a 17-target + * fan-out would otherwise grow a 600 px wall beside a six-pipe side); an empty column collapses + * and the node moves to the edge it is joined to (`solo`); and at a phone width the three columns + * stack with the node between them, because 320 px of card has no room for a pipe *and* its + * label, and an unlabelled pipe is a line nobody can read. + */ +.gate-tallies { display: flex; flex-wrap: wrap; gap: .9rem; align-items: flex-end; } +/* "listeners no config names" is a fact, and `.metric .label` truncates with an ellipsis at a card + width that cannot fit it — a clipped label is a clipped fact (phase 27's rule), so these wrap. + The five header tallies keep the shared truncating rule: their labels are one word each. */ +.gate-card .metric .label { white-space: normal; } +.gate-grid { min-width: 0; } +.pipe-columns { + display: grid; gap: .5rem 1rem; min-width: 0; align-items: center; + grid-template-columns: minmax(0, 1fr) auto minmax(0, 1fr); +} +/* A card with pipes on one side only. The stream map's fan-out is the case that ships: its + * inbound half is the nginx `443` listener and lives on the nginx card, because the payload never + * says which listener a map is reached through and the page does not guess one. Two classes, not + * one, because *which* side is empty decides which grid track the node belongs in. */ +.pipe-columns.solo-out { grid-template-columns: auto minmax(0, 1fr); } +.pipe-columns.solo-in { grid-template-columns: minmax(0, 1fr) auto; } +.pipe-col { display: grid; gap: .5rem; min-width: 0; align-content: center; } +.pipe-node { + align-self: center; justify-self: center; text-align: center; + min-width: 6rem; max-width: 9rem; padding: .6rem .45rem; + border: 1px solid var(--border-strong); border-radius: var(--radius); + background: var(--bg-sunken); color: var(--text-muted); + font: 600 11.5px var(--mono); letter-spacing: .04em; overflow-wrap: anywhere; +} +.pipe-row { display: grid; gap: .12rem; min-width: 0; } +.pipe-label { + display: flex; flex-wrap: wrap; align-items: baseline; gap: .05rem .35rem; min-width: 0; +} +/* A pipe's label is the page's whole answer — which port, which domain, which target — and this row + is what makes the answer readable. Task 06 measured it on live data: as one `nowrap` flex line the + name shrank to whatever the `by port` chip, the config's own `server` name and the `flex: none` + count left, which drew seven characters of `reeseapps-gitea.reeselink.com:2222`; wrapping the text + inside that squeezed box was worse (eleven lines, a word at a time). So the name claims the line: + `max-content` as its base size means a name longer than the column is alone on its line and gets + the *whole* column (~22 characters), with the chip, the footnote and the count wrapping beneath it, + while a short label like `:22` still sits beside its count on one line exactly as before. What is + still clipped after that is a target longer than the column, which is a column-width fact + (AGENTS.md gap 13) and not a lost one: `renderPipeRow` puts the full string on the element as + `title`, `pipeAria` carries the whole endpoint, and + `test_a_clipped_pipe_label_keeps_its_full_text_on_the_element` pins both declarations and both + strings. The live pass re-counts the clipped labels on every run. */ +.pipe-name { + flex: 1 1 max-content; font-size: 11.5px; min-width: 0; + overflow: hidden; text-overflow: ellipsis; white-space: nowrap; +} +.pipe-detail { font-size: 11px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } +/* the count sits at the node end of the label line, so the eye goes from the number straight + into the pipe that carries it */ +.pipe-count { + margin-inline-start: auto; padding-inline-start: .5rem; flex: none; + font: 600 12.5px var(--mono); +} +/* The pipe is a grid row of its own, the full width of its column — which is what makes it touch + * the node, since the inbound column is the node's left and the outbound column its right. */ +.pipe { display: block; width: 100%; min-width: 0; height: 10px; } +.pipe-path { fill: none; stroke: var(--border-strong); stroke-width: 2; stroke-dasharray: 7 9; } +/* One connection is enough to move. The dash travels the pipe's own direction, which is + * *towards* the node in the inbound column and *away from* it in the outbound one — the node is + * the far end of one and the near end of the other, so one keyframe is the whole direction + * story and a reversed second one would only be a second thing to keep in step. */ +.pipe.live .pipe-path { stroke: var(--accent); animation: pipe-flow 1.2s linear infinite; } +/* Zero is a static, faint line rather than an absent one: "this pipe exists and carries nothing" + * is a fact the drawing has to be able to say, and it is a different sentence from `n/a`. */ +.pipe.still .pipe-path { stroke: var(--text-faint); stroke-dasharray: 3 7; opacity: .55; } +@keyframes pipe-flow { + /* one full dash+gap cycle (7 + 9), so the loop is seamless */ + from { stroke-dashoffset: 0; } + to { stroke-dashoffset: -16; } +} +.pipe-clients { font-size: 10.5px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } +.pipe-empty { font-size: 12px; } +.gate-groups { list-style: none; margin: 0; padding: 0; display: grid; gap: .3rem; } +.gate-groups li { display: flex; gap: .45rem; align-items: baseline; min-width: 0; } +.gate-groups .pipe-name { flex: none; } + +@media (max-width: 640px) { + .pipe-columns, + .pipe-columns.solo-out, + .pipe-columns.solo-in { grid-template-columns: minmax(0, 1fr); } + .pipe-node { max-width: none; width: 100%; } +} + /* ---- dialog ------------------------------------------------------------- */ dialog { border: 1px solid var(--border-strong); border-radius: var(--radius); background: var(--bg-raised); @@ -1313,4 +1415,11 @@ footer.pagefoot { nav.tabs, nav.tabs.is-open { transition: none; } + + /* `/gateway`'s pipes, **both states**. The animated rule out-specifies the base one exactly + * the way `.is-open` out-specifies `nav.tabs`, so stilling the static state alone would leave + * a live pipe flowing for someone who asked for no motion — the same phase-46 lesson the menu + * pair above carries, applied the day a second animated component landed in this sheet. */ + .pipe.live .pipe-path, + .pipe.still .pipe-path { animation: none; } } diff --git a/src/dashboards/web/static/page-gateway.js b/src/dashboards/web/static/page-gateway.js new file mode 100644 index 0000000..0ae3ed4 --- /dev/null +++ b/src/dashboards/web/static/page-gateway.js @@ -0,0 +1,646 @@ +/* Gateway page: what this ingress is connected to, and how much is on right now. + + `/system` answers what the gateway box is doing. This answers the topology question it cannot: + which sockets face inbound, where each one's traffic goes, and how many connections are live on + each. The source is `/api/gateway/snapshot` — `sources/gateway.py`'s join of the agent's + `haproxy` and `nginx` config parses with its `/proc/net/tcp{,6}` tallies. + + The page draws **pipes**, and a pipe's only claim is a count. It animates while that count is at + least one connection and rests at zero; it is never coloured by a verdict (`net.*`'s precedent: + a busy ingress is the job, and `severity.py` rates no `gateway.*` metric). `n/a` means the join + did not decide — a socket table that did not answer, or a port whose configured targets cannot be + told apart — and `n/a` never becomes `0` here (rule 2). + + Two rules keep the drawing honest: + + * **The label is the config's own text.** A bind keeps its port, a `server_name` keeps its + domain, a `proxy_pass` keeps the address it was written with, and a target the join matched + on its port alone says `by port` rather than pretending the socket named the host. + * **The unreconciled remainder is printed, not hidden.** `unmatched` is the connections no + configured pipe claims, and it is the operator's fact that the gateway talks to something + the configs do not name. + + Charts follow the `/system` discipline: one collapsed `
` per card, painted only while + open, fed by **one** `/api/gateway/history` request per card per refresh naming every gauge that + card could draw. + */ +'use strict'; + +const { $, $$, fetchJSON, setTxt, setDot, num, duration, poll, markStale } = Dash; + +const SVG_NS = 'http://www.w3.org/2000/svg'; + +const state = { + /* No window chips: the page's history is the count trend, and one window for every card is + * the honest default for a source that re-reads every 30 s. */ + range: 86400, + rangeKey: '24h', + every: 30, // GATEWAY_INTERVAL_SECONDS, from /api/config — never guessed here + cards: new Map(), // card key -> its element (the skeletons the page ships) + rows: new Map(), // card key -> Map(pipe key -> row element): patched, not rebuilt + charts: new Map(), // card key -> LineChart, built on first expand only + history: new Map(), // card key -> { range, response } from the last snapshot pass + snapshot: null, // last payload, so a toggle paints without fetching + poller: null, + /* Half the job's interval: an age stamp cannot sit at "updated 29s" for a whole cycle, and + * nothing on the page claims a fresher reading than the agent took. */ + pollMs: 15000, +}; + +// ---- the cards -------------------------------------------------------------- +// +// One card per source family, keyed on the payload's own `source` word plus `unmatched`. That +// keying is not a layout preference: `source` is the only fact in the document that says which +// config a pipe came from, and the stream fan-out's pipes are `map`-sourced while the listener +// they are reached through is an `nginx` one — so the map card is outbound by construction, and +// joining the two would be the page inventing a relationship the agent never reported. +const CARDS = [ + { key: 'haproxy', title: 'haproxy', section: 'haproxy', + empty: 'haproxy names no listening port on this host' }, + { key: 'nginx', title: 'nginx', section: 'nginx', + empty: 'nginx names no listening port on this host' }, + { key: 'map', title: 'the stream map', section: 'nginx', + empty: 'the stream map routes to nothing' }, +]; + +const CHART_HEIGHT = 120; +const COUNT_FORMAT = (v) => num(v, 0); +const CLIENTS_SHOWN = 2; + +/** The measured count as text; the unknown is `null` so `setTxt` prints `n/a` and not `0`. */ +function countText(value) { + return typeof value === 'number' && isFinite(value) ? num(value, 0) : null; +} + +/** One pipe family's pipes, split on the payload's own `source` word. */ +function familyPipes(snapshot, key) { + return { + inbound: (snapshot.inbound || []).filter((pipe) => pipe.source === key), + outbound: (snapshot.outbound || []).filter((pipe) => pipe.source === key), + }; +} + +/** `host:port` -> the stream-map keys that route to it, out of `nginx.map_targets`. */ +function mapKeys(snapshot) { + const keys = new Map(); + ((snapshot.nginx || {}).map_targets || []).forEach((entry) => { + if (!entry || !entry.key) return; + const name = `${entry.host}:${entry.port}`; + const list = keys.get(name) || []; + list.push(entry.key); + keys.set(name, list); + }); + return keys; +} + +/** + * The label a pipe carries. + * + * Inbound is a port, because that is what the config binds. Outbound is the address the config + * was written with, and for a stream-map target the SNI name the map matched on rides in front of + * it — both of those strings are the document's own, and nothing here resolves or reverses one. + */ +function pipeName(pipe, keys) { + if (pipe.port != null) return `:${pipe.port}`; + const target = pipe.target || {}; + const host = target.host || '?'; + const endpoint = `${host}${target.port == null ? '' : `:${target.port}`}`; + const through = keys ? keys.get(`${target.host}:${target.port}`) : null; + return through && through.length === 1 ? `${through[0]} → ${endpoint}` : endpoint; +} + +/** A pipe's stable identity: its own gauge name, unique per listen port and per target. */ +function pipeKey(pipe) { + return pipe.metric || `${pipe.port}`; +} + +/** The row's footnote: the haproxy `server` name, or the count of map keys on one endpoint. */ +function pipeDetail(pipe, keys) { + const target = pipe.target || {}; + if (target.name) return `server ${target.name}`; + const through = keys ? keys.get(`${target.host}:${target.port}`) : null; + if (through && through.length > 1) return `${through.length} map names`; + return null; +} + +/** The accessible sentence for one pipe — every figure in it came from the payload. */ +function pipeAria(direction, pipe, keys) { + const bits = [direction === 'in' ? 'inbound pipe' : 'outbound pipe']; + bits.push(direction === 'in' ? `port ${pipe.port}` : `to ${pipeName(pipe, keys)}`); + const domains = pipe.domains || []; + if (domains.length) bits.push(`names ${domains.join(', ')}`); + const clients = pipe.clients || []; + if (clients.length) bits.push(`clients ${clients.join(', ')}`); + bits.push(pipe.connections == null ? 'connections unknown' + : `${pipe.connections} connections`); + return bits.join(', '); +} + +/** The pipe itself: hand-rolled SVG — one path, `role="img"`, an `aria-label` of its own. */ +function buildPipe() { + const svg = document.createElementNS(SVG_NS, 'svg'); + svg.setAttribute('class', 'pipe still'); + svg.setAttribute('viewBox', '0 0 100 10'); + svg.setAttribute('preserveAspectRatio', 'none'); + svg.setAttribute('focusable', 'false'); + const path = document.createElementNS(SVG_NS, 'path'); + path.setAttribute('class', 'pipe-path'); + // One geometry for both columns: the dash runs from x=0 to x=100, which is *towards* the + // gateway node in the inbound column and *away from* it in the outbound one. The node is the + // far end of one and the near end of the other, so a single keyframe is the whole direction + // story and a reversed second one would be a second thing to keep in step. + path.setAttribute('d', 'M0 5 H100'); + path.setAttribute('vector-effect', 'non-scaling-stroke'); + svg.appendChild(path); + return svg; +} + +/** + * A pipe row: the label line (name, `by port`, the config's own name for the target, count), + * the SVG pipe under it, and the client addresses under that. Built once, patched afterwards. + * + * The pipe is the full width of its column, which is what makes it touch the node: the inbound + * column is the node's left and the outbound column its right, so one geometry pointing right + * runs *into* the gateway in one column and *away from* it in the other. + */ +function pipeRow(direction) { + const el = document.createElement('div'); + el.className = `pipe-row ${direction}`; + el.innerHTML = ` + + + + + + + `; + el.insertBefore(buildPipe(), el.querySelector('.pipe-clients')); + return el; +} + +/** + * Patch one row from its pipe. + * + * Every string here is upstream-provided — a `server_name`, a `proxy_pass`, a client address — + * so it goes out through `textContent` and `title`, never through markup. + */ +function renderPipeRow(el, direction, pipe, keys) { + const name = el.querySelector('.pipe-name'); + const text = pipeName(pipe, keys); + if (name.textContent !== text) name.textContent = text; + const domains = pipe.domains || []; + name.title = domains.length ? domains.join(', ') : text; + + const flag = el.querySelector('.pipe-flag'); + const byPort = !!(pipe.target && pipe.target.by_port); + flag.hidden = !byPort; + if (byPort) { + setTxt(flag, 'by port'); + flag.title = 'matched on the port alone — this target is a name and the port carries more ' + + 'than one configured host, so the count is the port\'s, not this target\'s'; + } else { + flag.removeAttribute('title'); + } + + // the config's own label for this target — the haproxy `server` name, or how many map names + // arrive at one endpoint + const detail = el.querySelector('.pipe-detail'); + const note = pipeDetail(pipe, keys); + detail.hidden = note == null; + if (note != null) setTxt(detail, note); + + const shown = countText(pipe.connections); + setTxt(el.querySelector('.pipe-count'), shown); + + const svg = el.querySelector('.pipe'); + const live = typeof pipe.connections === 'number' && isFinite(pipe.connections) + && pipe.connections > 0; + svg.classList.toggle('live', live); + svg.classList.toggle('still', !live); + svg.setAttribute('role', 'img'); + const aria = pipeAria(direction, pipe, keys); + if (svg.getAttribute('aria-label') !== aria) svg.setAttribute('aria-label', aria); + + // The clients are the row's third line and its tooltip: the addresses are the operator's + // shortest path from "this port is busy" to "that machine is busy". + const clients = (pipe.clients || []).slice(0, CLIENTS_SHOWN); + const rest = (pipe.clients || []).length - clients.length; + const track = el.querySelector('.pipe-clients'); + const shown2 = clients.length ? `${clients.join(' ')}${rest > 0 ? ` +${rest}` : ''}` : ''; + track.hidden = shown2.length === 0; + setTxt(track, shown2); +} + +/** Make a column's rows match `pipes`, patching the ones that stay and dropping the rest. */ +function renderPipes(card, direction, key, pipes, keys) { + const box = card.querySelector(`[data-k="${direction}"]`); + const seen = state.rows.get(key) || new Map(); + state.rows.set(key, seen); + const wanted = new Set(); + pipes.forEach((pipe) => { + const id = `${direction}:${pipeKey(pipe)}`; + wanted.add(id); + let row = seen.get(id); + if (!row) { + row = pipeRow(direction); + seen.set(id, row); + box.appendChild(row); + } + renderPipeRow(row, direction, pipe, keys); + }); + Array.from(seen.keys()).forEach((id) => { + if (wanted.has(id) || !id.startsWith(`${direction}:`)) return; + seen.get(id).remove(); + seen.delete(id); + }); + // An empty column collapses rather than holding half the card for nothing — which is what + // moves the node to the edge it is being joined to (the `solo` class below). + box.hidden = pipes.length === 0; + return pipes.length; +} + +/** The gateway node: one rounded rect mid-card, labelled with the host the reading came from. */ +function renderNode(card, snapshot, columns) { + const node = card.querySelector('[data-k="node"]'); + const visible = columns.inCount + columns.outCount > 0; + node.hidden = !visible; + // One column of pipes only: the node moves to the edge it is joined to instead of floating in + // the middle of an empty half-card. Two classes, because *which* half is empty decides which + // grid track the node belongs in, and one shared class could only ever be right about one. + const frame = card.querySelector('.pipe-columns'); + frame.classList.toggle('solo-out', visible && columns.inCount === 0); + frame.classList.toggle('solo-in', visible && columns.outCount === 0); + const label = snapshot.host || 'gateway'; + if (node.dataset.text !== label) { + node.dataset.text = label; + node.textContent = label; + } +} + +// ---- the unmatched card ------------------------------------------------------ +// +// Not pipes: there is no config to draw these from, only the socket table's own leftovers. The +// two tallies are two different facts and never one sum — `other` counts *ports* no config +// claims, the other counts *connections* the join could not attribute to a target. + +/** Does this payload leave anything unaccounted for? An all-zero family gets no card. */ +/** + * Is there a remainder to report? + * + * A positive count or a group, yes. Three `null`s is not "unknown worth printing" — it is the join + * having measured nothing at all (a pre-1.2.0 agent, a silent socket table), and a card of three + * `n/a`s says less than the page's own `why` line above it already says (the dead-cell rule). + */ +function unmatchedVisible(snapshot) { + const un = snapshot.unmatched || {}; + return [un.other, un.inbound, un.outbound].some( + (value) => typeof value === 'number' && value > 0) || (un.groups || []).length > 0; +} + +/** One `
  • ` of the "could belong to any of N" list — built, not templated: the names are data. */ +function groupRow(group) { + const li = document.createElement('li'); + const port = document.createElement('span'); + port.className = 'pipe-name mono'; + port.textContent = `:${group.port}`; + const why = document.createElement('span'); + why.className = 'faint'; + const hosts = (group.hosts || []).slice(0, CLIENTS_SHOWN); + const rest = (group.hosts || []).length - hosts.length; + why.textContent = `${countText(group.connections) || 'n/a'} connections · could be any of ` + + `${group.host_count} targets${hosts.length ? ` (${hosts.join(', ')}${rest > 0 + ? ` +${rest}` : ''})` : ''}`; + li.appendChild(port); + li.appendChild(why); + return li; +} + +function renderUnmatched(card, snapshot, row) { + const un = snapshot.unmatched || {}; + card.hidden = false; + setDot(card.querySelector('[data-k="dot"]'), dotFor(row, snapshot)); + setTxt(card.querySelector('.updated'), stalenessText(row)); + markStale(card, dimmed(snapshot)); + setTxt(card.querySelector('[data-k="um-other"]'), countText(un.other)); + setTxt(card.querySelector('[data-k="um-in"]'), countText(un.inbound)); + setTxt(card.querySelector('[data-k="um-out"]'), countText(un.outbound)); + + const groups = un.groups || []; + const list = card.querySelector('[data-k="groups"]'); + const wanted = groups.map((group) => `${group.port}|${group.connections}|${group.host_count}`); + const same = list.dataset.wanted === wanted.join(',') && list.children.length === groups.length; + list.hidden = groups.length === 0; + if (same) return; + list.dataset.wanted = wanted.join(','); + list.textContent = ''; + groups.forEach((group) => list.appendChild(groupRow(group))); +} + +// ---- card chrome --------------------------------------------------------------- + +/** The reachability dot: the job's own health, never a number. */ +function dotFor(row, snapshot) { + if (!row || !row.configured) return 'idle'; + if (row.ok === false || snapshot.unreachable) return 'fail'; + return Dash.ageSeconds(row.last_ok) > 3 * state.every ? 'warn' : 'ok'; +} + +function stalenessText(row) { + if (!row || !row.configured) return 'not configured'; + if (row.last_ok == null) return row.error || 'never reached'; + const seconds = Dash.ageSeconds(row.last_ok); + return seconds > state.every ? `stale ${duration(seconds)}` + : `updated ${seconds < 2 ? 'now' : duration(seconds)}`; +} + +/** Did this payload fail to measure at all? Every arm of it dims; none of it reddens. */ +function dimmed(snapshot) { + return !!snapshot.unreachable || snapshot.supported !== true; +} + +/** Every card skeleton, wired once to the markup the page ships. */ +function cardElements() { + const cards = new Map(); + $$('#gate-cards [data-gate]').forEach((el) => { + cards.set(el.dataset.gate, el); + const box = el.querySelector('[data-k="charts"]'); + if (box) box.addEventListener('toggle', () => chartsToggled(el.dataset.gate)); + const chip = el.querySelector('[data-k="charts-window"]'); + if (chip) chip.textContent = state.rangeKey; + }); + return cards; +} + +/** + * Paint one card from the snapshot. + * + * A card appears when it has a fact: pipes of its own. A family whose config section did not + * answer has none, and the page's own `why` line is the one sentence that says which section — + * so the card stays off the grid instead of repeating it in a smaller font. + */ +function renderCard(card, snapshot, row) { + const key = card.dataset.gate; + if (key === 'unmatched') { + if (unmatchedVisible(snapshot)) renderUnmatched(card, snapshot, row); + else card.hidden = true; + return; + } + const definition = CARDS.find((entry) => entry.key === key); + const { inbound, outbound } = familyPipes(snapshot, key); + if (inbound.length + outbound.length === 0) { + // Two different nothings. The section answered and named no pipe: that is a fact about this + // gateway's config, and the card says it. The section did not answer at all: the page's own + // `why` line is the one sentence that says which section, so no card repeats it. + const answered = (snapshot.sections || {})[definition.section] === true; + card.hidden = !answered; + if (answered) setTxt(card.querySelector('[data-k="empty"]'), definition.empty); + return; + } + card.hidden = false; + card.querySelector('[data-k="empty"]').hidden = true; + const keys = mapKeys(snapshot); + setDot(card.querySelector('[data-k="dot"]'), dotFor(row, snapshot)); + setTxt(card.querySelector('.updated'), stalenessText(row)); + markStale(card, dimmed(snapshot)); + setTxt(card.querySelector('[data-k="counts"]'), + `${inbound.length} in · ${outbound.length} out`); + + const columns = { + inCount: renderPipes(card, 'in', key, inbound, keys), + outCount: renderPipes(card, 'out', key, outbound, keys), + }; + renderNode(card, snapshot, columns); +} + +// ---- per-card charts ------------------------------------------------------------ +// +// One plot per card, one request per card, every refresh: the query names every gauge that card +// could draw (`/api/gateway/history` answers an unknown name as an empty series on purpose) and +// the legend names **pipes**, not hosts — the card heading already says which config family this +// is, and the page is about one machine. + +/** Every gauge this card's pipes publish. */ +function cardStreams(snapshot, key) { + const keys = mapKeys(snapshot); + const { inbound, outbound } = familyPipes(snapshot, key); + return inbound.concat(outbound) + .filter((pipe) => !!pipe.metric) + .map((pipe) => ({ metric: pipe.metric, label: pipeName(pipe, keys) })); +} + +/** The chart inside one card, built the first time that card's stack is open. */ +function cardChart(key) { + const built = state.charts.get(key); + if (built) return built; + const card = state.cards.get(key); + if (!card) return null; + const definition = CARDS.find((entry) => entry.key === key); + const chart = new DashCharts.LineChart(card.querySelector('[data-chart="conns"]'), { + height: CHART_HEIGHT, xSpan: state.range, yMin: 0, yFormat: COUNT_FORMAT, + ariaLabel: `${(state.snapshot || {}).host || 'gateway'} — live connections on each ` + + `${definition.title} pipe over the last ${state.rangeKey}`, + }); + state.charts.set(key, chart); + return chart; +} + +function seriesFor(response, definitions) { + return definitions.map((definition) => { + const found = ((response || {}).series || []).find((s) => s.metric === definition.metric); + return { name: definition.label, + points: ((found || {}).points || []).map((p) => ({ ts: p.ts, value: p.value })) }; + }); +} + +function chartsOpen(key) { + const card = state.cards.get(key); + const box = card && card.querySelector('[data-k="charts"]'); + return !!(box && box.open); +} + +/** Opening paints from the response this card already has, so the first expand needs no poll. */ +function chartsToggled(key) { + if (!chartsOpen(key) || !state.snapshot) return; + const cached = state.history.get(key); + if (cached) drawCardCharts(key, cached); + else renderCardCharts(key, state.snapshot); +} + +/** The paint pass, and the only place a `LineChart` here is built or drawn. */ +function drawCardCharts(key, response) { + if (!chartsOpen(key) || !state.snapshot) return; + const card = state.cards.get(key); + if (!card) return; + const definitions = cardStreams(state.snapshot, key); + if (!definitions.length) return; + const chart = cardChart(key); + if (!chart) return; + const series = seriesFor(response, definitions); + chart.opts.xSpan = state.range; + chart.opts.area = false; + chart.setData(series); + // The legend names the lines the plot actually drew. A pipe whose join published no sample has + // no line, and a permanent `n/a` row for it is the dead-cell anti-pattern the vLLM slot grid + // avoids; `LineChart`'s own "no samples yet" says the same thing once when none of them did. + // `setData` stamps every series with the colour its path uses, so a swatch still matches its + // line after the filter instead of sliding to the next one in the palette. + const plotted = chart.series.filter((s) => s.points && s.points.length); + DashCharts.renderLegend(card.querySelector('[data-legend="conns"]'), plotted, COUNT_FORMAT, + false); +} + +/** + * One card's history: one request, named by that card's own pipes. + * + * The request is per card per refresh whether or not the disclosure is open — the `/system` + * shape, whose cached response is what lets the first expand paint from data in hand instead of + * waiting a poll — while the *plot* is built only while it is open (`drawCardCharts`). + */ +async function renderCardCharts(key, snapshot) { + const definitions = cardStreams(snapshot, key); + if (!definitions.length) return; + const range = state.range; + const query = `/api/gateway/history?hosts=${encodeURIComponent(snapshot.host || '')}` + + `&metrics=${encodeURIComponent(definitions.map((d) => d.metric).join(','))}` + + `&range=${range}`; + const response = await fetchJSON(query).catch(() => null); + if (!response || !state.cards.has(key)) return; // the card went away mid-flight + state.history.set(key, response); + drawCardCharts(key, response); +} + +// ---- the page header ------------------------------------------------------------- + +/** + * What the pipes on one side carry: the sum of the ones that answered, plus how many did not. + * + * This is the page's only arithmetic and it is a sum of published gauges — adding two counts is + * neither a rate nor a verdict. A pipe whose join could not decide contributes nothing and is + * counted in `unknown`, which is `/api/summary`'s `partial` flag drawn rather than a second + * opinion: the sockets behind it are already inside the `unmatched` column beside it, so + * `attributed + unmatched` stays the kernel's number and no figure on the row is invented. The + * marker is the whole point — a total that quietly left unknowns out reads as the whole number. + */ +function sumConnections(snapshot, side) { + const pipes = snapshot[side] || []; + if (!pipes.length) return null; + let attributed = 0; + let unknown = 0; + for (const pipe of pipes) { + if (typeof pipe.connections === 'number' && isFinite(pipe.connections)) { + attributed += pipe.connections; + } else { + unknown += 1; + } + } + // Nothing answered is not "nothing carrying": with every pipe unknown the side has no number at + // all, and `0` here would be a claim about traffic the join never measured (rule 2). + return unknown === pipes.length ? null : { attributed, unknown, total: pipes.length }; +} + +/** + * One reconciliation cell: its number, the shared `.stale` dim while it is a partial sum, and the + * tooltip naming what is missing — `/llm`'s fleet-block idiom for a total that left something out. + */ +function setTally(id, sum) { + const el = $(id); + setTxt(el, sum == null ? null : sum.attributed); + const metric = el.closest('.metric'); + if (metric) metric.classList.toggle('stale', !!sum && sum.unknown > 0); + el.title = sum == null ? 'the join did not measure this side' + : `${sum.attributed} of ${sum.total} pipes answered${sum.unknown + ? `; ${sum.unknown} could not be told apart, and their connections are under unmatched` + : ''}`; +} + +function renderHeader(snapshot) { + const sockets = snapshot.sockets || {}; + const un = snapshot.unmatched || {}; + const unmatched = un.inbound == null && un.outbound == null + ? null : (un.inbound || 0) + (un.outbound || 0); + setTxt($('#t-listeners'), countText(sockets.listens)); + setTxt($('#t-established'), countText(sockets.established)); + setTally('#t-inbound', sumConnections(snapshot, 'inbound')); + setTally('#t-outbound', sumConnections(snapshot, 'outbound')); + setTxt($('#t-unmatched'), countText(unmatched)); + markStale($('#tallies'), dimmed(snapshot)); + $('#sub').textContent = `${snapshot.host || 'gateway'} · agent ` + + `${snapshot.agent_version || 'n/a'} · ` + + `${(snapshot.inbound || []).length} inbound pipes · ` + + `${(snapshot.outbound || []).length} outbound`; + const banner = $('#gate-banner'); + const why = snapshot.why || null; + banner.hidden = why == null; + if (why != null) setTxt(banner, why); +} + +// ---- snapshot ---------------------------------------------------------------------- + +async function refresh() { + let snapshot; + try { + snapshot = await fetchJSON('/api/gateway/snapshot'); + } catch (err) { + // Two different refusals. `404 {configured: false}` is the unconfigured arm, answered before + // anything was dialled, and the route's own sentence names the variable to set — so the setup + // block appears. Anything else is a failed read, and invariant 4's rule holds: the last good + // drawing stays on screen with the reason up top, rather than blanking the page. + $('#sub').textContent = err.message; + $('#gate-empty').hidden = err.status !== 404; + return; + } + $('#gate-empty').hidden = true; + state.snapshot = snapshot; + + const states = {}; + (snapshot.states || []).forEach((row) => { states[row.dashboard || row.key] = row; }); + const row = states.gateway; + renderHeader(snapshot); + CARDS.forEach((definition) => renderCard(state.cards.get(definition.key), snapshot, row)); + renderCard(state.cards.get('unmatched'), snapshot, row); + + const visible = CARDS.filter((definition) => !state.cards.get(definition.key).hidden) + .map((definition) => definition.key); + pruneCards(visible); + await Promise.all(visible.map((key) => renderCardCharts(key, snapshot))); +} + +/** Clear the row and chart state of every family that left the payload. */ +function pruneCards(visible) { + CARDS.forEach((definition) => { + if (visible.includes(definition.key)) return; + const card = state.cards.get(definition.key); + if (card) { + card.hidden = true; + const rows = state.rows.get(definition.key); + if (rows) rows.clear(); + } + state.rows.delete(definition.key); + state.history.delete(definition.key); + // The `LineChart` instance stays: a card is hidden rather than removed, so its plot is + // built once and repainted whenever the family comes back. + }); +} + +document.addEventListener('DOMContentLoaded', async () => { + state.cards = cardElements(); + try { + const config = await fetchJSON('/api/config'); + const gateway = config.gateway || {}; + if (Number.isFinite(gateway.interval_seconds) && gateway.interval_seconds > 0) { + state.every = gateway.interval_seconds; + } + if (gateway.enabled === false) { + // The unconfigured arm before the first request: the setup block, no cards, and no poll + // loop dialling a route that answers 404 forever (invariant 3). + $('#sub').textContent = gateway.host + ? `no host named ${gateway.host} in HOST_TARGETS — set GATEWAY_HOST` + : 'no gateway configured — set GATEWAY_HOST to a HOST_TARGETS name'; + $('#gate-empty').hidden = false; + return; + } + } catch (err) { /* the defaults are the shipped ones */ } + + state.poller = poll(async () => { await refresh(); }, state.pollMs); + Dash.bindShortcuts(); +}); diff --git a/src/dashboards/web/static/page-index.js b/src/dashboards/web/static/page-index.js index 0991d51..c7f7cd2 100644 --- a/src/dashboards/web/static/page-index.js +++ b/src/dashboards/web/static/page-index.js @@ -96,6 +96,49 @@ function renderAlerts(alerts) { }).join(''); } +/* The two figures the overview's ingress line prints, in the order it prints them. Each `key` is + * a key of `/api/summary`'s `gateway` block, and `tests/test_api.py` resolves this table against + * the payload the route actually serves (the `SLOT_FIELDS` device): a rename on either side fails + * there rather than printing `n/a` beside a busy ingress. */ +const GATEWAY_FIELDS = [ + { key: 'inbound', word: 'inbound' }, + { key: 'outbound', word: 'outbound' }, +]; + +/** + * The ingress's one line under the fleet table: the pipe totals `/api/summary` summed, linked to + * the page that draws them. + * + * Nothing here is counted in the browser and nothing here is a route. The link is the block's own + * `href` (`alerts.py`'s table is where a dashboard's page is written, the same rule the alert rows + * keep), the counts are that block's sums of the pipes, and `n/a` is what a `null` prints: an + * agent that has not answered, one that predates the proxy sections, and a socket table that did + * not reply all measured **nothing**, which is not the sentence "0 connections" (rule 2). + * `configured: false` renders no line at all — the `disks` block's invariant-3 rule: an ingress + * nobody installed gets no row here and no red one to learn to ignore. + */ +function renderGateway(gateway) { + const row = $('#fleet-gate'); + const block = gateway || {}; + if (block.configured !== true) { + row.hidden = true; + row.textContent = ''; + return; + } + row.hidden = false; + // the shared dim for "these are the last good numbers" (invariant 4) + row.classList.toggle('stale', block.unreachable === true); + const why = block.unreachable === true ? `${block.host} is not answering` + : block.partial ? 'a pipe could not be told from another on its port, so the total leaves it out' + : ''; + const counts = GATEWAY_FIELDS + .map(({ key, word }) => `${num(block[key], 0)} ${word}`).join(' · '); + row.innerHTML = `` + + `${DashCharts.escapeHtml(String(block.host || ''))} ` + + `${DashCharts.escapeHtml(counts)}`; + if (why) row.title = why; +} + /** * The fleet table: one row per host, columns unchanged. * @@ -161,6 +204,7 @@ async function refresh() { `${live}/${total} hosts · ${engines} · ${ups} · ${problems}`; renderAlerts(alerts); renderFleet(hosts, system.states); + renderGateway(summary.gateway); } catch (err) { $('#summary-line').textContent = `summary failed: ${err.message}`; } diff --git a/tests/conftest.py b/tests/conftest.py index 338fda1..bb70f78 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -243,6 +243,76 @@ def disk_snapshot(agent_snapshots: dict[str, dict[str, Any]], smart_shape: Any) return build +# ---- phase 29: the gateway page (the /gateway source, API and page) -------- # + +#: The captured gateway tree, as the paths the agent's own `--root` resolution expects. +#: These are the verbatim phase-28 captures (`scripts/capture-gateway-config.sh`), so the +#: config half of a `/gateway` test is the file the real ingress runs, not a memory of it. +GATEWAY_TREE = { + "etc/haproxy/haproxy.cfg": "gateway_haproxy.cfg", + "etc/nginx/nginx.conf": "gateway_nginx_main.conf", + "etc/nginx/conf.d/http.conf": "gateway_nginx_conf_d_http.conf", + "etc/nginx/stream.d/stream.conf": "gateway_nginx_stream_d_stream.conf", + "proc/net/tcp": "gateway_proc_net_tcp.txt", + "proc/net/tcp6": "gateway_proc_net_tcp6.txt", +} + +GATEWAY_MODES = ("full", "predates", "no_haproxy", "no_nginx", "sockets_silent") + + +@pytest.fixture +def gateway_snapshot(host_agent_module: Any, tmp_path: Path) -> Any: + """`build(mode="full")` -> a 1.2.0-style agent snapshot carrying the three proxy sections. + + Not a hand-written dict: the captured configs and `/proc` tables are laid into a fake + `--root` tree and read by the **real agent `Collector`**, so the sections handed to + `sources/gateway.py` are the bytes agent 1.2.0 serves. A field rename on either side of the + wire therefore fails a source test instead of drawing an empty pipe on the page (rule 7). + + Modes are the degradation arms the page has to render: + + * `full` — all three sections `available: true`, live sockets from the capture + * `predates` — none of the three keys at all (an agent older than 1.2.0, or `--no-proxy`) + * `no_haproxy` / `no_nginx` — that one config file is absent: `available: false` plus the + agent's own **not found** line, the other half of the page untouched + * `sockets_silent` — neither `/proc` table readable: `available: null`, so every live count + must come back `None` and never `0` + """ + def build(mode: str = "full") -> dict[str, Any]: + if mode not in GATEWAY_MODES: + raise AssertionError(f"unknown gateway snapshot mode {mode!r}") + root = tmp_path / f"root-{mode}" + for relative, fixture in GATEWAY_TREE.items(): + if not fixture: + continue + target = root / relative + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text((FIXTURES / fixture).read_text()) + if mode == "no_haproxy": + (root / "etc/haproxy/haproxy.cfg").unlink() + if mode == "no_nginx": + for path in (root / "etc/nginx/nginx.conf", root / "etc/nginx/conf.d/http.conf", + root / "etc/nginx/stream.d/stream.conf"): + path.unlink() + if mode == "sockets_silent": + (root / "proc/net/tcp").unlink() + (root / "proc/net/tcp6").unlink() + + errors: list[str] = [] + collector = host_agent_module.Collector(root) + snapshot: dict[str, Any] = { + "ok": True, "schema": 1, "ts": 1_760_000_000.0, "host": "gateway", + "agent_version": host_agent_module.AGENT_VERSION, "errors": errors, + } + if mode != "predates": + snapshot["haproxy"] = collector.haproxy(errors) + snapshot["nginx"] = collector.nginx(errors) + snapshot["sockets"] = collector.sockets(errors) + return snapshot + + return build + + @pytest.fixture def metrics_text() -> Any: """metrics_text('gufo') / metrics_text('vllm') -> the captured exposition verbatim.""" diff --git a/tests/test_alerts.py b/tests/test_alerts.py index 6eff584..ef8dec6 100644 --- a/tests/test_alerts.py +++ b/tests/test_alerts.py @@ -14,9 +14,10 @@ are the ones that make the landing page trustworthy rather than merely busy: from __future__ import annotations +import inspect from typing import Any -from dashboards import alerts +from dashboards import alerts, severity from dashboards.config import Settings from dashboards.sources import nut from dashboards.sources.nut_sim import NutSimulator @@ -481,6 +482,83 @@ def test_a_dead_upsd_is_one_alert_and_not_its_stale_cards_too() -> None: assert [item["source"] for item in out] == ["NUT server"] +# --------------------------------------------------------------------------- # +# the gateway page: reachability is the only voice, and a dead agent is still one line +# --------------------------------------------------------------------------- # +def gateway_state(**kwargs: Any) -> dict[str, Any]: + """The health record of the one `gateway` job (the `nut` shape: key `gateway`, label = host).""" + return state("gateway", "gateway", "gateway", **kwargs) + + +def test_a_stale_gateway_job_is_a_warn_measured_in_its_own_thirty_second_poll() -> None: + """The pipes are drawn from a 30 s poll, so 60 s of silence is two healthy polls. + + Judging this source by the 10 s system interval would amber the landing page every minute for + a job behaving exactly as configured — the same mistake the `disks` rule exists to avoid, one + cadence later. + """ + settings = Settings(gateway_interval=30.0) + assert alerts.build(settings, [gateway_state(age=60.0)]) == [] + out = alerts.build(settings, [gateway_state(age=100.0)]) + assert out[0]["severity"] == "warn" + assert out[0]["href"] == "/gateway", "the overview's row must link to the page it describes" + assert "every 30s" in out[0]["message"], "the message prints the gateway's own interval" + + +def test_a_gateway_job_that_fails_while_its_host_answers_is_the_only_voice_it_has() -> None: + """A wedged proxy/socket section is a fault nothing else can report (the `disks` rule, once more).""" + out = alerts.build(SETTINGS, [ + gateway_state(ok=False, error="UpstreamError: gateway: timed out"), + state("host:gateway", "system", "gateway"), + ], now=2_000.0) + assert len(out) == 1 + assert out[0]["source"] == "gateway" + assert out[0]["href"] == "/gateway" + assert "timed out" in out[0]["message"] + + +def test_a_gateway_job_that_failed_because_its_agent_is_down_is_one_line_not_two() -> None: + """Three jobs dial one agent; the operator has one problem, so the overview says it once.""" + down = {"ok": False, + "error": "UpstreamError: gateway: http://h:9120/snapshot unreachable (ConnectError)"} + out = alerts.build(SETTINGS, [ + gateway_state(**down), + state("disks:gateway", "disks", "gateway", **down), + state("host:gateway", "system", "gateway", **down), + ], now=2_000.0) + assert len(out) == 1, "the echo of one outage must not read as three problems" + assert out[0]["id"].startswith("gateway:host:"), out[0]["id"] + + +def test_an_unconfigured_gateway_source_says_nothing_even_when_its_job_says_down() -> None: + """No `HOST_TARGETS` entry named `GATEWAY_HOST` means no job — and a reported one stays quiet.""" + out = alerts.build(Settings(gateway_interval=30.0), + [gateway_state(ok=False, configured=False, error="not configured")], + now=2_000.0) + assert out == [] + + +def test_a_gateway_payload_with_a_fabricated_verdict_alerts_nothing() -> None: + """The phase's no-verdict rule, asserted as a rule and not only as an absence. + + Three ways, because deleting a rule satisfies an absence-only assertion: + + * `alerts.build` takes no gateway payload at all — its signature is the whole surface, so a + gateway count can reach an alert only through a threshold added to this module; + * `severity.THRESHOLDS` has no `gateway.*` row, so no such threshold could be written without + editing the one file that owns thresholds (and a busy proxy is the job, the `net.*` case); + * the module's `HREF` does carry `/gateway`, so reachability still reaches the overview. + """ + assert not {"gateway", "gateway_payload"} & set(inspect.signature(alerts.build).parameters) + assert not [name for name in severity.THRESHOLDS if name.startswith("gateway.")], ( + "a connection count is not a graded number") + assert alerts.HREF["gateway"] == "/gateway" + assert alerts.AGENT_ECHO_DASHBOARDS == ("disks", "gateway") + # and an answering gateway with nothing else going on is the all clear + assert alerts.build(Settings(gateway_interval=30.0), [gateway_state()], [], [], None, [], + now=2_000.0) == [] + + # --------------------------------------------------------------------------- # # configuration # --------------------------------------------------------------------------- # diff --git a/tests/test_api.py b/tests/test_api.py index 01b6866..9472637 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -7,11 +7,12 @@ asset the network can fail to deliver. """ from __future__ import annotations +import inspect import ipaddress import json import re import time -from collections.abc import Iterator +from collections.abc import Iterator, Mapping from contextlib import contextmanager from pathlib import Path from typing import Any @@ -27,7 +28,7 @@ import store_perf from auth_client import sign_in from auth_contract import ADMIN_PASSWORD, FAST_PBKDF2 from card_contract import HOST_SHAPES, assert_card_contract -from dashboards import alerts, auth, certs +from dashboards import alerts, api, auth, certs from dashboards import app as app_module from dashboards.api import MIN_PASSWORD_LENGTH, RANGE_SECONDS, _disk_visible, parse_range from dashboards.app import create_app @@ -35,6 +36,7 @@ from dashboards.collectors import UpstreamError from dashboards.config import Settings, Target from dashboards.rates import MIN_WINDOW_SECONDS, RateEstimator from dashboards.severity import classify +from dashboards.sources import gateway as gateway_source from dashboards.sources import nut from dashboards.sources.disks import SMART_SERIES, DisksCollector, parse_disks_snapshot from dashboards.sources.host import HostCollector, parse_host_snapshot @@ -60,9 +62,12 @@ from slots_contract import ( DEAD_URL = "http://127.0.0.1:1" #: Every dashboard route, in nav order. The markup pins below (no CDN, no `.sev-*`, no -#: narration, no orphan `id`) run across this list rather than a hand-written four, so a -#: fifth page cannot join the masthead and skip the checks the others had to pass. -PAGES = ("/", "/system", "/disks", "/llm", "/ups") +#: narration, no orphan `id`) run across this list rather than a hand-written four, so a new page +#: cannot join the masthead and skip the checks the others had to pass. +#: `/gateway` joined it in phase 29 task 05, which is the moment it became the **sixth** dashboard +#: rather than a route with a page: this tuple is the tab table, so entering it means the tab, its +#: order, and its `aria-current` are pinned on every signed-in page from here on. +PAGES = ("/", "/system", "/disks", "/llm", "/ups", "/gateway") #: `/settings` is a **page** and not a dashboard (phase 11): it carries the same masthead with #: no `aria-current` tab, and the account link in every dashboard's masthead is how you reach @@ -72,8 +77,14 @@ PAGES = ("/", "/system", "/disks", "/llm", "/ups") SETTINGS_PAGE = "/settings" #: `/certs` is the second admin page (phase 15 task 05), kept out of `PAGES` for exactly the same #: reason: it is not a tab. It is in `MARKUP_PAGES` and `PAGE_SCRIPTS`, so the CDN, `.sev-`, -#: flavour-text and orphan-`id` audits cover it like the five dashboards. +#: flavour-text and orphan-`id` audits cover it like the six dashboards. CERTS_PAGE = "/certs" +#: The name the `/gateway` pins use (`PAGE_SCRIPTS`, its page-shape parametrises) now that the +#: tuple above owns its tab. +GATEWAY_PAGE = "/gateway" +#: Every page that renders the masthead while signed in: the six dashboards and the two admin +#: pages. The nav pins loop this list, which is why eight pages carry six tabs and exactly two of +#: them carry nothing current. MARKUP_PAGES = (*PAGES, SETTINGS_PAGE, CERTS_PAGE) @@ -276,8 +287,10 @@ def test_summary_shape_did_not_shrink(data_client) -> None: assert fleet_keys <= set(body["system"]["fleet"]), fleet_keys - set(body["system"]["fleet"]) assert host_keys <= set(body["system"]["hosts"][0]), host_keys - set(body["system"]["hosts"][0]) assert ups_keys <= set(body["ups"]["ups"][0]), ups_keys - set(body["ups"]["ups"][0]) - # `disks` joined in phase 09 and changed nothing else: the set grows, never shrinks - assert set(body) == {"ts", "system", "disks", "llm", "ups", "recent_events", "alerts"} + # `disks` joined in phase 09 and `gateway` in phase 29, and changed nothing else: the set + # grows, never shrinks + assert set(body) == {"ts", "system", "disks", "gateway", "llm", "ups", "recent_events", + "alerts"} # ...and the new keys are there beside the old ones, not instead of them assert "severity" in body["system"]["fleet"] assert "severity" in body["system"]["hosts"][0] @@ -287,8 +300,16 @@ def test_summary_shape_did_not_shrink(data_client) -> None: # `/` paints two panels; the payload is still the whole-fleet contract. Every path here is # read by something that is not the overview page: the live suite, an external probe, a # notifier later. Deleting a card must never delete a key. +#: `/gateway`'s entries are the phase-29 block, and they are listed rather than spot-checked +#: because two consumers read it: a notifier walking the totals, and the overview's one line +#: (`page-index.js`'s `GATEWAY_FIELDS` + `href`), which is pinned against this same document by +#: `test_the_overview_line_reads_the_gateway_block_it_claims`. SUMMARY_CONTRACT = ("system.enabled", "system.fleet", "system.hosts", "disks.enabled", "disks.fleet", "disks.hosts", + "gateway.configured", "gateway.host", "gateway.href", "gateway.supported", + "gateway.unreachable", "gateway.inbound", "gateway.outbound", + "gateway.map_targets", "gateway.unmatched", "gateway.unmatched_listeners", + "gateway.partial", "gateway.states", "llm.engines", "ups.ups", "recent_events", "alerts") @@ -326,6 +347,13 @@ def test_summary_serves_every_path_another_consumer_reads(request, fixture: str) assert {host["name"] for host in dig(body, "disks.hosts")} == names batteries = {item["name"] for item in dig(body, "ups.ups")} assert batteries == ({"apc", "cyberpower"} if populated else set()) + # the ingress block is present and empty in **both** fixtures: it needs a *name* among the + # targets (`GATEWAY_HOST`, default `gateway`), and `data_client` monitors `driveripper`. An + # unconfigured source is a block of nulls with no states, not a missing key and not a red row + # — and the nulls are nulls rather than `0`, because "no job ran" is not "no connections". + assert dig(body, "gateway.configured") is False + assert dig(body, "gateway.states") == [] + assert (dig(body, "gateway.inbound"), dig(body, "gateway.outbound")) == (None, None) assert dig(body, "llm.engines") == [], "neither fixture configures an engine" assert isinstance(dig(body, "recent_events"), list) assert isinstance(dig(body, "alerts"), list) @@ -589,6 +617,15 @@ def test_config_reports_every_source_as_off(client) -> None: assert body["disks"] == {"enabled": False, "interval_seconds": 300.0, "timeout_seconds": 30.0}, ( "the drive-health page is configured exactly when the host agents are") + assert body["gateway"] == { + "enabled": False, + # the default name is still printed when nothing matches it: "no host called `gateway` is + # being dialled" is the sentence, and the name is the half the operator has to change + "host": "gateway", + "url": None, + "interval_seconds": 30.0, + "timeout_seconds": 10.0, + }, "the ingress is configured by a name in HOST_TARGETS and by nothing else" assert body["llm"]["gufo"]["enabled"] is False assert body["llm"]["vllm"]["enabled"] is False assert body["ups"]["enabled"] is False @@ -961,7 +998,7 @@ def test_the_same_server_starts_once_the_environment_seeds_a_password(tmp_path: @pytest.mark.parametrize("path,title", [ ("/", "overview"), ("/system", "system"), ("/disks", "disks"), ("/llm", "llm"), - ("/ups", "ups"), + ("/ups", "ups"), (GATEWAY_PAGE, "gateway"), ]) def test_pages_serve_and_declare_themselves(client, path, title) -> None: response = client.get(path) @@ -1038,7 +1075,7 @@ def test_the_settings_page_is_not_one_of_the_dashboards(client) -> None: """The tabs are there to leave with; nothing marks itself current, because nothing is. A tab marked current on a page that is not in the tab list claims a location the app does not - have, and the four dashboards' own masthead pin (`aria-current` exactly once) relies on that + have, and the six dashboards' own masthead pin (`aria-current` exactly once) relies on that marker meaning something. """ html = client.get(SETTINGS_PAGE).text @@ -1047,7 +1084,7 @@ def test_the_settings_page_is_not_one_of_the_dashboards(client) -> None: for path in PAGES: assert f'href="{path}"' in html, f"/settings cannot navigate to {path}" # The fleet badges belong here too (`TODO.md:3`): the container is what made them appear on - # the five dashboards, and a page that renders the masthead without it is the page where they + # the six dashboards, and a page that renders the masthead without it is the page where they # "disappear". `Dash.initChrome()` fills it from `/api/health` exactly as it fills the others. assert 'id="nav-status"' in html, "/settings lost the fleet badges the operator asked for" @@ -1078,10 +1115,10 @@ def test_the_settings_page_refuses_a_caller_with_no_session(tmp_path: Path) -> N def test_every_page_links_to_both_admin_pages(client, tmp_path: Path) -> None: """`⚙ Account` and `⚷ Certs` are the only way to the two admin pages, so they are everywhere. - One loop over all seven signed-in pages (`MARKUP_PAGES`), because the control row is the *same* + One loop over all eight signed-in pages (`MARKUP_PAGES`), because the control row is the *same* on all of them: `⚷ Certs` → `⚙ Account` → theme toggle → clock, in `.masthead-right` and - **outside** `.tabs` (the tab strip's structure — five dashboards, one of them current — is pinned by - `test_the_disks_tab_is_in_the_masthead_of_every_page`, and a sixth entry with no + **outside** `.tabs` (the tab strip's structure — six dashboards, one of them current — is pinned + by `test_the_disks_tab_is_in_the_masthead_of_every_page`, and a seventh entry with no `aria-current` would break what that marker means). Both links are on the pages they name, which **inverts** phase 15's "no self-link" @@ -1154,14 +1191,16 @@ def test_every_page_links_to_both_admin_pages(client, tmp_path: Path) -> None: assert link not in login_html, f"/login offers {link} to a caller who cannot use it" -def test_the_certs_page_is_a_page_and_not_a_sixth_dashboard(client) -> None: +def test_the_certs_page_is_a_page_and_not_a_dashboard(client) -> None: """`/certs` borrows the dashboard chrome and none of the dashboard meaning (task 15.05). - Five tabs with nothing current, the four fleet badges like every other masthead (phase 25: + Six tabs with nothing current, the five fleet badges like every other masthead (phase 25: the badges report the fleet, and that report is the point of the site on this page too), and the same head and stylesheet as every other page — this is the `test_the_settings_page_is_not_one_of_the_dashboards` assertion for the second admin page, - and it is what a sixth tab in the strip would fail on. + and it is what a tab in the strip for the certificate authority would fail on. (Phase 29 + renamed it: `/gateway` **is** the sixth dashboard now, so "sixth" no longer means "not a + dashboard".) """ html = client.get(CERTS_PAGE).text assert html.count('aria-current="page"') == 0 @@ -1373,7 +1412,7 @@ def test_a_rotated_secret_clears_the_cached_session_every_page_uses(client) -> N ASSETS = ("/static/dash.css", "/static/app.js", "/static/charts.js", "/static/page-index.js", "/static/page-disks.js", "/static/page-system.js", "/static/page-llm.js", "/static/page-ups.js", "/static/page-login.js", - "/static/page-settings.js", "/static/page-certs.js") + "/static/page-settings.js", "/static/page-certs.js", "/static/page-gateway.js") def _css(client: TestClient) -> str: @@ -2556,6 +2595,21 @@ def test_sparkline_ships_the_fixed_scale_and_verdict_options(client) -> None: assert "transition:" not in body and "animation:" not in body +def test_a_flat_series_keeps_the_axis_floor_the_page_declared(client) -> None: + """`_yExtent()` pads a flat series so its line has a band to sit in, and a declared `yMin` + is not negotiable inside that padding. + + Every idle gateway pipe publishes `0`, so the first `/gateway` charts were flat and the old + `lo === hi` shortcut answered `[lo - 1, hi + 1]` whatever the page passed: the axis grew a + `-1` tick, which is a scale describing a number of connections that cannot exist — the same + mistake as printing `0` where nothing measured, wearing an axis instead of a cell. + """ + charts = _code(client.get("/static/charts.js").text) + extent = charts[charts.index("_yExtent()"):charts.index(" draw()")] + assert "this.opts.yMin !== null ? Math.min(lo, this.opts.yMin) : lo - pad" in extent, \ + "a flat series is padding below the floor the page declared" + + # ---- the battery glyph (phase 05 task 02) --------------------------------- # # `DashCharts.battery()` is the one visual in this app whose *shape* is the data: the fill @@ -3819,6 +3873,9 @@ PAGE_SCRIPTS = { "/disks": "page-disks.js", "/llm": "page-llm.js", "/ups": "page-ups.js", + # `/gateway` (phase 29 task 04): the pipe map is built entirely in JS over the payload, so + # every `id` the page ships is written by this script or by a stylesheet rule. + GATEWAY_PAGE: "page-gateway.js", # `/settings` is audited like a dashboard even though it is not one: an id nothing reads is # dead markup whatever page it sits on, and this page has five new ones. SETTINGS_PAGE: "page-settings.js", @@ -3940,6 +3997,11 @@ def test_the_copy_the_operator_acts_on_survives_the_audit(client) -> None: assert "No UPS reporting" in pages["/ups"] assert "instcmds" in pages["/ups"], "the UPS setup note stopped naming the grant that matters" assert "3493" in pages["/ups"], "the page stopped warning that upsd's port is not always 3493" + # …and the sixth dashboard's empty state names the two things that make it read at all: the + # `HOST_TARGETS` name of the machine that routes, and an agent new enough to have read the + # proxy configs off it. A page that said "nothing to show" without either is not actionable. + assert "GATEWAY_HOST" in pages[GATEWAY_PAGE] + assert "HOST_TARGETS" in pages[GATEWAY_PAGE] # …and the setup copy names the grant NUT actually checks. `README.md` is explicit: a # dashboard user gets `instcmds`, never `actions = -`, so a page that told the operator # to write that was sending them to fix the wrong line in `upsd.users`. @@ -4595,9 +4657,11 @@ def test_the_disks_surface_is_gets_only_and_the_write_stays_the_ups_one( if "GET" in (getattr(route, "methods", None) or set()) and not str(route.path).startswith("/api/")) # `/settings` (phase 11) is a fifth dashboard-shaped **page** and it grew no verb: a page - # route is a GET, so the write surface this test guards stays the one UPS command. - assert pages == ["/", "/certs", "/disks", "/llm", "/login", "/settings", "/system", - "/ups"] + # route is a GET, so the write surface this test guards stays the one UPS command. Same for + # `/gateway` (phase 29 task 04): the connection map is a read of one job's payload, and the + # app still dials an upstream on exactly one verb. + assert pages == ["/", "/certs", "/disks", "/gateway", "/llm", "/login", "/settings", + "/system", "/ups"] def test_config_prints_the_disks_cadence_the_page_explains_itself_with( @@ -4710,6 +4774,1257 @@ def test_the_metrics_endpoint_answers_for_the_disks_family(disks_client) -> None if name.startswith("disk.temp_c")], "SMART never mixes into the system family" +# ---- the `gateway` job: wiring, health, alerts, summary (phase 29 task 02) -- # +# +# The `gateway` source is a job of its own reading the same agent `/snapshot` the `host:` job reads +# (the `disks` precedent), so the pins below are about the wiring and the consequences: one job and +# only for the host `GATEWAY_HOST` names, the one certificate entry shared by all three of that +# host's jobs, an outage that stays one line on the overview, and a summary block that never turns +# an unknown into a zero. + +#: The host name every gateway test here uses — `GATEWAY_HOST`'s own default, so the settings line +#: under test is "a target whose name matches" and nothing else. +GATEWAY_TARGET = Target("gateway", "http://127.0.0.1:1") + + +def hub_of(tmp_path: Path, *, tls: Any | None = None, + **overrides: Any) -> tuple[Any, SampleStore]: + """A hub built exactly as the app builds it, with no event loop and nothing dialled. + + `build_hub` wires jobs and starts no task, so a shared client that is never used is enough — + which is also the point: a job that does not exist cannot open a socket. `tls` is passed + through as the app passes it, because the shared-entry property below is about that argument. + """ + db = str(tmp_path / "gateway-hub.sqlite3") + store = SampleStore(db) + settings = Settings(db_path=db, admin_password=ADMIN_PASSWORD, + pbkdf2_iterations=FAST_PBKDF2, **overrides) + hub, _monitor = app_module.build_hub(settings, store, httpx.AsyncClient(), tls=tls) + return hub, store + + +def test_a_configured_gateway_host_gets_exactly_one_job_of_its_own( + tmp_path: Path) -> None: + """One `gateway` job, on the gateway's clock, labelled with the host it reads. + + One and not one-per-host: the page is about the single ingress, and a job against every + machine would put a `gateway` health record on hosts that were never asked to route. Its + interval is `GATEWAY_INTERVAL_SECONDS`, not the 10 s system one — a connection count is the + shape of half a minute (invariant 4's isolation, the `disks` shape). + """ + hub, store = hub_of(tmp_path, + host_targets=(Target("deskwork", DEAD_URL), + Target("router", DEAD_URL), + Target("driveripper", DEAD_URL)), + gateway_host="router", gateway_interval=15.0) + jobs = [job for job in hub.jobs.values() if job.dashboard == "gateway"] + store.close() + assert [job.key for job in jobs] == ["gateway"], "the source is one job, not one per host" + assert jobs[0].label == "router", "the label is the host: an alert and an event print it" + assert jobs[0].interval == 15.0, "its cadence is GATEWAY_INTERVAL_SECONDS, not the host's" + assert jobs[0].enabled is True + assert sorted(hub.jobs) == ["disks:deskwork", "disks:driveripper", "disks:router", + "gateway", "host:deskwork", "host:driveripper", "host:router"], ( + "the gateway job is a third reader of one agent, not a replacement for either") + + +def test_a_gateway_name_that_is_not_a_target_gets_no_job_at_all(tmp_path: Path) -> None: + """Unconfigured means **no task**, not a task that reports itself red (invariant 3). + + `GATEWAY_HOST` defaults to `gateway`, and the trap is a deployment that monitors hosts with + other names: the default must not manufacture a source dialling a machine nobody configured. + """ + hub, store = hub_of(tmp_path, host_targets=(Target("deskwork", DEAD_URL),)) + keys = sorted(hub.jobs) + store.close() + assert "gateway" not in keys, keys + assert [key for key in keys if key.startswith("host:")] == ["host:deskwork"] + + +async def test_the_gateway_job_holds_the_same_credential_as_its_hosts_other_jobs( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """One `AgentTls` per host, handed to all three of its jobs. + + The dashboard has **one** identity, and the revocation answer is per name — so if the gateway + job were given a second, separately-built entry it could keep dialling a host the operator just + struck out (`collectors.AgentTls.status` is a live read, and `agent_tls_map` builds one entry). + Asserted on object identity, because a structurally-equal second entry would pass a field test + and still be a second `status` callable to go stale. + """ + seen: dict[str, Any] = {} + entry = app_module.AgentTls(verify="/ca.pem", cert=("/c.pem", "/k.pem"), + status=lambda: certs.ACTIVE) + + def record(name: str) -> Any: + async def scrape(client: Any, target: Target, *_args: Any, **kwargs: Any) -> Any: + from dashboards.poller import Result # noqa: PLC0415 - the fake's return shape + seen[f"{name}:{target.name}"] = kwargs.get("tls") + return Result(payload={}, samples=[]) + return scrape + + monkeypatch.setattr("dashboards.collectors.fetch_host_snapshot", record("system")) + monkeypatch.setattr("dashboards.collectors.fetch_disk_snapshot", record("disks")) + monkeypatch.setattr("dashboards.collectors.fetch_gateway_snapshot", record("gateway")) + hub, store = hub_of(tmp_path, host_targets=(Target("gateway", "https://gw.test"), + Target("deskwork", "http://desk.test")), + tls={"gateway": entry}) + try: + for job in hub.jobs.values(): + await job.fn() + finally: + store.close() + assert seen == {"system:gateway": entry, "disks:gateway": entry, "gateway:gateway": entry, + "system:deskwork": None, "disks:deskwork": None}, ( + "the ingress gets its host's entry and no other host gets one") + + +def test_the_gateway_job_is_handed_no_rate_estimator(tmp_path: Path) -> None: + """A count is not a rate, and the wiring is where that could silently break. + + `build_hub` spans two estimators for the two cadences that measure rates; the `gateway` job + joins neither, because a connection count pushed through a Δ-window becomes a rate with a + `Δ` label and no measurement behind it (rule 2). Asserted on the closure the factory + built: the only arguments it captured are the client, the target, the timeout and the TLS entry + — no estimator object. + """ + hub, store = hub_of(tmp_path, host_targets=(GATEWAY_TARGET,)) + job = hub.jobs["gateway"] + captured = dict(zip(job.fn.__code__.co_freevars, + tuple(c.cell_contents for c in job.fn.__closure__ or ()), strict=True)) + store.close() + assert captured, "the job's collector captured nothing: it cannot be dialling its own target" + assert not [name for name, value in captured.items() + if isinstance(value, RateEstimator)], "a count must not be handed a rate estimator" + assert set(captured) == {"client", "target", "settings", "entry"}, ( + "the gateway fetch takes a client, a target, its settings and one TLS entry — nothing else") + + +def make_gateway_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, + **overrides: Any) -> Any: + """An app whose only host is the ingress, on clocks fast enough to poll inside a test.""" + return build_client(tmp_path, monkeypatch, host_targets=(GATEWAY_TARGET,), + host_interval=0.05, disks_interval=0.05, gateway_interval=0.05, + **overrides) + + +def answer_as(client: TestClient, key: str, payload: dict[str, Any]) -> None: + """Give one job a collector that hands back `payload`, so its `ok` is truthful. + + The `healthy_client` idiom: a health record is overwritten only by a poll, so a fixture that + claims a source is answering swaps the poll's function rather than the record. A failed poll + keeps the last good payload, which is why a test wanting a *failing* source leaves the dead-port + `fn` in place and injects the payload as the last good one. + """ + from dashboards.poller import Result # noqa: PLC0415 - only these helpers need it + + async def collect() -> Result: + return Result(payload=payload, samples=[]) + + client.app.state.ctx.hub.jobs[key].fn = collect # type: ignore[attr-defined] + + +def gateway_block(client: TestClient) -> dict[str, Any]: + return client.get("/api/summary").json()["gateway"] + + +def test_the_summary_gateway_block_is_the_stub_of_a_source_that_never_answered( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Configured, dialling, nothing in yet: every count is `null` and the row says unreachable. + + The `system.hosts` stub rule (invariants 3 and 4): a source in the deployment that has not + answered is present with unknowns, because dropping it would read as "never configured" and + zeroing it would read as an idle gateway. The poll has to have happened for the stub to mean + what it says, so the test waits for the dead port's sentence first. + """ + with make_gateway_client(tmp_path, monkeypatch) as client: + wait_for_source_error(client, "gateway", "unreachable") + block = gateway_block(client) + assert block["configured"] is True + assert block["host"] == "gateway" + assert block["unreachable"] is True + assert block["supported"] is None, "nobody has answered, so nobody knows the version" + for field in ("inbound", "outbound", "map_targets", "unmatched", "unmatched_listeners"): + assert block[field] is None, f"{field} must be unknown, never 0" + assert [row["key"] for row in block["states"]] == ["gateway"] + assert block["states"][0]["dashboard"] == "gateway" + # the link travels even here: the overview renders it, and a block that grew `href` only + # when configured would make the page test for two shapes of the same row + assert block["href"] == GATEWAY_HREF + + +@pytest.mark.parametrize("mode", ["full", "predates", "sockets_silent"]) +def test_the_summary_gateway_block_is_the_payloads_sums_not_the_pages_arithmetic( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, gateway_snapshot: Any, + mode: str) -> None: + """The block re-derives nothing: each figure is the projection's own, in all three arms. + + Driven from `gateway.parse` over the **captured** configs and `/proc` tables rather than a + hand-written payload, so a rename in the source's shape fails here instead of printing a + plausible zero (rule 7). `partial` is the half that keeps a subtotal honest: a port whose + configured targets cannot be told apart leaves its pipes unknown, and the total says so. + """ + payload = gateway_source.parse(gateway_snapshot(mode), "gateway") + counting = (payload["sockets"]["established"] is not None + and payload["supported"] is not False) + with make_gateway_client(tmp_path, monkeypatch) as client: + answer_as(client, "gateway", payload) + deadline = time.time() + 10 + while time.time() < deadline and not client.app.state.ctx.hub.jobs["gateway"].state.ok: + time.sleep(0.05) + block = gateway_block(client) + + assert block["configured"] is True and block["unreachable"] is False + assert block["supported"] is payload["supported"] + assert block["host"] == "gateway" + if not payload["supported"]: + # nothing was read, so "no map targets" would be a claim about a file nobody opened + assert block["map_targets"] is None + assert block["inbound"] is block["outbound"] is None + assert block["unmatched"] is block["unmatched_listeners"] is None + return + assert block["map_targets"] == len(payload["nginx"]["map_targets"]) + for field, pipes in (("inbound", payload["inbound"]), ("outbound", payload["outbound"])): + known = [pipe["connections"] for pipe in pipes if pipe["connections"] is not None] + assert block[field] == (sum(known) if counting else None) + assert (block[field] is None) is (not counting), "unknown is not zero" + unknown = [pipe for pipe in payload["outbound"] if pipe["connections"] is None] + assert block["partial"] is (counting and bool(unknown)), ( + "a total that leaves an unknown pipe out has to say it did") + assert block["unmatched"] == ((payload["unmatched"]["inbound"] + payload["unmatched"]["outbound"]) + if counting else None) + assert block["unmatched_listeners"] == (payload["unmatched"]["other"] if counting else None) + + +def test_the_summary_gateway_block_of_the_captured_ingress_reads_the_live_join( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, gateway_snapshot: Any) -> None: + """The real capture, one figure at a time — the block an operator would actually get. + + The parametrised test above proves the rule; this one pins the numbers, so a change to the join + that moves a total is visible somewhere rather than only in a browser. + """ + with make_gateway_client(tmp_path, monkeypatch) as client: + answer_as(client, "gateway", gateway_source.parse(gateway_snapshot("full"), "gateway")) + deadline = time.time() + 10 + while time.time() < deadline and not client.app.state.ctx.hub.jobs["gateway"].state.ok: + time.sleep(0.05) + block = gateway_block(client) + assert block["inbound"] == 9, "9 of the 18 live sockets are inbound on a configured port" + assert block["outbound"] == 4 + assert block["map_targets"] == 17, "the stream map's routing entries are the fan-out" + assert block["unmatched"] == 5, "1 on an unclaimed listener, 4 nobody could attribute" + assert block["unmatched_listeners"] == 2 + assert block["partial"] is True, "the shared :443 group is unknown, and the sum says so" + assert "severity" not in block + assert block["href"] == GATEWAY_HREF + + +#: The route the ingress's own line links to, taken from the one table that writes a dashboard's +#: page (`alerts.py`) rather than from a string this file would have to keep in sync with it. +GATEWAY_HREF = alerts.HREF["gateway"] + + +def test_the_gateway_block_links_to_the_page_the_alert_row_links_to() -> None: + """One table says where a dashboard lives, and the summary block reads it out of that table. + + `/api/summary`'s `gateway.href` is what turns the overview's line into a link, and an alert + about the same source carries an `href` too (`test_a_gateway_job_that_fails_while_its_host_…`). + Two literals would be two chances to disagree, so `api._gateway_summary` reads + `alerts.HREF[GATEWAY_DASHBOARD]`, and this pins both halves: the value is a path of this app by + `auth.safe_next`'s own rule, and the derivation is still the table rather than a fresh string. + """ + assert GATEWAY_HREF == "/gateway" + assert auth.safe_next(GATEWAY_HREF) == GATEWAY_HREF, "the block's href is not a path of this app" + assert api._gateway_summary(None, Settings())["href"] == GATEWAY_HREF + assert "alert_rules.HREF[GATEWAY_DASHBOARD]" in inspect.getsource(api._gateway_summary), ( + "`_gateway_summary` stopped deriving `href` from `alerts.HREF` — the overview's link and " + "an alert about the same source can now spell the page differently") + + +# ---- alerts: one dead agent is one line, and a count is never a verdict ----- # + +def test_a_dead_gateway_host_is_one_overview_line_not_three( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Three jobs read one agent; a dead router is one outage (`AGENTS.md`, one line per fault). + + Without the echo rule the operator would get `gateway` three times on `/` — the host, its + drives, and its pipes — for the one fact that the machine is not answering. The `system` row + is the voice that stays, because it names the machine and links to the page that says most. + """ + with make_gateway_client(tmp_path, monkeypatch) as client: + deadline = time.time() + 15 + while time.time() < deadline: + rows = {row["key"]: row for row in client.get("/api/health").json()["sources"]} + if rows and all(not row["ok"] for row in rows.values()) and len(rows) == 3: + break + time.sleep(0.1) + else: + pytest.fail("the three jobs of one dead agent never all recorded a failure") + alerts = client.get("/api/summary").json()["alerts"] + assert len(alerts) == 1, alerts + assert alerts[0]["source"] == "gateway" + assert alerts[0]["href"] == "/system", "the outage is the host's, and /system is its page" + + +def test_a_gateway_job_that_fails_while_its_host_answers_is_the_only_voice_it_has( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The other arm: the agent answers and its proxy/socket half does not. + + Nothing else can say this — `/system` renders a healthy host, so a gateway job silenced by the + echo rule whenever its host is down would also be silenced here, and the fault would vanish. + The line links to `/gateway`, the page that explains which section went quiet. + """ + with make_gateway_client(tmp_path, monkeypatch) as client: + answer_as(client, "host:gateway", {"name": "gateway"}) + answer_as(client, "disks:gateway", {"name": "gateway", "disks": [], "summary": {}}) + deadline = time.time() + 15 + while time.time() < deadline: + rows = {row["key"]: row for row in client.get("/api/health").json()["sources"]} + if (rows.get("host:gateway") or {}).get("ok") and not (rows.get("gateway") or {}).get( + "ok", True): + break + time.sleep(0.1) + else: + pytest.fail("the gateway job never failed while its host answered") + alerts = client.get("/api/summary").json()["alerts"] + assert [item["source"] for item in alerts] == ["gateway"] + assert alerts[0]["href"] == "/gateway" + assert "unreachable" in alerts[0]["message"] + + +#: The two figures the overview's ingress line prints, in the order it prints them. Read back out +#: of the served `GATEWAY_FIELDS` and resolved against the block the route answers with, so a +#: rename on either side fails here instead of printing `n/a` beside a busy gateway. +GATEWAY_LINE_FIELDS = ("inbound", "outbound") + + +def test_the_overview_line_reads_the_gateway_block_it_claims(client) -> None: + """The sixth dashboard's one line on `/` indexes the summary block and nothing else. + + The `SLOT_FIELDS` device on the row phase 29 task 05 added: the served script's field table is + parsed and every name in it is resolved against the live `/api/summary` document, so a key that + quietly changes spelling fails a test rather than rendering `n/a` next to an ingress carrying + traffic. Three more claims ride on the same slice, because each is a rule the row could break + silently: the link is the block's own `href` (**no route literal in the renderer**, the rule the + alert rows keep), the row hides itself when `configured` is false instead of printing a red + nothing (invariant 3), and it carries no `severity` — a connection count is not a graded number + (`net.*` precedent), so a colour here would be a threshold invented in JavaScript. + """ + js = _code(client.get("/static/page-index.js").text) + table = re.search(r"const GATEWAY_FIELDS = \[(.*?)\n\];", js, re.S) + assert table, "page-index.js no longer has a GATEWAY_FIELDS table" + keys = re.findall(r"key: '(\w+)'", table.group(1)) + words = re.findall(r"word: '(\w+)'", table.group(1)) + assert keys == list(GATEWAY_LINE_FIELDS), f"the line prints {keys}, not the pinned pair" + assert words == list(GATEWAY_LINE_FIELDS), f"the line labels them {words}" + + body = client.get("/api/summary").json() + assert "gateway" in body, "/api/summary lost the block the line reads" + for key in (*keys, "configured", "href", "host", "unreachable", "partial"): + assert key in body["gateway"], f"the page names `{key}` and the block does not send it" + + line = js[js.index("function renderGateway("):] + line = line[:line.index("\n}")] + assert "block.href" in line, "the row stopped linking with the block's own href" + for literal in ("'/gateway'", '"/gateway"'): + assert literal not in line, f"the renderer hard-codes {literal} instead of reading `href`" + assert "if (block.configured !== true) {" in line and "row.hidden = true;" in line, ( + "an unconfigured ingress no longer renders no line — the disks block's invariant-3 rule") + assert "row.classList.toggle('stale'" in line, ( + "a gateway that stopped answering keeps its last counts, so the row must say they are old") + assert "severity" not in line, "the ingress line grew a verdict the block does not send" + assert "escapeHtml" in line, "a config string reached innerHTML unescaped" + + # …and the element it fills is in the fleet panel and ships hidden, so the row cannot appear + # before the payload says the source exists. + html = client.get("/").text + assert re.search(r']*id="fleet-gate"[^>]*hidden>', html), ( + "`/` has no hidden #fleet-gate for renderGateway() to reveal") + assert html.count(" None: + """The rule the phase wrote down as an assertion: `alerts.py` has no `gateway` threshold. + + Deliberately fed a payload with `severity` dicts stamped into every level of it — if a future + rule ever *read* a gateway verdict it would light up the overview over a busy proxy, which is + the job on a gateway and not a fault (`severity.py` rates no connection count, the `net.*` + precedent). An answering fleet with a fully-populated gateway payload stays all clear. + """ + payload = gateway_source.parse(gateway_snapshot("full"), "gateway") + payload["severity"] = {"inbound": "fail", "outbound": "fail", "unmatched": "fail"} + for pipe in (*payload["inbound"], *payload["outbound"]): + pipe["severity"] = "fail" + with make_gateway_client(tmp_path, monkeypatch) as client: + answer_as(client, "host:gateway", {"name": "gateway"}) + answer_as(client, "disks:gateway", {"name": "gateway", "disks": [], "summary": {}}) + answer_as(client, "gateway", payload) + deadline = time.time() + 15 + while time.time() < deadline: + rows = {row["key"]: row for row in client.get("/api/health").json()["sources"]} + if rows and all(row["ok"] for row in rows.values()): + break + time.sleep(0.1) + else: + pytest.fail("the three jobs never all reported ok") + body = client.get("/api/summary").json() + assert body["alerts"] == [], "a connection count is not a verdict from any direction" + assert "severity" not in body["gateway"] + + +def test_a_revoked_gateway_agent_fails_all_three_of_its_jobs_without_a_dial( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """`/certs` says revoke; the pipes, the drives and the host all stop talking to it. + + The phase-15 pin extended to the third job. What must not happen is a *socket*: the refusal + lives in `collectors.fetch_snapshot`, in front of the network, and it is reached through the + same `AgentTls` entry the host's other two jobs hold (`agent_tls_map`, built once at startup + with a live revocation read) — which is the only reason one click covers three sources. + """ + db = str(tmp_path / "revoked-gateway.sqlite3") + store = SampleStore(db) + settings = Settings(db_path=db, admin_password=ADMIN_PASSWORD, + pbkdf2_iterations=FAST_PBKDF2, + host_interval=0.05, disks_interval=0.05, gateway_interval=0.05, + host_targets=(Target(name="gateway", url="https://127.0.0.1:1"),)) + with TestClient(create_app(settings, store)) as client: + sign_in(client) + # the control: this is what a dial of a dead host says, so the sentence after it means + # something else than "the socket failed" + wait_for_source_error(client, "host:gateway", "unreachable") + state = certs.issue_agent_cert(store, certs.ensure_certs(store), "gateway", + ["127.0.0.1"]) + certs.revoke(store, state, "gateway") + + calls: list[str] = [] + + async def spy_get(self: Any, url: Any, **kwargs: Any) -> Any: + calls.append(str(url)) + raise AssertionError(f"a revoked agent was dialled at {url}") + + monkeypatch.setattr(httpx.AsyncClient, "get", spy_get) + for key in ("host:gateway", "disks:gateway", "gateway"): + error = wait_for_source_error(client, key, "agent certificate revoked") + assert "unreachable" not in error + assert calls == [], "the gate is in front of the network, not behind it" + messages = [row["message"] for row in client.get("/api/summary").json()["alerts"]] + assert sum(1 for text in messages if "certificate revoked" in text) == 1, ( + "one revoked agent is one line, whatever number of jobs read it") + store.close() + + +# ---- the `/api/gateway/*` routes (phase 29 task 03) ------------------------ # +# +# The page's two data routes, in the shape the `/disks` pair established: the configured gate runs +# before anything that could reach the network (both arms spy-pinned), an answering payload keeps +# the projection's exact keys, a source that has never answered is the `unreachable` stub of +# `null`s, and the history route lands on the store's one batched door and answers the points a +# direct `query()` read returns. A fifth data page grows no write of its own (invariant 1), which +# is what the verb enumeration below is for. + +# Every key `/gateway` indexes, in the arm that measured. Read off a live `parse` rather than +# hand-written, so a rename in `sources/gateway.py` fails the route pins instead of turning a +# card into `n/a` beside a healthy ingress. +def gateway_payload_keys(snapshot: Any) -> set[str]: + return set(gateway_source.parse(snapshot, "gateway")) + + +def find_keys(node: Any, name: str) -> list[Any]: + """Every value named `name` anywhere in a payload — the no-verdict audit.""" + if isinstance(node, Mapping): + found = [value for key, value in node.items() if key == name] + for value in node.values(): + found.extend(find_keys(value, name)) + return found + if isinstance(node, list): + return [hit for item in node for hit in find_keys(item, name)] + return [] + + +@pytest.fixture +def gateway_client(tmp_path: Path, monkeypatch: pytest.MonkeyPatch, + gateway_snapshot: Any) -> Iterator[TestClient]: + """The ingress answering with the captured configs and sockets, its gauges already stored. + + Both halves come from `sources/gateway` over the **captured** tree, so every path a page reads + and every series name a chart asks for is one the source really emits (rule 7). The target + points at a dead port: no poll can overwrite the payload, dial out, or add a second sample + between the store read and the route assertion below. + """ + db = str(tmp_path / "gateway-routes.sqlite3") + store = SampleStore(db) + app = create_app(Settings(db_path=db, admin_password=ADMIN_PASSWORD, + pbkdf2_iterations=FAST_PBKDF2, + host_targets=(GATEWAY_TARGET,)), store) + with TestClient(app) as test_client: + sign_in(test_client) + now = time.time() + snapshot = gateway_snapshot("full") + snapshot["ts"] = now # "the agent answered just now", so charts are in range + result = gateway_source.parse_gateway_snapshot("gateway", snapshot, now=now) + store.add_many(result.samples) + app.state.ctx.hub.jobs["gateway"].state.payload = result.payload + yield test_client + store.close() + + +def dial_spies(monkeypatch: pytest.MonkeyPatch, calls: list[str]) -> None: + """Two spies that raise: a route that dials before its gate fails here, not in a timeout. + + The `disks` pair's instrument, named once because three routes are now pinned with it (the + snapshot's two unconfigured arms and the history route's) and a copy per test is how one of + them stops raising. + """ + async def spy_get(self: Any, url: Any, **kwargs: Any) -> Any: + calls.append(f"GET {url}") + raise AssertionError(f"the unconfigured gateway route dialled {url}") + + def spy_fetch(*args: Any, **kwargs: Any) -> Any: + calls.append("fetch_snapshot") + raise AssertionError("the unconfigured gateway route fetched an agent snapshot") + + monkeypatch.setattr(httpx.AsyncClient, "get", spy_get) + monkeypatch.setattr("dashboards.collectors.fetch_snapshot", spy_fetch) + + +def test_gateway_snapshot_404s_when_no_host_is_named_the_gateway( + client, monkeypatch: pytest.MonkeyPatch) -> None: + """`GATEWAY_HOST` unset and no targets: `404 {"configured": false}`, nothing dialled. + + Invariant 3, and the arm a deployment without an ingress lives in. The spies raise, so a route + that built a client or fetched a snapshot before testing `gateway_enabled` fails here instead + of costing a connect timeout and a red dot, and the empty hub is the structural half: with no + targets there is no `gateway` job that could age out. + """ + calls: list[str] = [] + dial_spies(monkeypatch, calls) + + response = client.get("/api/gateway/snapshot") + assert response.status_code == 404 + body = response.json() + assert body["configured"] is False + assert "GATEWAY_HOST" in body["detail"], "the sentence names the fix" + assert calls == [], "an unconfigured page must not touch the network" + assert client.app.state.ctx.hub.jobs == {}, ( # type: ignore[attr-defined] + "no targets means no gateway job, so nothing can age out or go red") + + +@pytest.mark.parametrize("gateway_host", [None, "router"]) +def test_gateway_snapshot_404s_when_the_name_is_not_a_target( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch, gateway_host: str | None) -> None: + """A name with no agent behind it is unconfigured, whatever the reason for the mismatch. + + `None` is the trap this default creates: `GATEWAY_HOST` defaults to `gateway`, so a deployment + that monitors `deskwork` only must not grow a source dialling a machine nobody configured. + `"router"` is the opposite typo — a name set on purpose that the target list does not hold. + Both answer the same refusal with the same no-dial proof. + + The two host jobs' first polls are awaited **before** the spies go in: the hub polls once the + moment it starts, and a spy that catches `host:deskwork` dialling its own dead target would be + a failure about a job this route never touched. With both intervals at an hour, the refusal is + measured against a hub that has already said everything it has to say. + """ + overrides: dict[str, Any] = {"host_targets": (Target("deskwork", DEAD_URL),), + "host_interval": 3_600.0, "disks_interval": 3_600.0} + if gateway_host is not None: + overrides["gateway_host"] = gateway_host + with build_client(tmp_path, monkeypatch, **overrides) as client: + wait_for_source_error(client, "host:deskwork", "unreachable") + wait_for_source_error(client, "disks:deskwork", "unreachable") + calls: list[str] = [] + dial_spies(monkeypatch, calls) + + body = client.get("/api/gateway/snapshot").json() + assert body["configured"] is False + assert "GATEWAY_HOST" in body["detail"], "the sentence names the fix" + assert calls == [], "the gate is in front of the network, not behind it" + jobs = client.app.state.ctx.hub.jobs # type: ignore[attr-defined] + assert "gateway" not in jobs, "an unconfigured name grew a job that could look red" + assert sorted(jobs) == ["disks:deskwork", "host:deskwork"] + + +def test_the_gateway_gate_keys_on_the_name_and_not_on_a_default( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """The control for both refusals above: name the host that **is** a target and it opens. + + Same fleet, one changed setting, and the route answers — with the stub, because the target is + still a dead port. Without this leg a broken router would look identical to a correct gate. + """ + with build_client(tmp_path, monkeypatch, host_targets=(Target("deskwork", DEAD_URL),), + gateway_host="deskwork", host_interval=3_600.0) as client: + body = client.get("/api/gateway/snapshot").json() + assert body["configured"] is True and body["unreachable"] is True + assert body["host"] == "deskwork" + assert [row["key"] for row in body["states"]] == ["gateway"] + + +def test_gateway_snapshot_answers_the_pipes_the_projection_joined( + gateway_client, gateway_snapshot: Any) -> None: + """The answering arm is `sources/gateway.parse`'s payload, key for key, plus `states`. + + Two things are pinned at once. The **shape**: every key the page indexes is there, so a rename + in the source fails this test rather than printing `n/a` beside a live ingress. And the + **reconciliation identity** the phase wrote down — the pipes plus everything unmatched equal the + kernel's ESTABLISHED row count — because that is what makes the counts on screen provable + against a direct `/proc/net/tcp` read instead of a plausible drawing. + """ + body = gateway_client.get("/api/gateway/snapshot").json() + assert set(body) - {"states"} == gateway_payload_keys(gateway_snapshot("full")), ( + "the route grew or lost a top-level key of the source's shape") + assert body["configured"] is True and body["supported"] is True + assert body["host"] == "gateway" + assert body["why"] is None, "a page with pipes has nothing to apologise for" + assert body["sections"] == {"haproxy": True, "nginx": True, "sockets": True} + + # the config half, verbatim from the capture the agent read (phase 28's six frontends, the + # stream map's 17 routing entries) — the echo the pipes were built from + assert len(body["haproxy"]["frontends"]) == 6 + assert len(body["nginx"]["map_targets"]) == 17 + stream = next(pipe for pipe in body["inbound"] if pipe["port"] == 443) + assert stream["connections"] == 6 and stream["metric"] == "gateway.in_conns@443", ( + "the pipe carries the series it is, so the page never rebuilds the name") + assert stream["clients"], "the inbound pipe names the addresses that are using it" + secure = next(pipe for pipe in body["inbound"] if pipe["port"] == 4443) + assert "nextcloud.reeseapps.com" in secure["domains"], "the listen port carries its server_names" + shared = [pipe for pipe in body["outbound"] if pipe["target"]["port"] == 443] + assert shared and all(pipe["connections"] is None for pipe in shared), ( + "a port several configured targets share is unknown on every one of them, never a guess") + + known_in = sum(pipe["connections"] for pipe in body["inbound"] + if pipe["connections"] is not None) + known_out = sum(pipe["connections"] for pipe in body["outbound"] + if pipe["connections"] is not None) + unmatched = body["unmatched"] + assert known_in + known_out + unmatched["inbound"] + unmatched["outbound"] == \ + body["sockets"]["established"], "every socket row is counted exactly once" + assert [row["key"] for row in body["states"]] == ["gateway"] + + +def test_the_gateway_snapshot_carries_no_verdict_and_no_window(gateway_client) -> None: + """A connection count is neither a graded number nor a rate, at any depth of the payload. + + `severity.py` rates no `gateway.*` metric and the source is handed no `RateEstimator`, so the + route must not grow either either: a `severity` anywhere would colour a busy proxy (the + phase-06 GPU reasoning, applied to the ingress) and a `window_seconds` would claim a Δ was + measured when nothing was (rule 2). + """ + body = gateway_client.get("/api/gateway/snapshot").json() + assert find_keys(body, "severity") == [], "no count on this page is graded" + assert find_keys(body, "window_seconds") == [], "nothing here was measured over a window" + + +def test_gateway_snapshot_stubs_the_ingress_that_has_never_answered( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + """Configured, dialling, nothing in yet: the row stays and every count is unknown. + + Invariant 4's page-side half, the `system.hosts` / `/api/disks` stub rule: dropping the host + would read as "never configured" and zeroing it would read as an idle gateway, so the answer + is `unreachable: true` with `null` counts and the `gateway` health record beside them. The + stub also has to carry **every key the answering payload carries** (plus its own + `unreachable`), or the page that renders both arms has to guess which keys exist. + """ + with make_gateway_client(tmp_path, monkeypatch) as client: + wait_for_source_error(client, "gateway", "unreachable") + body = client.get("/api/gateway/snapshot").json() + + assert body["configured"] is True, "the source is in the deployment" + assert body["unreachable"] is True + assert body["host"] == "gateway", "the name is a config fact, so it survives the outage" + assert body["supported"] is None, "nobody has read its version" + assert body["inbound"] == [] and body["outbound"] == [] + assert body["haproxy"]["frontends"] == [] + assert body["nginx"] == {"servers": [], "map_targets": []} + assert body["sockets"] == {"listens": None, "established": None}, "unknown, never 0" + assert body["unmatched"]["inbound"] is None + assert body["unmatched"]["outbound"] is None + assert body["unmatched"]["other"] is None + assert body["why"] and "not answered" in body["why"], "a sentence the page can print" + assert body["ts"] is not None, "the age of the attempt is what the page prints" + assert [row["key"] for row in body["states"]] == ["gateway"] + + +def test_the_gateway_stub_and_the_answer_share_one_key_set(gateway_client) -> None: + """The two arms differ in exactly two keys, so no page branch can index a missing one. + + The stub cannot publish an `agent_version` (nobody answered, so nobody said) and must publish + its own `unreachable`. Everything else — every section, pipe list, tally and socket count the + answering arm carries — is there in both, which is what lets `/gateway` render one card body + for a live ingress and for a host that has never replied. + """ + answered = set(gateway_client.get("/api/gateway/snapshot").json()) - {"states"} + assert set(gateway_source.stub("gateway", ts=1.0)) == ( + answered - {"agent_version"} | {"unreachable"}) + + +def test_gateway_history_serves_the_pipe_gauges_the_store_holds( + gateway_client) -> None: + """The points are the stored rows — read a third way, through `store.query`. + + Phase 14 task 02's pin shape: the route goes through `_history_batch` → `store.history_many` + and the test reads `store.query`, and neither is the per-series loop the batch replaced, so an + "optimisation" that moves, drops or reorders a point fails here. Both gauge families are read, + the outbound one **with its port** in the name (two Minecraft backends on one host are two + facts), and the `source` says the `gateway` family — a page must never ask the 10-second `host` + series for a connection count. + """ + store: SampleStore = gateway_client.app.state.ctx.store # type: ignore[attr-defined] + body = gateway_client.get("/api/gateway/history", params={ + "hosts": "gateway", + "metrics": "gateway.in_conns@443,gateway.in_conns@4443," + "gateway.out_conns@nextcloud.reeselink.com:11000", + "range": "1h"}).json() + + assert body["hosts"] == ["gateway"] + assert body["range_seconds"] == 3_600.0 + assert [series["metric"] for series in body["series"]] == [ + "gateway.in_conns@443", "gateway.in_conns@4443", + "gateway.out_conns@nextcloud.reeselink.com:11000"], "host-major, metric-minor, request order" + assert series_source(body) == {"gateway"}, "its own family, never the system one" + for series in body["series"]: + rows = store.query("gateway", "gateway", series["metric"], res=RES_RAW) + assert rows, f"{series['metric']} has no stored row: the fixture wrote nothing" + assert [(point["ts"], point["value"]) for point in series["points"]] == rows + + +def test_gateway_history_answers_empty_for_a_pipe_it_never_measured(gateway_client) -> None: + """Unknown names are an empty series, not a 404 — the contract the other routes have. + + A card asks for every pipe it could draw in one request: a target with no traffic in the + window, a port the configs stopped naming, and a name nobody ever stored all draw an empty + plot while the series that do exist still answer. + """ + body = gateway_client.get("/api/gateway/history", params={ + "hosts": "gateway", + "metrics": "gateway.in_conns@9999,gateway.out_conns@nowhere.test:1," + "gateway.in_conns@443", + "range": "24h"}).json() + by_metric = {series["metric"]: series["points"] for series in body["series"]} + assert by_metric["gateway.in_conns@9999"] == [] + assert by_metric["gateway.out_conns@nowhere.test:1"] == [] + assert by_metric["gateway.in_conns@443"], "the series that does exist still answers" + assert series_source(body) == {"gateway"} + + +def test_gateway_history_names_the_ingress_and_no_other_host(gateway_client) -> None: + """With no `hosts=` the answer is the one host this source reads — not the whole fleet. + + The gauges are stored under the `GATEWAY_HOST` name, so defaulting to every `HOST_TARGETS` + entry would draw a card of empty plots for machines that were never asked to route, and + defaulting to nothing would leave the page's disclosure with nothing to ask for. + """ + body = gateway_client.get("/api/gateway/history", + params={"metrics": "gateway.in_conns@22"}).json() + assert body["hosts"] == ["gateway"] + assert body["range_seconds"] == 86_400.0, "the page's default window, like /disks" + assert [(series["host"], series["metric"]) for series in body["series"]] == [ + ("gateway", "gateway.in_conns@22")] + assert body["series"][0]["points"], "the gauge was stored under that very name" + + +def test_gateway_history_dials_nothing_when_the_gateway_is_unconfigured( + client, monkeypatch: pytest.MonkeyPatch) -> None: + """History reads only SQLite, so an unconfigured ingress costs no dial and no error. + + The `/api/disks/history` rule: the configured gate belongs to the route that could reach the + network, and a page that asks for history anyway gets no hosts and no series rather than a + 404 it has to special-case. The spies are the proof, not the empty body — an empty answer that + had dialled would be the invariant-3 failure with a 200 attached. + """ + calls: list[str] = [] + dial_spies(monkeypatch, calls) + body = client.get("/api/gateway/history", + params={"metrics": "gateway.in_conns@443"}).json() + assert body["hosts"] == [] + assert body["series"] == [] + assert calls == [] + + +def test_the_metrics_endpoint_answers_for_the_gateway_family(gateway_client) -> None: + """"Why is this chart empty" has an answer per page: /api/metrics?source=gateway. + + The fifth family stays out of the 10-second `host` list, which is the other half of the same + promise — a connection count charted from the system series would be a number from a source + that never measured it. + """ + names = gateway_client.get("/api/metrics", params={"source": "gateway"}).json()["metrics"] + assert "gateway.in_conns@443" in names + assert not [name for name in names if not name.startswith("gateway.")], ( + "the gateway family holds only gateway series") + assert not [name for name in gateway_client.get( + "/api/metrics", params={"source": "host"}).json()["metrics"] + if name.startswith("gateway.")], "a pipe's gauge never mixes into the system family" + + +def test_the_gateway_surface_is_gets_only_and_the_write_stays_the_ups_one( + gateway_client) -> None: + """A fifth data page grows no write: the app's one upstream POST is still UPS's (invariant 1). + + The read-only half of the audit grows with every surface, and this is the half that catches a + POST arriving quietly — the two gateway routes are GETs, they refuse every other verb, and the + app-wide write table is still the UPS command plus the session and certificate-authority rows + that touch only this process. + """ + routes = flat_routes(gateway_client.app) + gate = [route for route in routes + if str(getattr(route, "path", "")).startswith("/api/gateway")] + assert {route.path for route in gate} == {"/api/gateway/snapshot", + "/api/gateway/history"} + verbs = {verb for route in gate for verb in (getattr(route, "methods", None) or set())} + assert verbs == {"GET"}, "a write appeared on the connection-topology surface" + for verb in ("POST", "PUT", "PATCH", "DELETE"): + response = gateway_client.request(verb, "/api/gateway/snapshot", json={}) + assert response.status_code == 405, f"{verb} /api/gateway/snapshot was accepted" + + writes = sorted(str(route.path) for route in routes + if "POST" in (getattr(route, "methods", None) or set())) + upstream = [path for path in writes if not path.startswith(("/api/auth/", "/api/certs/"))] + assert upstream == ["/api/ups/command"], "one write endpoint, four gates, unchanged" + + +# ---- the `/gateway` page (phase 29 task 04) -------------------------------- # +# +# The fifth dashboard, asserted the way the other four are: on the served bytes, because a JS +# test runner would be a build step (a locked anchor). What the page *is* decides what is worth +# pinning. It draws **pipes**, and a pipe's only claim is a connection count, so the pins are: the +# geometry animates while that count is at least one and rests at zero (a static drawing of a live +# connection reads as idle, and an animated zero reads as traffic — both are the lie this page can +# tell); every pipe says what it is to assistive tech, because a moving line says nothing to a +# screen reader; nothing is coloured by a verdict, because `severity.py` rates no `gateway.*` +# metric and the API sends none; every label is the config's own text, so a rename in +# `sources/gateway.py` fails here instead of printing `n/a` beside a healthy ingress; and the +# unreconciled remainder is printed, because the operator's fact is that the gateway talks to +# something the configs do not name. `data/logs/29-task04-pipes.cjs` is the browser half. + +#: Every payload path the page's script indexes, in the arm that measured. Resolved against a +#: live `GET /api/gateway/snapshot` below rather than written from the source, so the two sides +#: of the join are checked against each other (`SLOT_FIELDS`' / `COLUMNS`' device). +GATEWAY_PAGE_FIELDS = ( + "host", "ts", "supported", "why", "agent_version", + "sockets.listens", "sockets.established", + "inbound.port", "inbound.source", "inbound.domains", "inbound.connections", + "inbound.clients", "inbound.metric", + "outbound.target.host", "outbound.target.port", "outbound.target.name", + "outbound.target.by_port", "outbound.connections", "outbound.source", + "outbound.metric", + "unmatched.other", "unmatched.inbound", "unmatched.outbound", + "nginx.map_targets.key", "nginx.map_targets.host", "nginx.map_targets.port", + "haproxy.frontends.name", "haproxy.frontends.port", "haproxy.frontends.backend", +) + +#: The card families, in the order the page ships them. `unmatched` draws no pipes and so has no +#: chart; the other three are `source` words the payload itself uses. +GATEWAY_CARD_KEYS = ("haproxy", "nginx", "map", "unmatched") + + +def _payload_field(node: Any, path: str) -> tuple[bool, list[Any]]: + """`(found, values)` for one dotted path, where a segment naming a list maps over it. + + `found` is False only when a **key** is missing, which is the rename this checks for. A null + value is a found key: the arms that did not measure publish nulls on purpose (rule 2), and the + page's job is to render them as `n/a`, which the arms test below pins separately. + """ + segments = path.split(".") + if not segments[0]: + return False, [] + head, rest = segments[0], segments[1:] + if isinstance(node, list): + values: list[Any] = [] + for item in node: + found, mapped = _payload_field(item, path) + if not found: + return False, [] + values.extend(mapped) + return True, values + if isinstance(node, dict): + if head not in node: + return False, [] + if not rest: + return True, [node[head]] + return _payload_field(node[head], ".".join(rest)) + return False, [] + + +def test_the_gateway_page_is_a_card_per_source_family_with_a_reconciliation_row( + client) -> None: + """The page's shape, on the bytes it serves. + + Four cards keyed on `data-gate` — three config families plus `unmatched` — one node and two + pipe columns inside each of the three that draws pipes, and the socket table's tallies in the + page head so the page prints `listeners` / `established` beside what its pipes carry instead + of asserting the two agree. There is no `#range` control and that is a decision, not an + omission: the history here is a count trend and one window for every card is the honest + default for a source that re-reads every 30 seconds. + """ + html = _code(client.get(GATEWAY_PAGE).text) + for key in GATEWAY_CARD_KEYS: + assert f'data-gate="{key}"' in html, f"/gateway has no {key} card" + for key in GATEWAY_CARD_KEYS[:3]: + card = html[html.index(f'data-gate="{key}"'):] + card = card[:card.index("")] + assert 'class="pipe-columns"' in card, f"the {key} card has no pipe frame" + assert 'data-k="node"' in card, f"the {key} card has no gateway node" + for column in ("in", "out"): + assert f'data-k="{column}"' in card, f"the {key} card has no {column}bound column" + assert 'data-chart="conns"' in card, f"the {key} card has no count plot" + for tally in ("t-listeners", "t-established", "t-inbound", "t-outbound", "t-unmatched"): + assert f'id="{tally}"' in html, f"/gateway lost the {tally} reconciliation cell" + assert 'id="range"' not in html, "/gateway grew window chips over a gauge that has no rate" + assert 'src="/static/page-gateway.js"' in client.get(GATEWAY_PAGE).text + + +def test_the_gateway_page_prints_every_field_the_projection_publishes( + gateway_client) -> None: + """Every label the page builds is a key the payload really sends, resolved over HTTP. + + This is the rename catcher the slot grid and the `/certs` table already have, and the gateway + page needs it more than either: its labels are joined out of three different structures (a + pipe's `target`, the stream map's `key`, a group's `hosts`), so a rename in one of them would + otherwise draw a plausible pipe with the wrong host on it — worse than `n/a`, because nothing + about the picture would look broken. + """ + payload = gateway_client.get("/api/gateway/snapshot").json() + assert payload["supported"] is True and payload["host"] == "gateway" + for path in GATEWAY_PAGE_FIELDS: + found, _values = _payload_field(payload, path) + assert found, f"/gateway indexes {path}, which the payload does not publish" + + # The three families the cards are keyed on are the payload's own `source` words, and the + # card table would silently lose a family if `sources/gateway.py` ever renamed one. + sources = {pipe["source"] for pipe in payload["inbound"]} + sources |= {pipe["source"] for pipe in payload["outbound"]} + assert sources == set(GATEWAY_CARD_KEYS[:3]), \ + f"the payload's `source` words are {sorted(sources)} and the page ships {GATEWAY_CARD_KEYS[:3]}" + + +def test_every_pipe_is_an_accessible_image_built_from_the_payload(client) -> None: + """Each pipe is `role="img"` + an `aria-label` naming its port or target and its count. + + The battery glyph's convention (`DashCharts.battery`), for the same reason: the pipe carries + information the markup does not say in words, and here it is worse — a pipe is a *line*, so + without a name a screen reader gets a card that says `haproxy` and nothing about the six + connections on it. Built with `createElementNS` + `setAttribute`, never `innerHTML`, because + the domains and client addresses in that sentence are upstream strings. + """ + js = _code(client.get("/static/page-gateway.js").text) + assert "createElementNS(SVG_NS" in js, "the pipe is no longer hand-rolled SVG" + assert "http://www.w3.org/2000/svg" in js + pipe = js[js.index("function buildPipe("):js.index("function pipeRow(")] + assert "createElementNS" in pipe and "'path'" in pipe + rendered = js[js.index("function renderPipeRow("):js.index("function renderPipes(")] + assert "svg.setAttribute('role', 'img')" in rendered + assert "aria-label" in rendered and "pipeAria(" in rendered + aria = js[js.index("function pipeAria("):js.index("function buildPipe(")] + for fact in ("pipe.port", "pipe.connections", "pipe.domains", "pipe.clients"): + assert fact in aria, f"the pipe's accessible name stopped saying {fact}" + assert "connections unknown" in aria, \ + "a pipe whose join did not decide must not be announced as a zero" + # upstream text reaches the page through textContent and title, and nothing else + assert "innerHTML" not in rendered + assert "textContent" in js and "escapeHtml" not in js, \ + "the page hand-escapes strings it only ever assigns through textContent" + + +def test_a_pipe_moves_only_while_it_carries_a_connection(client) -> None: + """The animation is the connection, and nothing else. + + Three declarations and each is one half of the claim: `stroke-dasharray` + an `animation` on + the **live** rule and not the base one (so a pipe with no connections cannot inherit motion), + a keyframe whose only property is `stroke-dashoffset` (the dash travelling is the whole + animation — colour is not part of it, because colour would be a verdict), and a static state + that declares no animation and is drawn faint. `vector-effect="non-scaling-stroke"` is in the + script: the pipe is stretched to the card's width, and without it the stroke width would scale + with it and a wide card would draw a thick pipe. + """ + section = _code(_css_section(_css(client), "gateway pipes")) + live = _declarations(section, ".pipe.live .pipe-path") + assert live.get("animation", "").startswith("pipe-flow"), \ + "the animated state no longer runs the flow animation" + assert _declarations(section, ".pipe-path").get("animation") is None, \ + "motion moved onto the base rule: every pipe animates, idle ones included" + still = _declarations(section, ".pipe.still .pipe-path") + assert still.get("animation") is None, "the static state animates — zero is not traffic" + assert still.get("stroke-dasharray"), "the static pipe lost its dotted rest state" + keyframes = re.search(r"@keyframes\s+pipe-flow\s*\{(.+?)\n\}", section, re.S) + assert keyframes, "the flow animation has no keyframes to run" + body = keyframes.group(1) + assert body.count("stroke-dashoffset") == 2 and "stroke:" not in body, \ + "the keyframe must travel the dash and nothing else — animating colour is a verdict" + + js = _code(client.get("/static/page-gateway.js").text) + assert "svg.classList.toggle('live', live)" in js + assert "svg.classList.toggle('still', !live)" in js + assert "pipe.connections > 0" in js, \ + "the live test is no longer the connection count itself" + assert 'vector-effect' in js + + +def test_the_pipe_animation_stills_in_both_states_under_reduced_motion(client) -> None: + """The one reduced-motion block names `.pipe.live` **and** `.pipe.still`. + + The menu's pair above (phase 20 task 03) is the precedent and the reason: the animated rule + out-specifies the base one, so stilling only the static state would leave every busy pipe + flowing for someone who asked for no motion. Same single block, because two blocks drift. + """ + code = _code(_css(client)) + block = re.search(r"@media\s*\(prefers-reduced-motion:\s*reduce\)\s*\{(.*?)\n\}", + code, re.S) + assert block, "the stylesheet lost its reduced-motion block" + selector = re.search(r"(\.pipe\.live[^{]*)\{", block.group(1)) + assert selector, "the reduced-motion block stopped naming the pipes at all" + named = {" ".join(part.split()) for part in selector.group(1).split(",")} + assert named == {".pipe.live .pipe-path", ".pipe.still .pipe-path"}, ( + "the block must name both pipe states — naming only the static one leaves a live pipe " + f"animating for someone who asked for no motion; got {sorted(named)}") + assert _declarations(block.group(1), ".pipe.still .pipe-path").get("animation") == "none", \ + "the named rules stopped stilling the animation" + + +def test_no_gateway_count_is_coloured_by_a_verdict(client) -> None: + """The `net.*` precedent, applied to the fifth dashboard. + + A busy ingress is the job on this fleet, `severity.py` has no `gateway.*` row, and + `/api/gateway/snapshot` publishes no `severity` at any depth (task 03 pins that over HTTP); + so the page may not colour a pipe, a count, or a tally. What is pinned is the absence of the + whole vocabulary — the tokens, the class names, and the two JS helpers that would have applied + a verdict — across the stylesheet section, the script and the markup, comments stripped + because prose is allowed to *name* a colour the code must not paint. + + The reachability **dot** is the one thing here that reads `ok|warn|fail`, and it is not a + verdict on a number: it is the job's health, the same dot every other page's card header + draws from the same `states` row. + """ + section = _code(_css_section(_css(client), "gateway pipes")) + for token in ("--ok", "--warn", "--fail", ".ok", ".warn", ".fail", "severity"): + assert token not in section, f"the gateway stylesheet section paints with {token}" + js = _code(client.get("/static/page-gateway.js").text) + for helper in ("severityFrom", "severityFor", ".severity"): + assert helper not in js, f"/gateway grades a number with {helper}" + assert "setDot(" in js, "and the reachability dot is still drawn (that one is health)" + html = _code(client.get(GATEWAY_PAGE).text) + for token in ('class="ok', 'class="warn', 'class="fail', "severity"): + assert token not in html, f"/gateway markup carries {token}" + + +def test_gateway_charts_are_collapsed_lazy_and_cost_one_history_request_per_card( + client) -> None: + """`/system`'s chart discipline, per card: shut, built on open, one request per refresh. + + Three separate things, pinned because `/system` and `/disks` each lost one of them in review. + The disclosure ships without `open`; `drawCardCharts` returns while it is shut, so no + `LineChart` exists and no SVG is painted for a plot nobody is looking at (`LineChart` measures + its container, and a hidden one is 0 px wide); and `renderCardCharts` fires once per visible + card per refresh naming every gauge that card's pipes publish — the endpoint answers an + unknown name as an empty series on purpose, so asking for the whole card costs nothing. + """ + html = client.get(GATEWAY_PAGE).text + assert '
    s.points && s.points.length)" in js, \ + "the legend is naming pipes the plot never drew — a permanent `n/a` row per map target" + + +def test_the_unmatched_remainder_is_printed_and_never_folded_into_a_total( + client) -> None: + """The operator's fact: the gateway talks to something the configs do not name. + + Three cells and never one sum, because `other` counts **ports** while the other two count + **connections** — adding them would be a number with no unit. The `groups` list is the + "could belong to any of 16 targets" sentence, built node by node out of the payload (the + hostnames are upstream strings), and the page hides the whole card when nothing is left over + rather than keeping a permanent row of zeroes that trains the eye past the one that matters. + """ + html = _code(client.get(GATEWAY_PAGE).text) + card = html[html.index('data-gate="unmatched"'):] + card = card[:card.index("")] + for key in ("um-other", "um-in", "um-out", "groups"): + assert f'data-k="{key}"' in card, f"the unmatched card lost its {key} field" + assert "listeners no config names" in card + assert "connections inbound" in card and "connections outbound" in card + + js = _code(client.get("/static/page-gateway.js").text) + group = js[js.index("function groupRow("):js.index("function renderUnmatched(")] + assert "createElement('li')" in group and "innerHTML" not in group + assert "host_count" in group and "could be any of" in group + visible = js[js.index("function unmatchedVisible("):js.index("function groupRow(")] + assert "un.other" in visible and "un.inbound" in visible and "un.outbound" in visible + assert "> 0" in visible, "the card is no longer gated on there being something unaccounted for" + assert "== null" not in visible, \ + "an all-unknown remainder opened a card of three n/as — the why-line already says that" + + # …and the same two rules one level up, in the header's pipe totals: an unknown pipe adds + # nothing to the number and the cell says so (`/llm`'s fleet block drawing `/api/summary`'s + # `partial` flag), while a side where nothing measured at all has no total — `0` there would be + # a claim about traffic the join never made. + sums = js[js.index("function sumConnections("):js.index("function renderHeader(")] + assert "unknown += 1" in sums and "attributed" in sums + assert "unknown === pipes.length ? null" in sums, \ + "a side where nothing measured started reporting a zero total" + assert "classList.toggle('stale'" in sums, \ + "a partial pipe total stopped saying it was partial" + + # The three names above are sentences, not column headings, and `.metric .label` truncates + # with an ellipsis — so in this card they wrap rather than lose their tail (phase 27). + wrapped = _declarations(_code(_css(client)), ".gate-card .metric .label") + assert wrapped.get("white-space") == "normal", \ + "an unmatched-card label that does not fit is a clipped fact" + + +def test_the_by_port_join_is_explained_once_on_the_page(client) -> None: + """The one why-line a count on this page needs, and the one place it is printed. + + `by port` is the join's honest mark for "the socket matched the port and the config, not the + address, named the host" (sixteen stream targets share `:443` on this gateway). Without the + sentence the count reads as a per-target measurement, which is the same misreading the + `unmatched.groups` list exists to prevent. So: a chip on the pipe that carries the mark, a + tooltip that says it in full, and one faint footnote in the page footer — the page's single + why-line, not a paragraph per card. + """ + html = client.get(GATEWAY_PAGE).text + footer = html[html.index(" None: + """Three arms, three sentences, none of them invented in the browser. + + * **unconfigured** — `404 {configured: false}`. The route's own refusal names `GATEWAY_HOST`, + and the page's setup block is the other half of the same sentence; the cards stay hidden + because there is nothing to draw and nothing to age (invariant 3). + * **a pre-1.2.0 agent / `--no-proxy`** and **a socket table that did not answer** — both + arrive as `payload.why`, which `sources/gateway.py` writes (with the agent's own error line + when it had one), so the page prints `#gate-banner` from the payload and never re-derives a + sentence of its own about someone else's config. + * every case that did not measure keeps its numbers as `n/a`, which `setTxt` does with a + `null` and which `countText` is the only gate for — there is no `|| 0` anywhere on a count, + because a zero is a claim about traffic and `n/a` is not. + """ + html = _code(client.get(GATEWAY_PAGE).text) + empty = html[html.index('id="gate-empty"'):html.index('
    • ', html.index('id="gate-empty"'))] + assert "GATEWAY_HOST" in empty and "1.2.0" in empty + assert 'id="gate-banner"' in html + + js = _code(client.get("/static/page-gateway.js").text) + assert "$('#gate-banner')" in js and "snapshot.why" in js + assert "err.status !== 404" in js, ( + "only the unconfigured refusal may raise the setup block; a failed read keeps the drawing") + assert "markStale(" in js and "function dimmed(" in js, \ + "an arm that could not measure must dim, not redden" + dimmed = js[js.index("function dimmed("):js.index("function cardElements(")] + assert "unreachable" in dimmed and "supported !== true" in dimmed, \ + ("the dim is keyed on something other than the two arms that did not measure — a host that " + "has never answered and an agent that predates the sections are the `.stale` cases") + counts = js[js.index("function countText("):js.index("function familyPipes(")] + assert "return null" in counts or ": null" in counts, \ + "countText no longer turns the unknown into the `n/a` setTxt prints" + assert not re.search(r"connections\s*\|\|\s*0", js), "a missing count became a claimed zero" + + +def test_the_gateway_page_keeps_its_last_drawing_when_a_poll_fails(client) -> None: + """Invariant 4 on the fifth page: a failed read leaves the last picture on screen. + + The cards are hidden **by the payload**, never by the failure path — a `fetchJSON` rejection + (a proxy hiccup, a server mid-restart) updates the status line and returns, so the operator + keeps the topology they were reading instead of an empty grid whose emptiness would read as + "this gateway routes nothing". + """ + js = _code(client.get("/static/page-gateway.js").text) + refresh = js[js.index("async function refresh("):js.index("function pruneCards(")] + guard = refresh[:refresh.index("$('#gate-empty').hidden = true")] + assert "catch" in guard and "$('#gate-cards')" not in guard, \ + "the failure path blanks the card grid — the last good drawing must survive it" + assert "$('#sub').textContent = err.message" in guard, \ + "and the reason has to be printed where the operator is already looking" + + +def test_a_clipped_pipe_label_keeps_its_full_text_on_the_element(client) -> None: + """The ellipsis is only honest while the whole string is on the element, and pinned so. + + Phase 29 task 06 measured this on live data: **29 of 42** pipe labels are ellipsised at 1440px, + because two `minmax(0, 1fr)` columns share a third of a card with the node and a + 40-character `server_name → target` string does not fit in ~150px. Letting the label wrap + instead was tried and measured worse — the flex line squeezes the name to whatever the chip, + footnote and count leave, which tore one hostname over eleven lines — so the open fix is a + layout one (AGENTS.md gap 13), not a `white-space` one. + + What makes the ellipsis a truncation and not a lost fact is that the *same* string the row + renders goes out as the element's `title`, and the pipe's `aria-label` carries the full + endpoint whatever the eye can see. Both halves are pinned here, because the day either one + goes the ellipsis starts eating the page's answer in silence. + """ + css = _code(_css(client)) + assert _declarations(css, ".pipe-name").get("text-overflow") == "ellipsis", \ + ".pipe-name no longer truncates, so re-measure the wrap decision this pin documents" + # …and the two declarations that decide *how much* it truncates. Task 06 found the name showing + # seven characters of a 40-character target, because a `nowrap` flex row shrinks the name to what + # the chip, the footnote and the `flex: none` count leave. `max-content` as its base size means a + # name longer than the column is alone on its line and gets the whole column; the chip, footnote + # and count wrap beneath it. Losing either declaration puts the seven-character label back. + assert "wrap" in _declarations(css, ".pipe-label").get("flex-wrap", ""), \ + "a long pipe label is squeezed beside its chip again" + assert "max-content" in _declarations(css, ".pipe-name").get("flex", ""), \ + "the name no longer claims the line, so it shrinks to the leftover width" + + js = _code(client.get("/static/page-gateway.js").text) + row = js[js.index("function renderPipeRow("):js.index("function renderPipes(")] + assert "const text = pipeName(pipe, keys)" in row, \ + "the label and the tooltip must be built from one string, not two" + assert "name.title =" in row, "a clipped pipe label lost the full text on the element" + aria = js[js.index("function pipeAria("):js.index("function buildPipe(")] + assert "pipeName(pipe, keys)" in aria, \ + "the accessible sentence stopped carrying the whole endpoint a clipped label truncates" + + +def test_the_docs_name_the_page_its_knob_and_its_gauges() -> None: + """The phase's own completion criterion, pinned so it cannot rot quietly. + + A page an operator cannot find in the two documents they read is a page that does not exist, and + a knob named only in `config.py` is edited in the wrong place. Three names have to be readable + in prose: the route (both page tables), `GATEWAY_HOST` **as a name and not a URL** with its + default, and the two gauge families the history API is keyed by. The agent-version requirement + belongs in the page row too — `/gateway` is unconfigured-looking rather than red when the host + runs a pre-1.2.0 agent, and that sentence is what saves the support call. + """ + root = Path(__file__).resolve().parents[1] + agents = (root / "AGENTS.md").read_text() + readme = (root / "README.md").read_text() + + for name, text in (("AGENTS.md", agents), ("README.md", readme)): + row = next((line for line in text.splitlines() if line.startswith("| `/gateway`")), "") + assert row, f"{name}'s page table has no `/gateway` row" + assert "GATEWAY_HOST" in row, f"{name}'s page row never names the knob the page is gated on" + assert "1.2.0" in row, f"{name}'s page row never names the agent version the page needs" + assert "GATEWAY_HOST" in agents and "default `gateway`" in agents, \ + "the config.py bullet stopped saying what the knob defaults to" + + # The metric-naming paragraph: these are the names `/api/gateway/history` and `sample.source` + # speak, and the outbound one carries its port because two services on one host are two facts. + assert "gateway.in_conns@" in agents, "the inbound gauge left the naming paragraph" + assert "gateway.out_conns@:" in agents, "the outbound gauge left the naming paragraph" + assert "/api/gateway/*" in readme, \ + "README stopped documenting the page's own pair of routes" + + # ---- the `/disks` page (phase 09 task 04) ---------------------------------- # # # The fourth dashboard, asserted the way the other three are: on the served bytes, because a @@ -4723,20 +6038,54 @@ def _disks_js(client: TestClient) -> str: return client.get("/static/page-disks.js").text +#: The tab nav's whole content, in order: the six dashboards and nothing else. Phase 29 task 05 +#: appended `/gateway` rather than slotting it in, so the five pre-existing tabs keep their order +#: and the new one is last — the assertion below compares this list to the nav's hrefs, so the +#: shape of the nav is one document rather than five separate "is X present" greps. +TAB_ORDER = list(PAGES) + + +def _nav_tabs(html: str) -> str: + """The served page's `", start)] + + def test_the_disks_tab_is_in_the_masthead_of_every_page(client) -> None: - """Four tabs became five, on every page, with exactly one marking itself current. + """Five tabs became six, on every page, with exactly one marking itself current. The masthead is the app's only "which dashboard do I need" surface since phase 02 deleted the overview's table, so a tab missing from one page is a page you cannot get to, and a tab marked current on the wrong page is a lie about where you are. + + Phase 29 task 05 made this the whole tab audit, in both directions. The loop is over + `MARKUP_PAGES` (all eight signed-in pages), the nav's hrefs must equal `TAB_ORDER` exactly — + a tab that leaves the nav is unreachable and a tab that joins it outside the table is a link + the phase pins know nothing about — and the `aria-current` count is the page's own status: + one, naming itself, on the six dashboards; zero on `/settings` and `/certs`, which are pages + and not dashboards. So a page that grew a current tab is caught here, and so is `/gateway` + shipping the tab everywhere except on itself. """ - for path in PAGES: + for path in MARKUP_PAGES: html = client.get(path).text - assert '' in html, f"{path} does not mark itself" - assert (html.index('href="/system"') < html.index('href="/disks"') - < html.index('href="/llm"')), "tab order is machine, drive, engine, power" + nav = _nav_tabs(html) + assert [m.group(1) for m in re.finditer(r']*aria-current="page"', nav) + expected = [path] if path in PAGES else [] + assert current == expected, ( + f"{path} marks {current} current; a dashboard marks itself and an admin page " + f"marks nothing (it is not one of the tabs)") + assert (nav.index('href="/system"') < nav.index('href="/disks"') + < nav.index('href="/llm"') < nav.index('href="/ups"') + < nav.index('href="/gateway"')), ( + "tab order is machine, drive, engine, power, ingress") # ---- the collapsing masthead nav (phase 18) -------------------------------- # @@ -4768,11 +6117,11 @@ def _masthead(html: str) -> str: def test_the_nav_toggle_is_in_the_masthead_of_every_tabbed_page(client) -> None: - """Seven pages, one identical button, and the nav it controls still carrying five tabs. + """Eight pages, one identical button, and the nav it controls still carrying six tabs. The dropdown is the only way to change dashboard on a phone, so a page missing the button is a page you can enter and leave the tab bar on, and a page whose nav lost its `id` is a button - that toggles nothing. Pinned on all seven (`MARKUP_PAGES`) rather than on `PAGES` because the + that toggles nothing. Pinned on all eight (`MARKUP_PAGES`) rather than on `PAGES` because the two admin pages have the same masthead and therefore the same contract. """ for path in MARKUP_PAGES: @@ -4783,7 +6132,7 @@ def test_the_nav_toggle_is_in_the_masthead_of_every_tabbed_page(client) -> None: assert attr in button.group(0), f"{path}'s nav toggle is missing {attr}" assert header.count('id="nav-toggle"') == 1, f"{path} has more than one nav toggle" - # The nav is the menu: the id the button names, and all five dashboards inside it. + # The nav is the menu: the id the button names, and all six dashboards inside it. assert '", @@ -4838,29 +6187,32 @@ def test_the_login_page_carries_no_nav_and_no_toggle(tmp_path: Path) -> None: # # `TODO.md:3-4` — "hosts disks engines UPS disappear when in the certs/account page" and the # operator's correction: "I did want hosts disks engines UPS badges". `Dash.initChrome()` -# builds the four badges whenever `#nav-status` exists, so the **container** is the whole +# builds the five badges whenever `#nav-status` exists, so the **container** is the whole # difference between a page that has them and a page that does not — the two admin pages shipped # without one, which is why they read as the badges vanishing rather than as two odd pages. The # fix is the container and nothing else, so the pin is the container: on every page that renders -# the masthead, in the position the five dashboards already used (first child of +# the masthead, in the position the six dashboards already used (first child of # `.masthead-right`, which is what puts it on its own row — task 02 re-justifies that row and this # pin deliberately says nothing about which way the badges face). `tests/test_auth_api.py` keeps # its own assertion that `/login` has none: a stranger has no fleet to be told about. -#: The container exactly as all seven signed-in pages write it (`index.html` is the reference). +#: The container exactly as all eight signed-in pages write it (`index.html` is the reference). NAV_STATUS_ELEMENT = '' -#: The four badges `initChrome()` renders, in its own order. -NAV_STATUS_BADGES = ("hosts", "disks", "engines", "UPS") +#: The badges `initChrome()` renders, in its own order. `gateway` is the fifth and the phase 29 +#: one: the ingress has a job of its own, so it gets a dot of its own, and `refreshStatus` leaves +#: it grey when `GATEWAY_HOST` names no target (an unconfigured source is not a fault — invariant +#: 3 — so a red dot here would train the operator to ignore red). +NAV_STATUS_BADGES = ("hosts", "disks", "engines", "UPS", "gateway") def test_every_signed_in_page_carries_the_fleet_badges(client) -> None: - """`#nav-status` is the first child of `.masthead-right` on all seven signed-in pages. + """`#nav-status` is the first child of `.masthead-right` on all eight signed-in pages. - One loop over `MARKUP_PAGES`, because the control is the same on all of them: the five + One loop over `MARKUP_PAGES`, because the control is the same on all of them: the six dashboards had it, `/certs` and `/settings` grew it. The `app.js` half of the assertion is what makes the markup mean something — the badges are built from the container's existence, - so a script that stopped reading `#nav-status` would turn seven `
    `s into dead markup and + so a script that stopped reading `#nav-status` would turn eight `
    `s into dead markup and the orphan-`id` audit alone would not say so (it also matches `#nav-status` inside the CSS). """ js = _code(client.get("/static/app.js").text) @@ -4881,7 +6233,7 @@ def test_every_signed_in_page_carries_the_fleet_badges(client) -> None: + len('
    '):] assert after_cluster.lstrip().startswith(NAV_STATUS_ELEMENT), \ f"{path} does not render #nav-status as the first child of .masthead-right "\ - "(the five dashboards' position is what puts the badges on their own row)" + "(the six dashboards' position is what puts the badges on their own row)" assert header.count('id="nav-status"') == 1, f"{path} renders the badge container twice" @@ -5051,8 +6403,13 @@ def test_the_mobile_masthead_block_declares_no_second_bar_height(client) -> None assert "row-gap: .4rem" in block.group(1), "the phone lost its tighter row gap" assert "padding-top: calc(.5rem + env(safe-area-inset-top, 0px))" in block.group(1), \ "the phone bar stopped padding for the system top-edge zone the toggle sits in" - # The dots stay one line and the cluster still wraps rather than hiding a control. - assert "#nav-status { flex-wrap: nowrap; }" in nav_css + # The cluster still wraps rather than hiding a control, and so do the dots (phase 29 task 05: + # five of them are 352px of chips against this bar's 320px content box, and `nowrap` on an item + # that may not shrink widens the *document* — the sideways scroll phase 18 forbids. `data/logs/ + # 29-task05-tabs.cjs` measures the document at 360px on all eight pages, both widths). + assert re.search(r"#nav-status\s*\{[^}]*flex-wrap: wrap", nav_css), ( + "the badge row is nowrap again: with five badges it pushes the document past the phone's " + "viewport instead of taking a second line") assert re.search(r"\.masthead-right\s*\{[^}]*flex-wrap: wrap", nav_css) @@ -5213,11 +6570,11 @@ PAUSE_ARTIFACTS = ('id="pause"', 'name="pause"', "⏸", "▶ Paused", def test_no_page_serves_a_button_that_stops_the_polls(client, tmp_path: Path) -> None: - """Eight pages, eleven assets, zero pause control. + """Nine pages, twelve assets, zero pause control. `/login` needs its own anonymous `TestClient` for the reason every other login-page audit gives: this `client` is signed in, and a signed-in `GET /login` is a `303` to the dashboard, - which would audit `/`'s bytes and call them `/login`. The other seven are read as themselves. + which would audit `/`'s bytes and call them `/login`. The other eight are read as themselves. """ bodies = {path: client.get(path).text for path in MARKUP_PAGES} @@ -5234,7 +6591,7 @@ def test_no_page_serves_a_button_that_stops_the_polls(client, tmp_path: Path) -> finally: store.close() - assert len(bodies) == 8, "the audit lost a page; every served page owes this check" + assert len(bodies) == 9, "the audit lost a page; every served page owes this check" for path, html in bodies.items(): for artifact in PAUSE_ARTIFACTS: assert artifact not in html, f"{path} serves the removed control again: {artifact!r}" @@ -5519,8 +6876,8 @@ def test_summary_names_the_drive_the_health_verdict_called_failed( # reach them. The fix is one rule inside the repo's existing ≤640px block, taken from the shape # the all-widths `table.data.scroll` modifier already proves — `display: block` plus a horizontal # scrollbar, so the scroll container is the table and never the document (phase 18's standing -# guarantee). It is a class rule rather than seven hand-placed `.scroll` classes because four of -# the seven tables exist only in JS-built markup and the class is the whole contract. +# guarantee). It is a class rule rather than hand-placed `.scroll` classes because most of those tables +# exist only in JS-built markup and the class is the whole contract. # # Two failure modes the pins below are aimed at, both named in the task's assumptions: a # `max-height` migrating in from `.scroll` (a phone then gets a height-capped table on top of a @@ -5684,10 +7041,10 @@ def test_a_card_disclosure_scrolls_what_it_reveals_at_every_width(client) -> Non def test_every_table_on_every_page_carries_the_class_the_rule_targets(client) -> None: """The mobile rule is class-level, so a `` without `data` would silently not scroll. - Both halves are read from the served bytes: the seven static tables in the page templates and + Both halves are read from the served bytes: the static tables in the page templates and the five built at runtime by `page-system.js` / `page-ups.js` (four inside `/system`'s "More detail", one the NUT variables list). This is the drift the task chose the class rule over - seven hand-placed `.scroll` classes for — and it cuts both ways, which is why it is a test. + the hand-placed `.scroll` classes it would replace — and it cuts both ways, which is why it is a test. """ tags = [tag for page in MARKUP_PAGES for tag in _table_tags(_code(client.get(page).text))] tags += [tag for script in PAGE_SCRIPTS.values() diff --git a/tests/test_auth_api.py b/tests/test_auth_api.py index 4af037e..6fdf813 100644 --- a/tests/test_auth_api.py +++ b/tests/test_auth_api.py @@ -56,7 +56,12 @@ CURL_HEADERS = {"accept": "*/*", "user-agent": "curl/8.5.0"} PROTECTED_API = ( "/api/config", "/api/events", "/api/metrics", "/api/history", "/api/summary", "/api/system/snapshot", "/api/system/history", "/api/disks/snapshot", - "/api/disks/history", "/api/llm/snapshot", "/api/llm/history", "/api/ups/snapshot", + "/api/disks/history", + # Phase 29: the ingress's pipes. `…/snapshot` carries the gateway's whole routing map — every + # domain it serves and every address it dials — so it is listed here for the same reason the + # bundle download is, whatever the payload happens to hold today. + "/api/gateway/snapshot", "/api/gateway/history", + "/api/llm/snapshot", "/api/llm/history", "/api/ups/snapshot", "/api/ups/history", "/api/ups/commands", "/api/auth/status", # Phase 15: the certificate authority's own surface. `/api/certs//bundle` is every # agent's private key, so "it needs a session" is the whole point of listing it here. @@ -68,7 +73,11 @@ PROTECTED_API = ( #: thing that must never regress is a new route answering anonymously. `/certs` is the sharper #: case of the two — an anonymous caller reaching it would see the CA's serials and every host #: the dashboard dials. -PROTECTED_PAGES = ("/", "/system", "/disks", "/llm", "/ups", "/settings", "/certs") +#: `/gateway` (phase 29 task 04) is the sixth dashboard: it prints this fleet's ingress map — +#: every listening port, every `server_name`, every internal target — so it is gated by being a +#: page and nothing else. +PROTECTED_PAGES = ("/", "/system", "/disks", "/llm", "/ups", "/gateway", "/settings", + "/certs") #: The one write this app has, with the fields the route needs (the UPS is unconfigured, so a @@ -1598,8 +1607,8 @@ def test_the_served_route_table_is_the_surface_the_gate_describes(anon: TestClie pages = sorted(str(route.path) for route in served_routes(anon) if isinstance(getattr(route, "path", None), str) and not str(route.path).startswith(("/api", "/static"))) - assert pages == ["/", "/certs", "/disks", "/llm", "/login", "/settings", "/system", - "/ups"] + assert pages == ["/", "/certs", "/disks", "/gateway", "/llm", "/login", "/settings", + "/system", "/ups"] for path in pages: response = anon.get(path, headers=BROWSER_HEADERS) if path == "/login": diff --git a/tests/test_collectors.py b/tests/test_collectors.py index 7c751db..f037879 100644 --- a/tests/test_collectors.py +++ b/tests/test_collectors.py @@ -28,11 +28,13 @@ from dashboards.collectors import ( disks_jobs, fetch_disk_snapshot, fetch_engine, + fetch_gateway_snapshot, fetch_host_snapshot, fetch_snapshot, + gateway_job, host_jobs, ) -from dashboards.config import Target +from dashboards.config import Settings, Target from dashboards.poller import Result from dashboards.rates import RateEstimator from dashboards.sources import nut @@ -409,6 +411,80 @@ async def test_smart_addon_errors_reach_the_disk_payload_verbatim( assert "degraded" not in system.payload, "and /system stays unbothered by it" +# --------------------------------------------------------------------------- # +# gateway topology (the /gateway source's fetch half, phase 29) +# --------------------------------------------------------------------------- # +async def test_the_gateway_fetch_reads_the_same_agent_snapshot_as_the_other_two( + gateway_snapshot: Any) -> None: + """One endpoint, three readers: the pipes come out of `/snapshot` and nothing else. + + The gateway job has no URL of its own (`GATEWAY_HOST` is a *name*), so this is the pin that + keeps it that way — and that the projection's own honesty rules survive the trip through the + collector: gauges under the `gateway` family, **no** events (a count has no transition worth a + line), and no `window_seconds` for a number that was never diffed (rule 2). + """ + body = gateway_snapshot("full") + seen: list[str] = [] + + def handler(request: httpx.Request) -> httpx.Response: + assert request.url.path == "/snapshot", "the gateway page has no endpoint of its own" + seen.append(request.url.path) + return httpx.Response(200, json=body) + + target = Target("gateway", "http://agent.test") + async with client_for(handler) as client: + result = await fetch_gateway_snapshot(client, target, timeout=10) + assert seen == ["/snapshot"] + assert result.payload["host"] == "gateway" + assert result.payload["supported"] is True + assert result.payload["inbound"], "the configured pipes arrive with the snapshot" + assert result.samples and all(sample.source == "gateway" for sample in result.samples), ( + "the gauges are their own sample family, not `host`'s") + assert result.events == [], "a connection count has no transition to report" + assert "window_seconds" not in result.payload, "a count is not a rate" + + +async def test_the_gateway_poll_carries_its_own_timeout_and_never_disables_one() -> None: + """The third budget on one endpoint: the client default until the job asks for its own. + + Same rule the `disks` poll was pinned with — a literal `None` means *no timeout at all* to + httpx, which would hang the only poller this source has (invariant 4) — and the reason + `GATEWAY_TIMEOUT_SECONDS` is a setting rather than a constant inherited from `system`. + """ + def handler(request: httpx.Request) -> httpx.Response: + return httpx.Response(200, json={"ok": True, "ts": 1.0, "errors": []}) + + client = TimeoutRecordingClient(transport=httpx.MockTransport(handler), + timeout=4.0, base_url="http://agent.test") + target = Target("gateway", "http://agent.test") + async with client: + await fetch_gateway_snapshot(client, target) + assert client.timeouts[-1] is httpx.USE_CLIENT_DEFAULT + await fetch_gateway_snapshot(client, target, timeout=10) + assert client.timeouts[-1] == 10, "the gateway poll waits on its own budget" + assert all(timeout is not None for timeout in client.timeouts) + + +async def test_the_gateway_job_is_one_job_named_after_the_host_it_reads() -> None: + """`key`/`dashboard`/`label`/`interval` at the factory, and `None` for a name nobody dials. + + The label is what an alert and an event print and what `alerts.py` uses to find the `host:` job + reading the same agent, so it is the host's name and not the source's — while the key stays the + single word `gateway` because there is exactly one such job per deployment. + """ + settings = Settings(host_targets=(Target("gateway", "https://gw.test"),), + gateway_interval=45.0, gateway_timeout=7.0) + entry = AgentTls(verify="/ca.pem", cert=("/c.pem", "/k.pem"), status=lambda: certs.ACTIVE) + async with httpx.AsyncClient() as client: + job = gateway_job(settings, client, tls={"gateway": entry}) + assert job is not None + assert (job.key, job.dashboard, job.label) == ("gateway", "gateway", "gateway") + assert job.interval == 45.0 + assert job.enabled is True, "a job that exists is a job that runs" + missing = Settings(host_targets=(Target("deskwork", "http://desk.test"),)) + assert gateway_job(missing, httpx.AsyncClient()) is None, "unconfigured is no task at all" + + # --------------------------------------------------------------------------- # # TLS: the dashboard dials a protected exporter with its own client certificate # --------------------------------------------------------------------------- # diff --git a/tests/test_config.py b/tests/test_config.py index 4fa5121..a512d2e 100644 --- a/tests/test_config.py +++ b/tests/test_config.py @@ -87,6 +87,54 @@ def test_settings_default_to_no_sources_configured() -> None: assert settings.any_source_enabled is False +def test_the_gateway_source_names_the_ingress_host_and_its_own_cadence() -> None: + """Phase 29: one `HOST_TARGETS` name is the gateway, and it is polled on its own clock. + + `gateway_host` defaults to the fleet's real name so installing agent 1.2.0 on a host called + `gateway` is the whole configuration — but the **default is not a target**: with no + `HOST_TARGETS` entry of that name the source is unconfigured, `/api/gateway/*` 404s without + dialling, and the page says so (invariant 3). The interval sits between the system poll and + SMART because a connection count is the shape of half a minute, not of ten seconds. + """ + settings = Settings() + assert settings.gateway_host == "gateway" + assert settings.gateway_interval == 30.0 + assert settings.gateway_timeout == 10.0 + assert settings.gateway_enabled is False + assert settings.any_source_enabled is False + + +def test_gateway_enabled_is_true_only_for_a_name_that_is_a_target() -> None: + """A name with no agent behind it is unconfigured, not broken — the `nut_enabled` shape.""" + def with_targets(raw: str, name: str = "gateway") -> bool: + targets, _problems = parse_targets(raw) + return Settings(host_targets=tuple(targets), gateway_host=name).gateway_enabled + + assert with_targets("gateway=https://gw.example.net:9120") is True + assert with_targets( + "deskwork=https://d.example.net:9120 gateway=https://gw.example.net:9120") is True + assert with_targets("deskwork=https://d.example.net:9120") is False + assert with_targets("router=https://gw.example.net:9120", name="router") is True + assert with_targets("Gateway=https://gw.example.net:9120") is False, ("names are lowercase") + assert with_targets("") is False + + +def test_the_gateway_settings_come_from_the_environment(monkeypatch: pytest.MonkeyPatch) -> None: + """`GATEWAY_HOST=router` is how a deployment whose ingress is called something else opts in.""" + monkeypatch.setenv("GATEWAY_HOST", "router") + monkeypatch.setenv("GATEWAY_INTERVAL_SECONDS", "15") + monkeypatch.setenv("GATEWAY_TIMEOUT_SECONDS", "20") + reloaded = importlib.reload(config) + try: + assert reloaded.SETTINGS.gateway_host == "router" + assert reloaded.SETTINGS.gateway_interval == 15.0 + assert reloaded.SETTINGS.gateway_timeout == 20.0 + finally: + for name in ("GATEWAY_HOST", "GATEWAY_INTERVAL_SECONDS", "GATEWAY_TIMEOUT_SECONDS"): + monkeypatch.delenv(name) + importlib.reload(config) + + def test_the_disk_source_defaults_to_a_five_minute_cadence() -> None: """SMART is not a per-second signal (phase 09), and the fetch waits for the agent's own 20 s smartctl budget. Both are env-only, like every other interval.""" diff --git a/tests/test_gateway_source.py b/tests/test_gateway_source.py new file mode 100644 index 0000000..e8c0e26 --- /dev/null +++ b/tests/test_gateway_source.py @@ -0,0 +1,718 @@ +"""`sources/gateway.py`: the config + socket join that becomes the `/gateway` pipes. + +The shape half of these tests runs the **captured** gateway configs and `/proc` tables through +the real agent `Collector` (the `gateway_snapshot` fixture), so a rename of a section key on +either side of the wire fails here rather than drawing an empty pipe on a live page. The +arithmetic half uses synthetic sections, because the point of those tests is which socket lands +in which pipe and what the source refuses to decide — facts a capture states once and a +re-capture may lawfully change (the same rule `tests/test_agent.py` holds for the socket tables). +""" +from __future__ import annotations + +import json +import socket as socket_module +from collections.abc import Mapping, Sequence +from typing import Any + +from dashboards.sources import gateway + +# --------------------------------------------------------------------------- # +# synthetic builders: a section exactly as the agent serves it, minus the file +# --------------------------------------------------------------------------- # + +def bind(port: int | None, addr: str = "::") -> dict[str, Any]: + return {"addr": addr, "port": port} + + +def haproxy_section(frontends: Sequence[dict[str, Any]], + backends: Sequence[dict[str, Any]] = (), *, + available: bool | None = True) -> dict[str, Any]: + return {"available": available, "config": "/etc/haproxy/haproxy.cfg", + "frontends": list(frontends), "backends": list(backends)} + + +def frontend(name: str, ports: Sequence[int | None], *, default: str | None = None, + use: Sequence[str] = ()) -> dict[str, Any]: + return {"name": name, "binds": [bind(port) for port in ports], + "default_backend": default, "backends": list(use)} + + +def backend(name: str, servers: Sequence[dict[str, Any]]) -> dict[str, Any]: + return {"name": name, "servers": list(servers)} + + +def server(name: str, target: str | None, port: int | None = None) -> dict[str, Any]: + return {"name": name, "target": target, "port": port} + + +def nginx_section(servers: Sequence[dict[str, Any]] = (), maps: Sequence[dict[str, Any]] = (), *, + available: bool | None = True) -> dict[str, Any]: + return {"available": available, "config": "/etc/nginx/nginx.conf", "files": [], + "servers": list(servers), "upstreams": [], "maps": list(maps)} + + +def nginx_server(ports: Sequence[int | None], names: Sequence[str] = (), + passes: Sequence[dict[str, Any]] = ()) -> dict[str, Any]: + return {"listen": [bind(port, addr="") for port in ports], "server_names": list(names), + "proxy_passes": list(passes)} + + +def proxy_pass(target: tuple[str, int] | None, *, variable: str | None = None, + location: str | None = "/") -> dict[str, Any]: + host, port = target if target else (None, None) + return {"location": location, "scheme": "http" if target else None, + "target": None if host is None else {"host": host, "port": port}, + "variable": variable} + + +def mapping_block(variable: str, + entries: Sequence[tuple[str, tuple[str, int | None] | None]]) -> dict[str, Any]: + return {"variable": variable, + "entries": [{"key": key, + "target": None if target is None else {"host": target[0], + "port": target[1]}} + for key, target in entries]} + + +def sockets_section(listens: Sequence[dict[str, Any]] = (), + established: Sequence[dict[str, Any]] = (), *, + available: bool | None = True) -> dict[str, Any]: + return {"available": available, "listens": list(listens), "established": list(established)} + + +def listen(port: int, addr: str = "0.0.0.0", family: str = "v4") -> dict[str, Any]: + return {"family": family, "addr": addr, "port": port} + + +def conn(local: tuple[str, int], remote: tuple[str, int], *, + family: str = "v4") -> dict[str, Any]: + return {"family": family, "local": {"addr": local[0], "port": local[1]}, + "remote": {"addr": remote[0], "port": remote[1]}} + + +def snapshot(*, haproxy: dict[str, Any] | None = None, nginx: dict[str, Any] | None = None, + sockets: dict[str, Any] | None = None, errors: Sequence[str] = (), + agent_version: str | None = "1.2.0") -> dict[str, Any]: + """The document one agent `/snapshot` would answer, with only what a test names.""" + doc: dict[str, Any] = {"ok": True, "schema": 1, "ts": 1_760_000_000.0, "host": "gateway", + "errors": list(errors)} + if agent_version is not None: + doc["agent_version"] = agent_version + if haproxy is not None: + doc["haproxy"] = haproxy + if nginx is not None: + doc["nginx"] = nginx + if sockets is not None: + doc["sockets"] = sockets + return doc + + +def pipe_by_port(payload: Mapping[str, Any], port: int) -> dict[str, Any]: + for pipe in payload["inbound"]: + if pipe["port"] == port: + return pipe + raise AssertionError(f"no inbound pipe on port {port}: {[p['port'] for p in payload['inbound']]}") + + +def pipe_to(payload: Mapping[str, Any], host: str, port: int | None) -> dict[str, Any]: + for pipe in payload["outbound"]: + if pipe["target"]["host"] == host and pipe["target"]["port"] == port: + return pipe + raise AssertionError(f"no outbound pipe to {host}:{port}") + + +def walk_keys(node: Any) -> set[str]: + """Every key name anywhere in a payload — the device the "no severity / no window" pins use.""" + if isinstance(node, Mapping): + found = set(node.keys()) + for value in node.values(): + found |= walk_keys(value) + return found + if isinstance(node, Sequence) and not isinstance(node, str | bytes): + return set().union(*(walk_keys(item) for item in node)) if node else set() + return set() + + +# --------------------------------------------------------------------------- # +# the captured gateway: every configured pipe, from the real agent code path +# --------------------------------------------------------------------------- # + +def test_the_captured_configs_yield_every_configured_pipe(gateway_snapshot: Any) -> None: + """Six haproxy frontends, four nginx listen ports, the stream map's fan-out — all present.""" + payload = gateway.parse(gateway_snapshot("full"), "gateway") + assert payload["configured"] is True + assert payload["supported"] is True + assert payload["why"] is None, "a gateway whose every section answered gets no why-line" + assert payload["sections"] == {"haproxy": True, "nginx": True, "sockets": True} + + haproxy_ports = [row["port"] for row in payload["haproxy"]["frontends"]] + assert haproxy_ports == [22, 2000, 5201, 8080, 25565, 25566], "config order, one row per bind" + assert {row["backend"] for row in payload["haproxy"]["frontends"]} == { + "gitea-2222", "sec-gitea-2000", "iperf3-5201", "unifi-external-8080", + "testcraft-25565", "nimcraft-25566"} + + assert [row["port"] for row in payload["nginx"]["servers"]] == [80, 443, 4443, 8443] + names = {row["port"]: row["server_names"] for row in payload["nginx"]["servers"]} + assert names[80] == ["_"], "the catch-all is the config's own text, kept verbatim" + assert names[443] == [], "the stream server routes by SNI, so it names no domain of its own" + assert "nextcloud.reeseapps.com" in names[4443] and "gitea.reeseapps.com" in names[4443], ( + "five vhosts share 4443 and their domains belong to the one pipe for that port") + + assert payload["nginx"]["map_targets"], "the stream map is the fan-out the page draws" + assert len(payload["nginx"]["map_targets"]) == 17 + homepage = next(row for row in payload["nginx"]["map_targets"] + if row["key"] == "homepage.reeseapps.com") + assert homepage == {"key": "homepage.reeseapps.com", + "host": "reeseapps-homepage.reeselink.com", "port": 443} + + +def test_the_source_reads_the_keys_the_agent_actually_publishes(gateway_snapshot: Any, + host_agent_module: Any) -> None: + """The coupling is pinned: `binds`/`port`, `servers`/`target`, `listen`/`server_names`. + + The fixture runs the real parser, so this compares the source's output against the agent's + own document rather than against a copy of its field names — which is what makes a rename + fail here instead of printing `n/a` beside a healthy gateway (AGENTS.md rule 7). + """ + snap = gateway_snapshot("full") + payload = gateway.parse(snap, "gateway") + agent_bind_ports = {bind["port"] for frontend in snap["haproxy"]["frontends"] + for bind in frontend["binds"]} + assert {row["port"] for row in payload["haproxy"]["frontends"]} == agent_bind_ports + agent_servers = {(row["name"], row["target"], row["port"]) + for block in snap["haproxy"]["backends"] for row in block["servers"]} + assert {(pipe["target"]["name"], pipe["target"]["host"], pipe["target"]["port"]) + for pipe in payload["outbound"] if pipe["source"] == "haproxy"} == agent_servers + agent_listen_ports = {listen["port"] for server in snap["nginx"]["servers"] + for listen in server["listen"]} + assert {row["port"] for row in payload["nginx"]["servers"]} == agent_listen_ports + # and the three section names themselves, as the snapshot publishes them + assert {"haproxy", "nginx", "sockets"} <= set(snap) + assert host_agent_module.AGENT_VERSION == "1.2.0", "the sections these keys belong to" + + +def test_the_live_counts_reconcile_with_the_socket_table(gateway_snapshot: Any) -> None: + """Every ESTABLISHED row is counted exactly once — in a pipe, or in `unmatched`. + + This is the property phase 29 task 06 checks against a direct `/proc/net/tcp` read; pinning + it on the capture means the arithmetic is already right before anything is dialled. + """ + snap = gateway_snapshot("full") + payload = gateway.parse(snap, "gateway") + established = len(snap["sockets"]["established"]) + in_pipes = sum(pipe["connections"] for pipe in payload["inbound"]) + out_pipes = sum(pipe["connections"] or 0 for pipe in payload["outbound"]) + unmatched = payload["unmatched"] + assert payload["sockets"] == {"listens": len(snap["sockets"]["listens"]), + "established": established} + assert in_pipes + out_pipes + unmatched["inbound"] + unmatched["outbound"] == established + # the ambiguous fan-out is inside `unmatched.outbound`, not on top of it + assert sum(group["connections"] for group in unmatched["groups"]) <= unmatched["outbound"] + + +def test_a_listening_port_no_config_claims_stays_visible(gateway_snapshot: Any) -> None: + """Phase 28's "an unclaimed socket stays visible as one", counted, not silently dropped.""" + snap = gateway_snapshot("full") + payload = gateway.parse(snap, "gateway") + live = {row["port"] for row in snap["sockets"]["listens"]} + configured = {pipe["port"] for pipe in payload["inbound"]} + assert payload["unmatched"]["other"] == len(live - configured) + # the agent's own port is dialled by nobody in either config, so it is always the example + assert 9120 in live - configured + + +def test_the_map_fanout_becomes_pipes_labelled_by_domain(gateway_snapshot: Any) -> None: + """Every map entry is a pipe the page can draw, and `$var` never becomes a host.""" + payload = gateway.parse(gateway_snapshot("full"), "gateway") + map_pipes = [pipe for pipe in payload["outbound"] if pipe["source"] == "map"] + assert len(map_pipes) == len(payload["nginx"]["map_targets"]) + assert not [pipe for pipe in payload["outbound"] + if pipe["target"]["host"] is None or "$" in str(pipe["target"]["host"])], ( + "a proxy_pass $var carries no address, so it must not become a pipe of its own") + brain = pipe_to(payload, "reeseapps-brain-of-reese.reeselink.com", 443) + assert brain["target"]["by_port"] is True, "a name can only ever be matched on its port" + + +# --------------------------------------------------------------------------- # +# the inbound join +# --------------------------------------------------------------------------- # + +def test_a_socket_lands_in_the_frontend_that_owns_its_local_port() -> None: + snap = snapshot( + haproxy=haproxy_section([frontend("ssh", [22], default="ssh-be")]), + nginx=nginx_section([nginx_server([443], ["a.example.test"])]), + sockets=sockets_section( + listens=[listen(22), listen(443)], + established=[conn(("10.1.1.27", 443), ("203.0.113.7", 51000)), + conn(("10.1.1.27", 443), ("203.0.113.9", 51001)), + conn(("10.1.1.27", 22), ("198.51.100.4", 33000))])) + payload = gateway.parse(snap, "gateway") + https = pipe_by_port(payload, 443) + assert https["source"] == "nginx" + assert https["connections"] == 2 + assert https["clients"] == ["203.0.113.7", "203.0.113.9"], "the client is the remote end" + assert https["domains"] == ["a.example.test"] + assert https["metric"] == "gateway.in_conns@443" + assert pipe_by_port(payload, 22)["source"] == "haproxy" + assert payload["unmatched"] == {"inbound": 0, "outbound": 0, "other": 0, "groups": []} + + +def test_a_frontend_that_is_idle_is_still_a_pipe_with_zero() -> None: + """A configured port with no connections is `0`, not `n/a` — the socket table did answer.""" + snap = snapshot(haproxy=haproxy_section([frontend("ssh", [22])]), + sockets=sockets_section(listens=[listen(22)])) + payload = gateway.parse(snap, "gateway") + assert pipe_by_port(payload, 22)["connections"] == 0 + + +def test_a_connection_on_a_port_no_config_names_is_reported_not_dropped() -> None: + """`unmatched.inbound`: this host is serving something neither file mentions.""" + snap = snapshot( + nginx=nginx_section([nginx_server([443], ["a.example.test"])]), + sockets=sockets_section( + listens=[listen(443), listen(8081)], + established=[conn(("10.1.1.27", 8081), ("10.1.0.9", 41000)), + conn(("10.1.1.27", 8081), ("10.1.0.9", 41001))])) + payload = gateway.parse(snap, "gateway") + assert pipe_by_port(payload, 443)["connections"] == 0 + assert payload["unmatched"]["inbound"] == 2, "the page's 'N other connections'" + assert payload["unmatched"]["other"] == 1, "8081 is a listener no frontend claims" + + +def test_one_port_named_by_both_configs_is_one_pipe_carrying_both_domains() -> None: + """A host cannot bind a port twice, so the socket space is one pipe and the facts merge.""" + snap = snapshot( + haproxy=haproxy_section([frontend("http", [80])]), + nginx=nginx_section([nginx_server([80], ["a.example.test"])]), + sockets=sockets_section(listens=[listen(80)], + established=[conn(("10.1.1.27", 80), ("203.0.113.1", 1234))])) + payload = gateway.parse(snap, "gateway") + ports = [pipe["port"] for pipe in payload["inbound"]] + assert ports == [80], "no second pipe, and no socket counted down both" + assert pipe_by_port(payload, 80)["connections"] == 1 + assert pipe_by_port(payload, 80)["domains"] == ["a.example.test"] + + +def test_the_client_list_is_unique_sorted_and_bounded() -> None: + ports = [80] + many = [conn(("10.1.1.27", 80), (f"203.0.113.{i % 256}", 40000 + i)) + for i in range(gateway.MAX_CLIENTS + 12)] + many.append(conn(("10.1.1.27", 80), ("203.0.113.1", 45678))) + snap = snapshot(nginx=nginx_section([nginx_server(ports, ["a.example.test"])]), + sockets=sockets_section(listens=[listen(80)], established=many)) + pipe = pipe_by_port(gateway.parse(snap, "gateway"), 80) + assert len(pipe["clients"]) == gateway.MAX_CLIENTS + assert pipe["clients"] == sorted(pipe["clients"]) + assert pipe["connections"] == len(many), "the count stays the truth when the list is capped" + + +# --------------------------------------------------------------------------- # +# the outbound join +# --------------------------------------------------------------------------- # + +def test_a_target_written_as_an_address_matches_exactly_and_says_by_port_false() -> None: + snap = snapshot( + nginx=nginx_section([nginx_server([8443], ["app.example.test"], + [proxy_pass(("127.0.0.1", 8081))])]), + sockets=sockets_section( + listens=[listen(8443)], + established=[conn(("10.1.1.27", 39000), ("127.0.0.1", 8081)), + conn(("10.1.1.27", 39002), ("10.9.9.9", 8081))])) + payload = gateway.parse(snap, "gateway") + exact = pipe_to(payload, "127.0.0.1", 8081) + assert exact["target"]["by_port"] is False + assert exact["connections"] == 1, "the one socket dialled to that address" + assert exact["metric"] == "gateway.out_conns@127.0.0.1:8081" + assert payload["unmatched"]["outbound"] == 1, ( + "the same port on a different address is not this target's connection") + + +def test_a_domain_target_is_matched_on_port_alone_and_labelled_by_port() -> None: + """The address half is the config's, the socket half is the kernel's, and `by_port` says so.""" + snap = snapshot( + nginx=nginx_section([nginx_server([4443], ["cloud.example.test"], + [proxy_pass(("nextcloud.example.test", 11000))])]), + sockets=sockets_section( + listens=[listen(4443)], + established=[conn(("10.1.1.27", 4443), ("10.1.0.4", 55555)), + conn(("10.1.1.27", 54340), ("10.1.0.254", 11000))])) + payload = gateway.parse(snap, "gateway") + pipe = pipe_to(payload, "nextcloud.example.test", 11000) + assert pipe["target"]["by_port"] is True + assert pipe["connections"] == 1 + assert pipe["source"] == "nginx" + assert payload["unmatched"]["groups"] == [] + + +def test_a_port_shared_by_several_names_resolves_none_of_them() -> None: + """The real `stream.conf` shape: sixteen names on 443, and no basis for picking one. + + The sockets go to `unmatched.outbound` with a group naming the candidates; each candidate's + own count stays `null` — a repeated port total drawn on sixteen pipes reads as sixteen times + the traffic, and a split of it is arithmetic nobody measured (rule 2). + """ + entries = [(f"site{i}.example.test", (f"backend{i}.example.test", 443)) for i in range(4)] + snap = snapshot( + nginx=nginx_section([nginx_server([443], [], [proxy_pass(None, variable="targetBackend")])], + maps=[mapping_block("$targetBackend", entries)]), + sockets=sockets_section( + listens=[listen(443)], + established=[conn(("10.1.1.27", 443), ("203.0.113.5", 4000)), + conn(("10.1.1.27", 50000), ("10.2.2.2", 443)), + conn(("10.1.1.27", 50002), ("10.2.2.3", 443))])) + payload = gateway.parse(snap, "gateway") + assert len(payload["nginx"]["map_targets"]) == 4 + for _key, (host, port) in entries: + assert pipe_to(payload, host, port)["connections"] is None, "unknown, never 0 and never 2" + group = payload["unmatched"]["groups"][0] + assert group == {"port": 443, "connections": 2, + "hosts": ["backend0.example.test", "backend1.example.test", + "backend2.example.test", "backend3.example.test"], + "host_count": 4} + assert payload["unmatched"]["outbound"] == 2 + assert payload["unmatched"]["inbound"] == 0 + + +def test_the_group_host_list_is_bounded_and_reports_the_real_total() -> None: + entries = [(f"site{i}.example.test", (f"backend{i}.example.test", 443)) + for i in range(gateway.MAX_GROUP_HOSTS + 6)] + snap = snapshot( + nginx=nginx_section([nginx_server([443], [], [proxy_pass(None, variable="be")])], + maps=[mapping_block("$be", entries)]), + sockets=sockets_section(established=[conn(("10.1.1.27", 50000), ("10.2.2.2", 443))])) + group = gateway.parse(snap, "gateway")["unmatched"]["groups"][0] + assert len(group["hosts"]) == gateway.MAX_GROUP_HOSTS + assert group["host_count"] == len(entries) + + +def test_no_join_ever_asks_a_resolver() -> None: + """The whole point of `by_port`: this module must not turn a name into an address. + + Spied on the live `socket` module the stdlib resolvers use, and run over the captured + gateway as well as a synthetic one, so a "helpful" future `socket.getaddrinfo` call inside + the join fails here rather than making 33 DNS queries every 30 seconds from the poll loop. + """ + calls: list[tuple[Any, ...]] = [] + + def spy(*args: Any, **kwargs: Any) -> Any: # pragma: no cover - must never run + calls.append((args, kwargs)) + raise AssertionError("the gateway join must never resolve a name") + + original = socket_module.getaddrinfo + socket_module.getaddrinfo = spy + try: + gateway.parse(snapshot( + haproxy=haproxy_section([frontend("mc", [25565])], + [backend("mc-be", [server("mc1", "minecraft.example.test", + 25565)])]), + nginx=nginx_section([nginx_server([443], ["a.example.test"], + [proxy_pass(("app.example.test", 8080))])]), + sockets=sockets_section(listens=[listen(25565), listen(443)], + established=[conn(("10.1.1.27", 443), ("203.0.113.5", 900)), + conn(("10.1.1.27", 1), ("10.2.2.2", 8080))])), + "gateway") + finally: + socket_module.getaddrinfo = original + assert calls == [] + + +def test_two_services_on_one_host_are_two_pipes_not_one_sum() -> None: + """The reason the outbound series name carries its port (`minecraft` 25565 and 25566).""" + snap = snapshot( + haproxy=haproxy_section([frontend("mc1", [25565]), frontend("mc2", [25566])], + [backend("testcraft", [server("mc1", "mc.example.test", 25565)]), + backend("nimcraft", [server("mc2", "mc.example.test", 25566)])]), + sockets=sockets_section( + established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 25565)), + conn(("10.1.1.27", 40002), ("10.5.5.5", 25566)), + conn(("10.1.1.27", 40004), ("10.5.5.5", 25566))])) + payload = gateway.parse(snap, "gateway") + assert pipe_to(payload, "mc.example.test", 25565)["connections"] == 1 + assert pipe_to(payload, "mc.example.test", 25566)["connections"] == 2 + assert [pipe["metric"] for pipe in payload["outbound"]] == [ + "gateway.out_conns@mc.example.test:25565", "gateway.out_conns@mc.example.test:25566"] + names = {pipe["target"]["name"] for pipe in payload["outbound"]} + assert names == {"mc1", "mc2"}, "the config's own server names survive the dedup" + + +def test_a_target_with_no_port_is_a_pipe_with_an_unknown_count() -> None: + """`server x host` names no port, so there is nothing to join on and nothing to invent.""" + snap = snapshot( + haproxy=haproxy_section([], [backend("be", [server("solo", "solo.example.test")])]), + sockets=sockets_section(established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 8080))])) + pipe = pipe_to(gateway.parse(snap, "gateway"), "solo.example.test", None) + assert pipe["target"]["by_port"] is False + assert pipe["connections"] is None + assert pipe["metric"] == "gateway.out_conns@solo.example.test" + + +def test_the_same_target_from_two_configs_is_one_pipe() -> None: + snap = snapshot( + nginx=nginx_section([nginx_server([4443], ["a.example.test"], + [proxy_pass(("keycloak.example.test", 8080))]), + nginx_server([8443], ["admin.a.example.test"], + [proxy_pass(("keycloak.example.test", 8080))])]), + sockets=sockets_section(established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 8080))])) + payload = gateway.parse(snap, "gateway") + targets = [pipe for pipe in payload["outbound"] + if pipe["target"]["host"] == "keycloak.example.test"] + assert len(targets) == 1 + assert targets[0]["connections"] == 1, "one pipe, so one count, so no double draw" + + +# --------------------------------------------------------------------------- # +# the map: which maps route, and which are just header words +# --------------------------------------------------------------------------- # + +def test_a_map_no_proxy_pass_references_contributes_nothing() -> None: + """`$connection_upgrade`'s values are `upgrade`/`close` — header words, never a host.""" + snap = snapshot( + nginx=nginx_section([nginx_server([443], [], [proxy_pass(None, variable="targetBackend")])], + maps=[mapping_block("connection_upgrade", + [("default", ("upgrade", None)), + ("", ("close", None))]), + mapping_block("targetBackend", + [("a.example.test", ("be.example.test", 443))])]), + sockets=sockets_section(listens=[listen(443)])) + payload = gateway.parse(snap, "gateway") + assert [row["key"] for row in payload["nginx"]["map_targets"]] == ["a.example.test"] + assert [pipe["target"]["host"] for pipe in payload["outbound"]] == ["be.example.test"] + + +def test_a_map_entry_with_no_address_is_skipped() -> None: + snap = snapshot( + nginx=nginx_section([nginx_server([443], [], [proxy_pass(None, variable="be")])], + maps=[mapping_block("be", [("a.example.test", None), + ("b.example.test", ("be2.example.test", 443))])])) + payload = gateway.parse(snap, "gateway") + assert [row["key"] for row in payload["nginx"]["map_targets"]] == ["b.example.test"] + + +def test_a_duplicate_endpoint_keeps_the_name_the_config_has() -> None: + """haproxy is claimed first, so nginx writing the same endpoint loses nothing.""" + snap = snapshot( + haproxy=haproxy_section([frontend("app", [80])], + [backend("app-be", [server("app1", "app.example.test", 9000)])]), + nginx=nginx_section([nginx_server([8443], ["app.example.test"], + [proxy_pass(("app.example.test", 9000))])]), + sockets=sockets_section(established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 9000))])) + payload = gateway.parse(snap, "gateway") + app_pipes = [pipe for pipe in payload["outbound"] + if pipe["target"]["host"] == "app.example.test"] + assert len(app_pipes) == 1 + assert app_pipes[0]["target"]["name"] == "app1" + assert app_pipes[0]["source"] == "haproxy" + assert app_pipes[0]["connections"] == 1 + + +def test_a_proxy_pass_naming_no_host_is_no_pipe() -> None: + """`target: {host: ""}` is drift, and drift yields no pipe rather than a pipe to nowhere.""" + snap = snapshot(nginx=nginx_section([nginx_server( + [443], [], [{"location": "/", "scheme": "http", "target": {"host": " ", "port": 80}, + "variable": None}])])) + payload = gateway.parse(snap, "gateway") + assert payload["outbound"] == [] + assert payload["nginx"]["servers"] == [{"port": 443, "server_names": []}] + + +def test_the_readers_refuse_the_shapes_that_are_not_values() -> None: + """The three guards every section read goes through, on the inputs that are not data. + + Pinned directly (the way `tests/test_config.py` pins `_flag`/`_num`) because each one is the + difference between a pipe and no pipe: an empty string is not an address, a bare string is + not a list of names, and whitespace is not a host. + """ + assert gateway._is_ip_literal("") is False + assert gateway._is_ip_literal(None) is False + assert gateway._is_ip_literal("203.0.113.7") is True + assert gateway._is_ip_literal("[::1]") is False, "/proc spells addresses without brackets" + assert gateway._is_ip_literal("app.example.test") is False + assert gateway._names("not a list") == [] + assert gateway._names(None) == [] + assert gateway._names([None, "a-be", "a-be", " ", "b-be"]) == ["a-be", "b-be"] + assert gateway._target_of({"host": " ", "port": 80}) is None + assert gateway._target_of(["host", 80]) is None + assert gateway._target_of({"host": "be.example.test", "port": "nope"}) == ( + "be.example.test", None) + + +# --------------------------------------------------------------------------- # +# the degradation arms: three ways to have nothing, none of them red +# --------------------------------------------------------------------------- # + +def test_an_agent_without_the_sections_predates_gateway_support(gateway_snapshot: Any) -> None: + """The 1.1.0 / `--no-proxy` arm: same keys, empty and null, one sentence, no verdict.""" + payload = gateway.parse(gateway_snapshot("predates"), "gateway") + assert payload["supported"] is False + assert payload["why"] == gateway.WHY_PREDATES + assert "predates gateway support" in payload["why"] + assert payload["haproxy"] == {"frontends": []} + assert payload["nginx"] == {"servers": [], "map_targets": []} + assert payload["inbound"] == [] and payload["outbound"] == [] + assert payload["unmatched"] == {"inbound": None, "outbound": None, "other": None, + "groups": []} + assert payload["sockets"] == {"listens": None, "established": None} + assert payload["sections"] == {"haproxy": None, "nginx": None, "sockets": None} + assert gateway.samples(payload, 1.0) == [], "an unknown count publishes no gauge" + + +def test_a_host_running_no_haproxy_says_so_and_keeps_the_nginx_pipes(gateway_snapshot: Any) -> None: + payload = gateway.parse(gateway_snapshot("no_haproxy"), "gateway") + assert payload["supported"] is True, "the agent knows the feature; the host has no haproxy" + assert payload["sections"]["haproxy"] is False + assert "no haproxy" in (payload["why"] or "") + assert "not found" not in (payload["why"] or ""), "the page's sentence, not the agent's log" + assert payload["section_errors"], "the agent's own line still travels, verbatim" + assert any("not found" in line for line in payload["section_errors"]) + assert [row["port"] for row in payload["nginx"]["servers"]] == [80, 443, 4443, 8443] + assert [pipe["port"] for pipe in payload["inbound"] if pipe["source"] == "nginx"] == \ + [80, 443, 4443, 8443] + assert payload["haproxy"]["frontends"] == [] + + +def test_an_unreadable_config_is_a_different_sentence_from_no_config() -> None: + snap = snapshot(haproxy=haproxy_section([], available=None), + nginx=nginx_section([nginx_server([80], ["a.example.test"])]), + sockets=sockets_section(listens=[listen(80)])) + payload = gateway.parse(snap, "gateway") + assert payload["sections"]["haproxy"] is None + assert "could not be read or parsed" in (payload["why"] or "") + assert [pipe["port"] for pipe in payload["inbound"]] == [80] + + +def test_a_socket_table_that_did_not_answer_leaves_every_count_null( + gateway_snapshot: Any) -> None: + """`available: null` is "no answer", and the only honest counts are `null` (rule 2).""" + payload = gateway.parse(gateway_snapshot("sockets_silent"), "gateway") + assert payload["sections"]["sockets"] is None + assert "sockets:" in (payload["why"] or "") + assert payload["sockets"] == {"listens": None, "established": None} + assert payload["inbound"], "the configured pipes are still drawn — the config did answer" + assert all(pipe["connections"] is None for pipe in payload["inbound"]) + assert all(pipe["connections"] is None for pipe in payload["outbound"]) + assert payload["unmatched"] == {"inbound": None, "outbound": None, "other": None, + "groups": []} + assert gateway.samples(payload, 1.0) == [] + + +def test_a_partial_snapshot_names_the_missing_section_only() -> None: + """A hypothetical agent publishing `sockets` but no configs: supported, with a gap named.""" + snap = snapshot(sockets=sockets_section(listens=[listen(9999)])) + payload = gateway.parse(snap, "gateway") + assert payload["supported"] is True + assert payload["sections"] == {"haproxy": None, "nginx": None, "sockets": True} + assert payload["unmatched"]["other"] == 1, "9999 is a listener nothing claims" + assert "haproxy" in (payload["why"] or "") and "nginx" in (payload["why"] or "") + + +# --------------------------------------------------------------------------- # +# what the payload must never contain, and the gauges it must +# --------------------------------------------------------------------------- # + +def test_no_count_carries_a_verdict_or_a_window(gateway_snapshot: Any) -> None: + """`net.*` precedent for colour, rule 2 for the `Δ` label: a count is a count. + + Both arms are walked, so a `severity` bolted onto a pipe or a `window_seconds` copied from + `sources/host.py` fails here — the first would be a judgement `severity.py` never made, the + second would claim a measurement this source does not take. + """ + for mode in ("full", "predates", "sockets_silent"): + keys = walk_keys(gateway.parse(gateway_snapshot(mode), "gateway")) + assert "severity" not in keys, mode + assert not {key for key in keys if "window" in key or key.startswith("Δ")}, mode + + +def test_the_payload_is_json_serialisable_end_to_end(gateway_snapshot: Any) -> None: + """It goes out over `/api/gateway/snapshot` verbatim, so nothing in it may be exotic.""" + json.dumps(gateway.parse(gateway_snapshot("full"), "gateway")) + + +def test_samples_are_one_gauge_per_pipe_under_the_gateway_family(gateway_snapshot: Any) -> None: + snap = gateway_snapshot("full") + payload = gateway.parse(snap, "gateway") + rows = gateway.samples(payload, 1_760_000_000.0) + assert rows, "a live gateway publishes gauges" + assert {row.source for row in rows} == {gateway.SOURCE} + assert {row.host for row in rows} == {"gateway"} + named = {row.metric: row.value for row in rows} + assert named["gateway.in_conns@443"] == float(pipe_by_port(payload, 443)["connections"]) + counted = [pipe for pipe in payload["inbound"] + payload["outbound"] + if pipe["connections"] is not None] + assert len(named) == len(counted), "one gauge per pipe with a number, and none without" + for pipe in counted: + assert named[pipe["metric"]] == float(pipe["connections"]), ( + "the stored gauge is the number the page prints, never a second reading of it") + + +def test_parse_gateway_snapshot_stamps_its_samples_from_the_snapshot() -> None: + """The x-axis is when the agent took the snapshot, the way `sources/disks.py` does it.""" + result = gateway.parse_gateway_snapshot( + "gateway", snapshot(haproxy=haproxy_section([frontend("ssh", [22])]), + sockets=sockets_section(listens=[listen(22)])), now=999.0) + assert result.events == [], "a connection count has no transition worth an event line" + assert [sample.ts for sample in result.samples] == [1_760_000_000.0] + assert [sample.metric for sample in result.samples] == ["gateway.in_conns@22"] + + +def test_a_snapshot_with_no_timestamp_falls_back_to_the_callers_clock() -> None: + doc = snapshot(haproxy=haproxy_section([frontend("ssh", [22])]), + sockets=sockets_section(listens=[listen(22)])) + doc.pop("ts") + result = gateway.parse_gateway_snapshot("gateway", doc, now=42.0) + assert [sample.ts for sample in result.samples] == [42.0] + + +def test_the_gauges_skip_a_pipe_with_no_number() -> None: + """No sample for an unknown count — a chart of zeros for a dead socket table is the + failure mode rule 2 names, and `sources/disks.py` already refuses it for unreadable drives.""" + payload = gateway.parse(snapshot(haproxy=haproxy_section([frontend("ssh", [22])])), "gateway") + assert payload["inbound"][0]["connections"] is None + assert gateway.samples(payload, 1.0) == [] + + +def test_a_snapshot_whose_sections_are_empty_is_an_honest_empty_page() -> None: + """A 1.2.0 agent on a box with two empty configs: supported, and nothing drawn.""" + snap = snapshot(haproxy=haproxy_section([]), nginx=nginx_section(), + sockets=sockets_section()) + payload = gateway.parse(snap, "gateway") + assert payload["supported"] is True and payload["why"] is None + assert payload["inbound"] == [] and payload["outbound"] == [] + assert payload["sockets"] == {"listens": 0, "established": 0} + assert payload["unmatched"] == {"inbound": 0, "outbound": 0, "other": 0, "groups": []} + + +def test_junk_in_the_sections_yields_no_pipes_and_no_exception() -> None: + """The source is a projection of a document it does not control — drift is `n/a`, not a crash.""" + snap: dict[str, Any] = {"ts": 1.0, "host": "gateway", + "haproxy": {"available": True, "frontends": [3, None], + "backends": {"nope": 1}}, + "nginx": {"available": True, "servers": "not a list", "maps": None}, + "sockets": {"available": True, "listens": [None], + "established": [{"local": 5, "remote": None}]}} + payload = gateway.parse(snap, "gateway") + assert payload["inbound"] == [] and payload["outbound"] == [] + assert payload["unmatched"]["inbound"] == 0 + assert payload["sockets"] == {"listens": 0, "established": 1} + + +def test_a_port_of_zero_or_a_name_of_nothing_is_no_match() -> None: + """`0` is not a port and an empty target is not a host, so neither becomes a pipe (rule 2).""" + snap = snapshot( + haproxy=haproxy_section([frontend("u", [None, 0]), frontend("ok", [22])], + [backend("b", [server("?", "", 22), server("z", "z.example.test", + 0)])]), + sockets=sockets_section(listens=[listen(0), listen(22)])) + payload = gateway.parse(snap, "gateway") + assert [row["port"] for row in payload["haproxy"]["frontends"]] == [22] + assert [pipe["target"]["host"] for pipe in payload["outbound"]] == ["z.example.test"] + assert payload["outbound"][0]["target"]["port"] is None + assert payload["unmatched"]["other"] == 0, "a 0 is not a port, claimed or otherwise" + + +def test_a_use_backend_map_is_reported_as_routes_not_one_chosen_backend() -> None: + """haproxy picks per request here; the payload names every candidate and chooses none.""" + snap = snapshot(haproxy=haproxy_section( + [frontend("fe", [80], default="base-be", use=("blue-be", "green-be"))])) + row = gateway.parse(snap, "gateway")["haproxy"]["frontends"][0] + assert row["backend"] == "base-be" + assert row["routes"] == ["base-be", "blue-be", "green-be"] diff --git a/tests/test_live_fleet.py b/tests/test_live_fleet.py index 776e019..33cd05f 100644 --- a/tests/test_live_fleet.py +++ b/tests/test_live_fleet.py @@ -33,7 +33,7 @@ from dashboards import certs, collectors from dashboards.config import Settings, load from dashboards.rates import RateEstimator from dashboards.severity import classify -from dashboards.sources import nut +from dashboards.sources import gateway, nut from dashboards.sources.disks import SMART_SERIES, DisksCollector from dashboards.sources.host import HostCollector from slots_contract import assert_slot_shape @@ -255,6 +255,46 @@ async def test_disks_source_charts_the_physical_drives_the_agent_reports( pytest.skip("no host agent answered") +async def test_the_gateway_source_joins_the_live_ingress(settings: Settings, + agent_tls: dict[str, collectors.AgentTls]) -> None: + """Phase 29: `sources/gateway.py` over the **real** gateway agent, read-only. + + The production path — the same `fetch_snapshot` the poller will run, through the stored + mTLS authority — so what fails here is a drift between the agent's sections and the join, + not a typo in a fixture. The assertions are the ones that hold whatever the traffic is: + the agent answers the three sections, every configured frontend is a pipe, and the + inbound/outbound/unmatched tallies **partition** the kernel's ESTABLISHED table, which is + what lets the page's numbers reconcile against a direct `/proc/net/tcp` read. + """ + target = next((item for item in settings.host_targets if item.name == settings.gateway_host), + None) + if target is None: + pytest.skip(f"GATEWAY_HOST={settings.gateway_host!r} is not in HOST_TARGETS") + async with httpx.AsyncClient(timeout=settings.gateway_timeout) as client: + try: + snapshot = await collectors.fetch_snapshot(client, target, + timeout=settings.gateway_timeout, + tls=agent_tls.get(target.name)) + except collectors.UpstreamError as exc: + pytest.skip(f"{target.name} did not answer: {exc}") + payload = gateway.parse(snapshot, target.name) + if not payload["supported"]: + pytest.fail(f"{target.name} runs agent {payload.get('agent_version')} — the gateway page " + "needs 1.2.0's haproxy/nginx/sockets sections (AGENTS.md gap 11's rollout)") + assert payload["host"] == target.name + assert payload["sockets"]["established"] == len(snapshot["sockets"]["established"]) + assert payload["inbound"], "no frontend on either config would leave the page empty" + for pipe in payload["inbound"]: + assert isinstance(pipe["connections"], int), "the socket table answered, so a count is known" + for group in payload["unmatched"]["groups"]: + assert group["connections"] > 0 and group["host_count"] > 1, ( + "a group is only ever the unresolved fan-out of several names on one port") + attributed = sum(pipe["connections"] or 0 for pipe in payload["inbound"] + payload["outbound"]) + assert attributed + payload["unmatched"]["inbound"] + payload["unmatched"]["outbound"] == \ + payload["sockets"]["established"], "every socket counted exactly once" + assert all(sample.source == gateway.SOURCE for sample in gateway.samples(payload, 0.0)) + + @pytest.mark.parametrize("engine", ["gufo", "vllm"]) async def test_engine_still_publishes_the_series_we_parse(engine: str, settings: Settings) -> None: