phase: 29_gateway_dashboard_page
Phase 29 completion pass is done. Final state:
**Verification (all re-run this pass, no task files touched)**
- `uv run pytest` → **1746 passed, 16 skipped**; `uv run pytest --cov` → **99.05 %** (gate 94 %); `bash .agents/validate.sh` → **rc=0** (ruff clean, pyright `0 errors, 0 warnings`, `node --check` clean); `uv run pytest -m live --live` → **12 passed**.
- Live, read-only: dev server **10/10 sources `ok`** (incl. `gateway`); `data/logs/29-task06-live.cjs` → **49 PASS / 0 FAIL** with per-port inbound counts reconciled against a direct `/proc/net/tcp{,6}` read (divergence 0); `data/logs/verify-29-task06-predates.py` → 0 failed (1.1.0 arm renders `WHY_PREDATES`, no pipes, every tally `n/a`, job still `ok`); phase-27 mobile audit re-run over all nine served pages → **371 PASS / 0 FAIL / 0 FINDING** (`data/logs/29-final-mobile-audit.out`).
- Criteria: (1) unconfigured `GATEWAY_HOST` → `404 {configured:false}` with dial spies + `#gate-empty` sentence ✅; (2) 1.1.0/`--no-proxy` → `supported:false`, no `0`s, never red ✅; (3) live gateway 1.2.0 → pipes animate, totals reconcile ✅; (4) no `severity`/verdict colour anywhere, validate green ✅; (5) docs updated + gap 13 logged ✅; (6) the five existing dashboards, jobs and `/api/summary` unchanged (live control leg) ✅.
**Docs (consistency pass, incl. pre-existing drift)**
- `README.md`: architecture diagram now lists `sources/gateway.py`, the 30 s gateway job and the third `/snapshot` reader; `GATEWAY_HOST` row says *a name, never a URL*; the `POLL_MS_*` row states `/gateway`'s fixed 15 s browser poll; "nine pollers" → per-source cadences; phase-28 "until phase 29" wording → current state; new Testing paragraph for `tests/test_gateway_source.py` (join pinned from the captures, reconciliation identity, `getaddrinfo` spy).
- `AGENTS.md`: phase-28 stop-point wording replaced with the shipped phase-29 state; suite/coverage figures → 1746 / 16 skipped / 99.08 %→99.05 % band and `~99 %`; window-chips paragraph records that `/gateway` has **no** chips (fixed 24 h); new testing-policy bullet for the join; stale page counts fixed (seven → eight tabbed pages / `MARKUP_PAGES`, clock-order and badge-count transcripts annotated); gap 12 now records the phase-29 nine-page audit re-run.
- `tests/test_api.py`: comment/docstring-only fixes for the "seven pages" wording that phase 29 made eight.
**Deviations / notes**
- Nothing implemented: all six task files were already in `complete/`, so this run was the final verification + docs pass; no git index or phase file was touched.
- Coverage wobbles 99.05–99.08 % between runs on a pre-existing timing-dependent line (`collectors.py` `/slots`-alias `except`/`break`), not a phase-29 path (`sources/gateway.py` is 100 %).
- `uv pip list --outdated` still lists upgrades; dependencies did not move in this phase, so they were left alone rather than upgraded mid-verification.
- Historical browser-transcript counts (phase 18/20/25 script runs) were left as recorded history and annotated instead of rewritten.
**Next phase:** none — `.agents/phases/todo/` is empty after 29; further work needs a new phase authored first.
This commit is contained in:
@@ -0,0 +1,19 @@
|
|||||||
|
Phase 29 completion pass is done. Final state:
|
||||||
|
|
||||||
|
**Verification (all re-run this pass, no task files touched)**
|
||||||
|
- `uv run pytest` → **1746 passed, 16 skipped**; `uv run pytest --cov` → **99.05 %** (gate 94 %); `bash .agents/validate.sh` → **rc=0** (ruff clean, pyright `0 errors, 0 warnings`, `node --check` clean); `uv run pytest -m live --live` → **12 passed**.
|
||||||
|
- Live, read-only: dev server **10/10 sources `ok`** (incl. `gateway`); `data/logs/29-task06-live.cjs` → **49 PASS / 0 FAIL** with per-port inbound counts reconciled against a direct `/proc/net/tcp{,6}` read (divergence 0); `data/logs/verify-29-task06-predates.py` → 0 failed (1.1.0 arm renders `WHY_PREDATES`, no pipes, every tally `n/a`, job still `ok`); phase-27 mobile audit re-run over all nine served pages → **371 PASS / 0 FAIL / 0 FINDING** (`data/logs/29-final-mobile-audit.out`).
|
||||||
|
- Criteria: (1) unconfigured `GATEWAY_HOST` → `404 {configured:false}` with dial spies + `#gate-empty` sentence ✅; (2) 1.1.0/`--no-proxy` → `supported:false`, no `0`s, never red ✅; (3) live gateway 1.2.0 → pipes animate, totals reconcile ✅; (4) no `severity`/verdict colour anywhere, validate green ✅; (5) docs updated + gap 13 logged ✅; (6) the five existing dashboards, jobs and `/api/summary` unchanged (live control leg) ✅.
|
||||||
|
|
||||||
|
**Docs (consistency pass, incl. pre-existing drift)**
|
||||||
|
- `README.md`: architecture diagram now lists `sources/gateway.py`, the 30 s gateway job and the third `/snapshot` reader; `GATEWAY_HOST` row says *a name, never a URL*; the `POLL_MS_*` row states `/gateway`'s fixed 15 s browser poll; "nine pollers" → per-source cadences; phase-28 "until phase 29" wording → current state; new Testing paragraph for `tests/test_gateway_source.py` (join pinned from the captures, reconciliation identity, `getaddrinfo` spy).
|
||||||
|
- `AGENTS.md`: phase-28 stop-point wording replaced with the shipped phase-29 state; suite/coverage figures → 1746 / 16 skipped / 99.08 %→99.05 % band and `~99 %`; window-chips paragraph records that `/gateway` has **no** chips (fixed 24 h); new testing-policy bullet for the join; stale page counts fixed (seven → eight tabbed pages / `MARKUP_PAGES`, clock-order and badge-count transcripts annotated); gap 12 now records the phase-29 nine-page audit re-run.
|
||||||
|
- `tests/test_api.py`: comment/docstring-only fixes for the "seven pages" wording that phase 29 made eight.
|
||||||
|
|
||||||
|
**Deviations / notes**
|
||||||
|
- Nothing implemented: all six task files were already in `complete/`, so this run was the final verification + docs pass; no git index or phase file was touched.
|
||||||
|
- Coverage wobbles 99.05–99.08 % between runs on a pre-existing timing-dependent line (`collectors.py` `/slots`-alias `except`/`break`), not a phase-29 path (`sources/gateway.py` is 100 %).
|
||||||
|
- `uv pip list --outdated` still lists upgrades; dependencies did not move in this phase, so they were left alone rather than upgraded mid-verification.
|
||||||
|
- Historical browser-transcript counts (phase 18/20/25 script runs) were left as recorded history and annotated instead of rewritten.
|
||||||
|
|
||||||
|
**Next phase:** none — `.agents/phases/todo/` is empty after 29; further work needs a new phase authored first.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+74
@@ -0,0 +1,74 @@
|
|||||||
|
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................s............... [ 32%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 61%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
............................................................ssssssssssss [ 73%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
................................................sss..................... [ 98%]
|
||||||
|
.................................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover Missing
|
||||||
|
------------------------------------------------------------------
|
||||||
|
src/dashboards/__init__.py 3 0 100%
|
||||||
|
src/dashboards/__main__.py 14 1 93% 32
|
||||||
|
src/dashboards/alerts.py 182 0 100%
|
||||||
|
src/dashboards/api.py 491 0 100%
|
||||||
|
src/dashboards/app.py 190 0 100%
|
||||||
|
src/dashboards/auth.py 265 0 100%
|
||||||
|
src/dashboards/certs.py 292 0 100%
|
||||||
|
src/dashboards/collectors.py 216 5 98% 323-324, 328-329, 346
|
||||||
|
src/dashboards/config.py 126 0 100%
|
||||||
|
src/dashboards/debug.py 14 0 100%
|
||||||
|
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||||
|
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||||
|
src/dashboards/rates.py 73 0 100%
|
||||||
|
src/dashboards/severity.py 49 0 100%
|
||||||
|
src/dashboards/sources/__init__.py 0 0 100%
|
||||||
|
src/dashboards/sources/disks.py 134 0 100%
|
||||||
|
src/dashboards/sources/gateway.py 321 0 100%
|
||||||
|
src/dashboards/sources/host.py 209 0 100%
|
||||||
|
src/dashboards/sources/llm.py 178 1 99% 285
|
||||||
|
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||||
|
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||||
|
src/dashboards/store.py 313 2 99% 553-554
|
||||||
|
------------------------------------------------------------------
|
||||||
|
TOTAL 3894 37 99%
|
||||||
|
Required test coverage of 94% reached. Total coverage: 99.05%
|
||||||
|
1746 passed, 16 skipped, 1 warning in 167.20s (0:02:47)
|
||||||
|
== uv run ruff check src tests agent ==
|
||||||
|
All checks passed!
|
||||||
|
== uv run pyright ==
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
== node --check src/dashboards/web/static/*.js ==
|
||||||
|
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||||
|
certifi 2026.10.19 2026.10.23 wheel
|
||||||
|
coverage 7.16.2 7.17.0 wheel
|
||||||
|
fastapi 0.153.0 0.161.1 wheel
|
||||||
|
pycparser 3.11 3.12 wheel
|
||||||
|
ruff 0.17.1 0.17.2 wheel
|
||||||
+15
@@ -0,0 +1,15 @@
|
|||||||
|
**Implemented**
|
||||||
|
- `config.py`: `gateway_host` (`GATEWAY_HOST`, default `gateway`), `gateway_interval` (30 s), `gateway_timeout` (10 s), `gateway_enabled` (name must be a `HOST_TARGETS` entry, `nut_enabled` shape) and it now counts in `any_source_enabled`.
|
||||||
|
- New `src/dashboards/sources/gateway.py`: `parse(snapshot, host)` joins agent 1.2.0's `haproxy`/`nginx`/`sockets` into inbound pipes (per listen port: domains, live count, distinct clients) and outbound pipes (haproxy `server`, concrete `proxy_pass`, `$var`→`map` fan-out), plus `unmatched` tallies, `samples()` gauges (`gateway.in_conns@<port>`, `gateway.out_conns@<host>:<port>`) and `parse_gateway_snapshot() -> Result`. Gauges only: no estimator/`Δ` window, no `severity`, `null` (never `0`) whenever the socket table or the join cannot answer; `supported:false` (1.1.0/`--no-proxy`) + per-section `why` + verbatim `section_errors`.
|
||||||
|
- Join never resolves a name: `socket.getaddrinfo` spy pins zero calls; exact (ip,port) match `by_port:false`, name matched on port only when that port has one configured host (`by_port:true`), several names on one port → nobody chosen (`connections:null`, sockets in `unmatched.outbound`, one `unmatched.groups` entry naming port/count/candidates). `join_sockets` attributes each socket once, so inbound+outbound+unmatched == the socket table.
|
||||||
|
- Tests: `tests/test_gateway_source.py` (35, incl. a fixture built by running the **real agent `Collector`** over the captured configs via new `gateway_snapshot` conftest fixture), 3 new `tests/test_config.py` cases, and a read-only `--live` test against the real gateway agent. Probe: `data/logs/verify-29-task01.py` → `29-task01-live.out`.
|
||||||
|
- Docs: `.env.example` + `.env.dev.example` gateway block, README Configuration rows, AGENTS.md (config bullet, new `sources/gateway.py` responsibility bullet, metric-naming/gauge family), `store.py` family docstring (`disks`/`gateway` added).
|
||||||
|
|
||||||
|
**Results** — `uv run pytest`: 1691 passed, 16 skipped; `uv run pytest --cov`: 99.05% total, `sources/gateway.py` and `config.py` 100%; `uv run ruff check src tests agent` + `uv run pyright`: clean; `bash .agents/validate.sh`: green; `uv run pytest -m live --live`: 12 passed; dev server restarted, all nine sources `ok`. Live join: 10 inbound / 31 outbound / 17 map entries, partition `19 + 1 + 8 = 28 == established` OK, zero `severity` keys.
|
||||||
|
|
||||||
|
**Deviations (documented in code + AGENTS.md)**
|
||||||
|
- Outbound series carry the port (`@host:port`, vs the phase's host-only sketch): this gateway's two haproxy Minecraft backends share `minecraft.reeselink.com` on 25565/25566, and one name would `INSERT OR REPLACE` one pipe's gauge over the other in silence (rule 2).
|
||||||
|
- The shared-port fan-out (16 `*.reeselink.com:443` map entries) yields `null` counts + `unmatched.groups` rather than a guessed or repeated attribution; `unmatched.other` is defined as live listeners no config claims (phase 28's "unclaimed socket").
|
||||||
|
- No live infra changes were made (read-only dial only). Note: `NOTES.md` is empty in the working tree (pre-existing, not this task); its two items are already shipped (phase 24 mTLS, phase 20 task 04 admin links), so I left it as-is.
|
||||||
|
|
||||||
|
**Next pending task:** `.agents/phases/todo/29_gateway_dashboard_page/02_wiring.md` (the `gateway` poller job, alerts rule, `/api/summary` block, `/api/config` entry).
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+72
@@ -0,0 +1,72 @@
|
|||||||
|
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 21%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
.......s................................................................ [ 33%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 46%]
|
||||||
|
........................................................................ [ 50%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 63%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
.....ssssssssssss....................................................... [ 75%]
|
||||||
|
........................................................................ [ 80%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
.................................................................sss.... [ 97%]
|
||||||
|
................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover Missing
|
||||||
|
------------------------------------------------------------------
|
||||||
|
src/dashboards/__init__.py 3 0 100%
|
||||||
|
src/dashboards/__main__.py 14 1 93% 32
|
||||||
|
src/dashboards/alerts.py 176 0 100%
|
||||||
|
src/dashboards/api.py 440 0 100%
|
||||||
|
src/dashboards/app.py 186 0 100%
|
||||||
|
src/dashboards/auth.py 265 0 100%
|
||||||
|
src/dashboards/certs.py 292 0 100%
|
||||||
|
src/dashboards/collectors.py 206 4 98% 304, 308-309, 326
|
||||||
|
src/dashboards/config.py 126 0 100%
|
||||||
|
src/dashboards/debug.py 14 0 100%
|
||||||
|
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||||
|
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||||
|
src/dashboards/rates.py 73 0 100%
|
||||||
|
src/dashboards/severity.py 49 0 100%
|
||||||
|
src/dashboards/sources/__init__.py 0 0 100%
|
||||||
|
src/dashboards/sources/disks.py 134 0 100%
|
||||||
|
src/dashboards/sources/gateway.py 307 0 100%
|
||||||
|
src/dashboards/sources/host.py 209 0 100%
|
||||||
|
src/dashboards/sources/llm.py 178 1 99% 285
|
||||||
|
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||||
|
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||||
|
src/dashboards/store.py 313 2 99% 553-554
|
||||||
|
------------------------------------------------------------------
|
||||||
|
TOTAL 3809 36 99%
|
||||||
|
Required test coverage of 94% reached. Total coverage: 99.05%
|
||||||
|
1691 passed, 16 skipped, 1 warning in 163.88s (0:02:43)
|
||||||
|
== uv run ruff check src tests agent ==
|
||||||
|
All checks passed!
|
||||||
|
== uv run pyright ==
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
== node --check src/dashboards/web/static/*.js ==
|
||||||
|
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||||
|
coverage 7.16.2 7.17.0 wheel
|
||||||
|
fastapi 0.153.0 0.161.1 wheel
|
||||||
|
pycparser 3.11 3.12 wheel
|
||||||
|
ruff 0.17.1 0.17.2 wheel
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
Everything is green. Final state:
|
||||||
|
|
||||||
|
**Implementation** (`src/dashboards/`)
|
||||||
|
- `collectors.py`: `fetch_gateway_snapshot()` (third reader of the same agent `/snapshot`) + `gateway_job()` — **one** job (the `nut` shape), key/dashboard `gateway`, `label` = `GATEWAY_HOST`'s name, `GATEWAY_INTERVAL_SECONDS`/`GATEWAY_TIMEOUT_SECONDS`, **no** `RateEstimator`, `None` when the name isn't a target (invariant 3).
|
||||||
|
- `app.build_hub`: adds that job between `disks` and `llm` on the **same** `AgentTls` entry as `host:`/`disks:`, so one revoke fails all three without a dial.
|
||||||
|
- `alerts.py`: `HREF["/gateway"]`, staleness in the gateway's own 30 s interval, `AGENT_ECHO_DASHBOARDS = ("disks", "gateway")` with `_partner_host()` (gateway finds its host by **label**, not key) — dead agent = one line, wedged sections while the host answers = the only voice; **no** gateway threshold or payload argument.
|
||||||
|
- `api.py`: additive `/api/summary.gateway` (`configured, host, supported, unreachable, inbound, outbound, map_targets, unmatched, unmatched_listeners, partial, states`; `null` never `0` in the arms that didn't measure) and the `/api/config.gateway` row with redacted `display_url`.
|
||||||
|
|
||||||
|
**Tests/lint/coverage**
|
||||||
|
- 19 new tests (5 alerts, 11 api, 3 collectors); full suite **1712 passed, 16 skipped**, coverage **99.07%** (was 99.05%), `alerts.py`/`api.py`/`app.py` back to **100%**; `ruff` + `pyright` clean; `bash .agents/validate.sh` exit 0 (dep list is the pre-existing advisory).
|
||||||
|
- Mutation-checked: breaking the gateway label lookup fails 3 pins, so the echo rule isn't vacuous.
|
||||||
|
- Live (`--live`, `.env.dev`): **12 passed**; dev server restarted — `/api/health` **ten sources ok** incl. `gateway`, `/api/summary.gateway` live (in 12 / out 4 / map 17 / unmatched 10+2 / partial), `/api/config.gateway` `{enabled, gateway, https://10.1.1.27:9120, 30, 10}`.
|
||||||
|
|
||||||
|
**Docs**: AGENTS.md (poller job-of-its-own, collectors, alerts, api block, app, "ten sources" fleet fact) and README (alerts echo/interval rule, summary+config blocks).
|
||||||
|
|
||||||
|
**Decisions**: `states` as the repo-standard list of health records (not a `{gateway: str}` map); added `partial` so a sum that omitted an unattributable port isn't read as the total; no `severity` anywhere.
|
||||||
|
|
||||||
|
**Next pending task**: `29_gateway_dashboard_page/03_api_routes.md` — `/api/gateway/snapshot` + `/api/gateway/history`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+72
@@ -0,0 +1,72 @@
|
|||||||
|
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 29%]
|
||||||
|
.........................s.............................................. [ 33%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 50%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 58%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
..........................ssssssssssss.................................. [ 75%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
..............sss....................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover Missing
|
||||||
|
------------------------------------------------------------------
|
||||||
|
src/dashboards/__init__.py 3 0 100%
|
||||||
|
src/dashboards/__main__.py 14 1 93% 32
|
||||||
|
src/dashboards/alerts.py 182 0 100%
|
||||||
|
src/dashboards/api.py 473 0 100%
|
||||||
|
src/dashboards/app.py 189 0 100%
|
||||||
|
src/dashboards/auth.py 265 0 100%
|
||||||
|
src/dashboards/certs.py 292 0 100%
|
||||||
|
src/dashboards/collectors.py 216 4 98% 324, 328-329, 346
|
||||||
|
src/dashboards/config.py 126 0 100%
|
||||||
|
src/dashboards/debug.py 14 0 100%
|
||||||
|
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||||
|
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||||
|
src/dashboards/rates.py 73 0 100%
|
||||||
|
src/dashboards/severity.py 49 0 100%
|
||||||
|
src/dashboards/sources/__init__.py 0 0 100%
|
||||||
|
src/dashboards/sources/disks.py 134 0 100%
|
||||||
|
src/dashboards/sources/gateway.py 307 0 100%
|
||||||
|
src/dashboards/sources/host.py 209 0 100%
|
||||||
|
src/dashboards/sources/llm.py 178 1 99% 285
|
||||||
|
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||||
|
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||||
|
src/dashboards/store.py 313 2 99% 553-554
|
||||||
|
------------------------------------------------------------------
|
||||||
|
TOTAL 3861 36 99%
|
||||||
|
Required test coverage of 94% reached. Total coverage: 99.07%
|
||||||
|
1712 passed, 16 skipped, 1 warning in 164.73s (0:02:44)
|
||||||
|
== uv run ruff check src tests agent ==
|
||||||
|
All checks passed!
|
||||||
|
== uv run pyright ==
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
== node --check src/dashboards/web/static/*.js ==
|
||||||
|
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||||
|
coverage 7.16.2 7.17.0 wheel
|
||||||
|
fastapi 0.153.0 0.161.1 wheel
|
||||||
|
pycparser 3.11 3.12 wheel
|
||||||
|
ruff 0.17.1 0.17.2 wheel
|
||||||
+13
@@ -0,0 +1,13 @@
|
|||||||
|
**Task 03 complete — `/api/gateway/snapshot` + `/api/gateway/history`.**
|
||||||
|
|
||||||
|
**Implementation** — `api.gateway_router()` (both GETs, included after `disks_router()` in `app.create_app`): `/snapshot` tests `settings.gateway_enabled` in its first line and raises `404 {"configured": false}` naming `GATEWAY_HOST` before any client/fetch/store read; answering → `sources/gateway.parse`'s payload verbatim + `states` (payload `ts` untouched); never-answered → new `gateway_source.stub(host, ts)` (`sources/gateway.py`: `_skeleton()` shared with `parse`, `WHY_NO_ANSWER`, `unreachable: true`, `supported: null`, every count `null`). `/history` goes through the one `_history_batch()` → `store.history_many()` door, `hosts=` defaulting to the ingress alone (no configured gate, since history dials nothing — the `/api/disks/history` rule).
|
||||||
|
|
||||||
|
**Tests** — 16 new tests in `tests/test_api.py` (both unconfigured arms — unset name and a name that is not a target — spy-pinned on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`, with a control leg that names a real target; the stub's keys/counts/`states`; the payload↔`parse` key-set equality; inbound+outbound+unmatched == `sockets.established`; a recursive no-`severity`/no-`window_seconds` walk; history points re-read through `store.query`; unknown names → empty series; host default; family isolation in `/api/metrics?source=`; the two GETs, 405 on the rest, and the app's one upstream POST still `/api/ups/command`), plus the two paths added to `test_auth_api.py::PROTECTED_API`. Both controls were verified by mutation (gate moved after the fetch → refusal tests fail; batch source changed `gateway`→`host` → history tests fail).
|
||||||
|
|
||||||
|
**Gate** — `uv run pytest --cov`: **1728 passed, 16 skipped**, coverage **99.05%** (`api.py`, `app.py`, `sources/gateway.py` each 100%); `ruff` and `pyright` clean; `bash .agents/validate.sh` **exit 0** (only the pre-existing dependency advisory; no deps moved); `-m live --live` 12 passed. Dev server restarted: all ten sources `ok`, and live `/api/gateway/snapshot` against agent 1.2.0 answered 6 frontends / 17 map targets / 31 outbound pipes with the identity reconciling exactly (39 == 39), while `/history` served `gateway.in_conns@443` and empty for a name that never measured.
|
||||||
|
|
||||||
|
**Docs** — README and AGENTS `api.py` bullets now describe the pair (three arms, the gate-before-dial, the stub, the port-in-the-name gauges, no write), AGENTS records `gateway_source.stub()` and that all five `history` routes share the batched door.
|
||||||
|
|
||||||
|
**Deviations** — none. `tests/card_contract.py` untouched (task 04's `COLUMNS` device); task file left in `todo/`, no git commands run.
|
||||||
|
|
||||||
|
**Next task** — `.agents/phases/todo/29_gateway_dashboard_page/04_page.md`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+73
@@ -0,0 +1,73 @@
|
|||||||
|
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
.......................................s................................ [ 33%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 41%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 61%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
..........................................ssssssssssss.................. [ 74%]
|
||||||
|
........................................................................ [ 78%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
..............................sss....................................... [ 99%]
|
||||||
|
................ [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover Missing
|
||||||
|
------------------------------------------------------------------
|
||||||
|
src/dashboards/__init__.py 3 0 100%
|
||||||
|
src/dashboards/__main__.py 14 1 93% 32
|
||||||
|
src/dashboards/alerts.py 182 0 100%
|
||||||
|
src/dashboards/api.py 491 0 100%
|
||||||
|
src/dashboards/app.py 190 0 100%
|
||||||
|
src/dashboards/auth.py 265 0 100%
|
||||||
|
src/dashboards/certs.py 292 0 100%
|
||||||
|
src/dashboards/collectors.py 216 5 98% 323-324, 328-329, 346
|
||||||
|
src/dashboards/config.py 126 0 100%
|
||||||
|
src/dashboards/debug.py 14 0 100%
|
||||||
|
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||||
|
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||||
|
src/dashboards/rates.py 73 0 100%
|
||||||
|
src/dashboards/severity.py 49 0 100%
|
||||||
|
src/dashboards/sources/__init__.py 0 0 100%
|
||||||
|
src/dashboards/sources/disks.py 134 0 100%
|
||||||
|
src/dashboards/sources/gateway.py 321 0 100%
|
||||||
|
src/dashboards/sources/host.py 209 0 100%
|
||||||
|
src/dashboards/sources/llm.py 178 1 99% 285
|
||||||
|
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||||
|
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||||
|
src/dashboards/store.py 313 2 99% 553-554
|
||||||
|
------------------------------------------------------------------
|
||||||
|
TOTAL 3894 37 99%
|
||||||
|
Required test coverage of 94% reached. Total coverage: 99.05%
|
||||||
|
1728 passed, 16 skipped, 1 warning in 167.39s (0:02:47)
|
||||||
|
== uv run ruff check src tests agent ==
|
||||||
|
All checks passed!
|
||||||
|
== uv run pyright ==
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
== node --check src/dashboards/web/static/*.js ==
|
||||||
|
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||||
|
coverage 7.16.2 7.17.0 wheel
|
||||||
|
fastapi 0.153.0 0.161.1 wheel
|
||||||
|
pycparser 3.11 3.12 wheel
|
||||||
|
ruff 0.17.1 0.17.2 wheel
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
**Task 04 complete — `/gateway` page with animated pipes. Everything green.**
|
||||||
|
|
||||||
|
- **Page**: `web/pages/gateway.html` + `web/static/page-gateway.js` + a `/* ---- gateway pipes */` CSS section; route added in `app.py`. Four cards (`haproxy`, `nginx`, `map`, `unmatched`) keyed by each pipe's own `source`; hand-rolled SVG pipes (`<path>`, `vector-effect`), node label = `snapshot.host`, solo-side layouts for the stream map.
|
||||||
|
- **Honesty pins**: labels/counts/clients are `textContent` only; `null` → `n/a` (no `|| 0` on any count); a pipe animates **only** when `connections > 0`, `.still` otherwise; `prefers-reduced-motion` names **both** states; no `ok|warn|fail` token anywhere in the gateway CSS section; every plot carries `ariaLabel`; one `/api/gateway/history` request per card, plot built only while its `Charts` disclosure is open.
|
||||||
|
- **Degradation** (browser-proved, all from payload/route text): unconfigured → setup block naming `GATEWAY_HOST` + agent 1.2.0, zero red, all `n/a`; predates / never-answered → banner prints the payload's own `why`, nothing drawn, dimmed; silent socket table → config map stays, every count `n/a`.
|
||||||
|
- **Two defects found and fixed**: `LineChart._yExtent()` padded a flat series **below** its declared `yMin` (an idle pipe drew a `-1` connection axis) — shared fix, measured on `/system` `/disks` `/llm` `/ups` (no axis moved); and an all-unknown remainder/total rendered a card of `n/a`s or a `0` — now hidden/`n/a`.
|
||||||
|
- **Tests**: 14 new served-byte pins in `tests/test_api.py` (+3 existing audits widened to nine pages/twelve assets), `/gateway` added to `tests/test_auth_api.py`'s protected pages and served route table.
|
||||||
|
- **Browser (live fleet)**: `data/logs/29-task04-{pipes,degradation,unconfigured,charts-floor}.cjs` → all pass; 41 live pipes, `15 + 4 + 11 = 30 = established`; screenshots `29-task04-{gateway,charts-open,phone,predates,sockets_silent,unanswered,unconfigured}.png`, transcripts `.out`.
|
||||||
|
- **Gate**: `pytest` 1742 passed / 16 skipped, coverage 99.08 % (floor 94), `ruff` + `pyright` clean, `node --check` clean; dev server up with all ten sources `ok`.
|
||||||
|
- Docs (`AGENTS.md`/`README.md`) and the sixth masthead tab are task 05/06's deliverables and were deliberately left untouched; `PAGES` in `test_api.py` stays the five-tab table so task 05 moves it as one contract.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+73
@@ -0,0 +1,73 @@
|
|||||||
|
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
....................................................s................... [ 32%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 61%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................ssssssssssss.... [ 73%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
............................................sss......................... [ 98%]
|
||||||
|
.............................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover Missing
|
||||||
|
------------------------------------------------------------------
|
||||||
|
src/dashboards/__init__.py 3 0 100%
|
||||||
|
src/dashboards/__main__.py 14 1 93% 32
|
||||||
|
src/dashboards/alerts.py 182 0 100%
|
||||||
|
src/dashboards/api.py 491 0 100%
|
||||||
|
src/dashboards/app.py 190 0 100%
|
||||||
|
src/dashboards/auth.py 265 0 100%
|
||||||
|
src/dashboards/certs.py 292 0 100%
|
||||||
|
src/dashboards/collectors.py 216 5 98% 323-324, 328-329, 346
|
||||||
|
src/dashboards/config.py 126 0 100%
|
||||||
|
src/dashboards/debug.py 14 0 100%
|
||||||
|
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||||
|
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||||
|
src/dashboards/rates.py 73 0 100%
|
||||||
|
src/dashboards/severity.py 49 0 100%
|
||||||
|
src/dashboards/sources/__init__.py 0 0 100%
|
||||||
|
src/dashboards/sources/disks.py 134 0 100%
|
||||||
|
src/dashboards/sources/gateway.py 321 0 100%
|
||||||
|
src/dashboards/sources/host.py 209 0 100%
|
||||||
|
src/dashboards/sources/llm.py 178 1 99% 285
|
||||||
|
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||||
|
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||||
|
src/dashboards/store.py 313 2 99% 553-554
|
||||||
|
------------------------------------------------------------------
|
||||||
|
TOTAL 3894 37 99%
|
||||||
|
Required test coverage of 94% reached. Total coverage: 99.05%
|
||||||
|
1742 passed, 16 skipped, 1 warning in 161.90s (0:02:41)
|
||||||
|
== uv run ruff check src tests agent ==
|
||||||
|
All checks passed!
|
||||||
|
== uv run pyright ==
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
== node --check src/dashboards/web/static/*.js ==
|
||||||
|
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||||
|
coverage 7.16.2 7.17.0 wheel
|
||||||
|
fastapi 0.153.0 0.161.1 wheel
|
||||||
|
pycparser 3.11 3.12 wheel
|
||||||
|
ruff 0.17.1 0.17.2 wheel
|
||||||
+18
@@ -0,0 +1,18 @@
|
|||||||
|
All gates green. Final state:
|
||||||
|
|
||||||
|
**Done (task 05)**
|
||||||
|
- `Gateway` tab added to all **eight** signed-in pages (six dashboards + `/settings` + `/certs`), `aria-current` on `/gateway` only; tab audit rewritten as `nav hrefs == TAB_ORDER` (`PAGES` is now the six dashboards) with per-page `aria-current` asserted in both directions; `test_the_certs_page_is_a_page_and_not_a_sixth_dashboard` renamed `…_not_a_dashboard` now that `/gateway` *is* the sixth.
|
||||||
|
- Overview ingress line: `#fleet-gate` in the fleet panel, rendered by `renderGateway()` from `/api/summary`'s block (`GATEWAY_FIELDS` resolved against the live payload — the `SLOT_FIELDS` device), `.stale` when unreachable, **hidden when `configured: false`**. `/api/summary`'s `gateway` block gained an additive `href` derived from `alerts.HREF`, so the row links with data instead of a page-side route literal (same rule the alert rows keep).
|
||||||
|
- Pins: `SUMMARY_CONTRACT` now names every gateway key; `NAV_STATUS_BADGES` gained `gateway`; new `test_the_overview_line_reads_the_gateway_block_it_claims` and `test_the_gateway_block_links_to_the_page_the_alert_row_links_to`.
|
||||||
|
- Docs: AGENTS.md (six-page/nav/badge counts, the ingress line, the `href` derivation, the wrap rule, the nine-page audit) and README.md updated.
|
||||||
|
|
||||||
|
**Deviations / defects found**
|
||||||
|
- Task assumption "the fifth `#nav-status` badge with **no JS change**" was false — `initChrome()`'s badge table is a literal, so app.js gained `{ key: 'gateway', label: 'gateway', href: '/gateway' }` (pinned).
|
||||||
|
- The fifth badge **broke phase 18's guarantee**: `#nav-status` was `nowrap` at ≤640px, so five chips (352px) widened the document to 372px — a sideways scroll on a phone. Fixed by letting the badge row wrap inside its own row (no label shrunk, nothing hidden); the served-CSS pin flipped to demand `wrap`, and the phase-27 mobile audit (extended to include `/gateway`) re-runs clean.
|
||||||
|
|
||||||
|
**Verification**
|
||||||
|
- `bash .agents/validate.sh`: **1744 passed, 16 skipped**, coverage 99.08%, ruff + pyright + `node --check` clean.
|
||||||
|
- `data/logs/29-task05-tabs.cjs` → `29-task05-tabs.out`: **97 PASS / 0 FAIL** (8 pages × 1280/360 six tabs + one-own-current + five populated badges + no sideways scroll; 360px dropdown lists all six; the tab navigates and draws the tallies; the `/` line reads the block's counts/href, and a wire-simulated `configured:false` renders nothing). Screenshots `data/logs/masthead-29-*.png`.
|
||||||
|
- `data/logs/27-task02-mobile-audit.cjs` (now including `/gateway`): **0 failures, 0 findings**.
|
||||||
|
- Dev server up; `/api/health` and `dev-server.sh status` report all ten sources `ok`.
|
||||||
|
- validate.sh's dependency advisory (coverage/fastapi/pycparser/ruff newer) is pre-existing; this task touched no dependencies.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+73
@@ -0,0 +1,73 @@
|
|||||||
|
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
......................................................s................. [ 32%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 61%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
..........................................................ssssssssssss.. [ 73%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 90%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
..............................................sss....................... [ 98%]
|
||||||
|
................................ [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover Missing
|
||||||
|
------------------------------------------------------------------
|
||||||
|
src/dashboards/__init__.py 3 0 100%
|
||||||
|
src/dashboards/__main__.py 14 1 93% 32
|
||||||
|
src/dashboards/alerts.py 182 0 100%
|
||||||
|
src/dashboards/api.py 491 0 100%
|
||||||
|
src/dashboards/app.py 190 0 100%
|
||||||
|
src/dashboards/auth.py 265 0 100%
|
||||||
|
src/dashboards/certs.py 292 0 100%
|
||||||
|
src/dashboards/collectors.py 216 5 98% 323-324, 328-329, 346
|
||||||
|
src/dashboards/config.py 126 0 100%
|
||||||
|
src/dashboards/debug.py 14 0 100%
|
||||||
|
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||||
|
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||||
|
src/dashboards/rates.py 73 0 100%
|
||||||
|
src/dashboards/severity.py 49 0 100%
|
||||||
|
src/dashboards/sources/__init__.py 0 0 100%
|
||||||
|
src/dashboards/sources/disks.py 134 0 100%
|
||||||
|
src/dashboards/sources/gateway.py 321 0 100%
|
||||||
|
src/dashboards/sources/host.py 209 0 100%
|
||||||
|
src/dashboards/sources/llm.py 178 1 99% 285
|
||||||
|
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||||
|
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||||
|
src/dashboards/store.py 313 2 99% 553-554
|
||||||
|
------------------------------------------------------------------
|
||||||
|
TOTAL 3894 37 99%
|
||||||
|
Required test coverage of 94% reached. Total coverage: 99.05%
|
||||||
|
1744 passed, 16 skipped, 1 warning in 164.35s (0:02:44)
|
||||||
|
== uv run ruff check src tests agent ==
|
||||||
|
All checks passed!
|
||||||
|
== uv run pyright ==
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
== node --check src/dashboards/web/static/*.js ==
|
||||||
|
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||||
|
coverage 7.16.2 7.17.0 wheel
|
||||||
|
fastapi 0.153.0 0.161.1 wheel
|
||||||
|
pycparser 3.11 3.12 wheel
|
||||||
|
ruff 0.17.1 0.17.2 wheel
|
||||||
+34
@@ -0,0 +1,34 @@
|
|||||||
|
Task 06 is complete. Final state:
|
||||||
|
|
||||||
|
## Done
|
||||||
|
|
||||||
|
**Docs**
|
||||||
|
- `README.md` — intro now *five* dashboards; `/gateway` page-table row (knob + **agent 1.2.0** requirement + the unconfigured-rather-than-red arm), the `GATEWAY_HOST` / `GATEWAY_INTERVAL_SECONDS` / `GATEWAY_TIMEOUT_SECONDS` config rows, and the user-facing paragraph on the topology question and both degraded arms.
|
||||||
|
- `AGENTS.md` — route-table row, the `/gateway` UI-conventions paragraph (pipes, `n/a`-not-`0`, no verdict colour, the label-line rule, one history request per card), the metric-naming paragraph's `gateway.in_conns@<listen-port>` / `gateway.out_conns@<host>:<port>`, the config bullet's `default \`gateway\``, and **gap 13** (the residual label width, with the rejected alternative and three costed options).
|
||||||
|
- New pin `tests/test_api.py::test_the_docs_name_the_page_its_knob_and_its_gauges` so prose can't rot: both page rows name `/gateway` + `GATEWAY_HOST` + 1.2.0, the bullet names the default, the gauges and `/api/gateway/*` stay named.
|
||||||
|
|
||||||
|
**Live read-only pass** — `data/logs/29-task06-live.cjs` → `29-task06-live.out`: **49 PASS / 0 FAIL**
|
||||||
|
- payload: `configured`/`supported`, agent `1.2.0`, **6 haproxy frontends** (each port→backend), nginx listening on **80/443/4443/8443**, **17 stream-map entries** (fan-out 443×16 + 4443×1), every entry's key/host/port complete.
|
||||||
|
- page: 4 cards, 41 aria-labelled pipes matching the payload's sentence shape, header tallies = payload figures, `in + out + unmatched == sockets.established` exact, `by port` chip on every `by_port` pipe, collapsed→open chart labelled with legend rows naming exactly that card's pipes.
|
||||||
|
- `/proc/net/tcp{,6}` via `ssh gateway-ops cat`: per-port inbound == ESTABLISHED-on-that-port (≤1 tolerance), `unmatched.inbound` == the port complement (`2222`, `9120`), `by_port` spot-check page 2 vs /proc 2, ambiguous `:443` → 16 null pipes + one `groups` sentence, and nulls never became `0`.
|
||||||
|
- control leg: nine pre-existing sources `ok`; `/` 3 fleet rows, `/system` 3 cards, `/disks` 2 hosts (gateway still hidden), `/llm` 2 engines, `/ups` 2 UPSes; `/`'s ingress line renders and disappears on `configured:false`; 360px has no sideways scroll.
|
||||||
|
- Screenshots: `29-task06-{gateway,card-haproxy,card-stream,charts-expanded,phone}.png`. Transcripts grepped clean of PEM/token/cookie.
|
||||||
|
|
||||||
|
**Pre-1.2.0 arm** — `data/logs/verify-29-task06-predates.py` (+ `.cjs`) → **PASS, 0 failed checks**: throwaway loopback dashboard + scratch SQLite + a stub agent from `tests/fixtures/agent_gateway.json` stamped `1.1.0` with the three sections stripped → `supported: false` with `WHY_PREDATES`, page prints that sentence, draws no cards, every tally `n/a`, banner `info` not red, job `ok`. Screenshot `29-task06-predates.png`.
|
||||||
|
|
||||||
|
## Defect found and fixed
|
||||||
|
The pass caught the page clipping its own answer: pipe labels showed **seven characters** of a 40-character target (`.pipe-label` was one `nowrap` flex row, so the name shrank to what the chip/footnote/count left). Wrapping the text inside that box was tried and measured worse (eleven lines of one word), so the fix gives the label the line: `flex-wrap: wrap` + `flex: 1 1 max-content` on `.pipe-name` → the name now gets the whole column and the chip/footnote/count wrap beneath. `test_a_clipped_pipe_label_keeps_its_full_text_on_the_element` pins both declarations plus the `title`/`aria-label` that keep the remaining ellipsis from being a lost fact; the residual (28 of 42 at 1440px, 16 of 42 at 360px) is gap 13 with the options ordered by cost.
|
||||||
|
|
||||||
|
## Gates
|
||||||
|
- `bash .agents/validate.sh` → **rc=0**: 1746 passed, 16 skipped, coverage **99.05 %**, ruff clean, pyright 0 errors, `node --check` clean.
|
||||||
|
- `./scripts/dev-server.sh status`: all **ten** sources `ok`.
|
||||||
|
- Regression control for the CSS change: phase 27's nine-page audit re-ran — **371 PASS, 0 failures, 0 findings**.
|
||||||
|
- validate.sh's dependency advisory (coverage/fastapi/pycparser/ruff newer) is pre-existing; no dependency moved.
|
||||||
|
|
||||||
|
## Notable decisions
|
||||||
|
- The kernel cross-check asserts the **payload's own partition** (`attributed + unmatched == established`) and reports the raw `/proc` total as a finding — two reads 0.8 s apart cannot be equal on a busy ingress, and pretending otherwise would make the pass flaky about a fact that isn't a bug. The per-port table is the real join proof.
|
||||||
|
- Payload and DOM are read payload → probe → payload, retrying if `ts` moved, so a 30 s poll can't compare a picture to a newer document.
|
||||||
|
- The 1.1.0 arm was proven on a loopback stub, never by rolling a fleet host back.
|
||||||
|
|
||||||
|
## Next pending task
|
||||||
|
This was phase 29's last file — `.agents/phases/todo/29_gateway_dashboard_page/` now holds only `00_phase.md` and `06_docs_and_verify.md`, so the phase is ready to move to `complete/`. No other phase is in `todo/`; gap 13 (pipe-label column width) is the one open item this task surfaced.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+74
@@ -0,0 +1,74 @@
|
|||||||
|
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||||
|
........................................................................ [ 4%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 12%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 24%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................s............... [ 32%]
|
||||||
|
........................................................................ [ 36%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 53%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 61%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
............................................................ssssssssssss [ 73%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 85%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
................................................sss..................... [ 98%]
|
||||||
|
.................................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover Missing
|
||||||
|
------------------------------------------------------------------
|
||||||
|
src/dashboards/__init__.py 3 0 100%
|
||||||
|
src/dashboards/__main__.py 14 1 93% 32
|
||||||
|
src/dashboards/alerts.py 182 0 100%
|
||||||
|
src/dashboards/api.py 491 0 100%
|
||||||
|
src/dashboards/app.py 190 0 100%
|
||||||
|
src/dashboards/auth.py 265 0 100%
|
||||||
|
src/dashboards/certs.py 292 0 100%
|
||||||
|
src/dashboards/collectors.py 216 4 98% 324, 328-329, 346
|
||||||
|
src/dashboards/config.py 126 0 100%
|
||||||
|
src/dashboards/debug.py 14 0 100%
|
||||||
|
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||||
|
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||||
|
src/dashboards/rates.py 73 0 100%
|
||||||
|
src/dashboards/severity.py 49 0 100%
|
||||||
|
src/dashboards/sources/__init__.py 0 0 100%
|
||||||
|
src/dashboards/sources/disks.py 134 0 100%
|
||||||
|
src/dashboards/sources/gateway.py 321 0 100%
|
||||||
|
src/dashboards/sources/host.py 209 0 100%
|
||||||
|
src/dashboards/sources/llm.py 178 1 99% 285
|
||||||
|
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||||
|
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||||
|
src/dashboards/store.py 313 2 99% 553-554
|
||||||
|
------------------------------------------------------------------
|
||||||
|
TOTAL 3894 36 99%
|
||||||
|
Required test coverage of 94% reached. Total coverage: 99.08%
|
||||||
|
1746 passed, 16 skipped, 1 warning in 165.41s (0:02:45)
|
||||||
|
== uv run ruff check src tests agent ==
|
||||||
|
All checks passed!
|
||||||
|
== uv run pyright ==
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
== node --check src/dashboards/web/static/*.js ==
|
||||||
|
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||||
|
certifi 2026.10.19 2026.10.23 wheel
|
||||||
|
coverage 7.16.2 7.17.0 wheel
|
||||||
|
fastapi 0.153.0 0.161.1 wheel
|
||||||
|
pycparser 3.11 3.12 wheel
|
||||||
|
ruff 0.17.1 0.17.2 wheel
|
||||||
@@ -44,6 +44,13 @@ export HOST_TARGETS="driveripper=https://driveripper.example.net:9120 deskwork=h
|
|||||||
# export DISKS_INTERVAL_SECONDS=60
|
# export DISKS_INTERVAL_SECONDS=60
|
||||||
# export DISKS_TIMEOUT_SECONDS=30
|
# export DISKS_TIMEOUT_SECONDS=30
|
||||||
|
|
||||||
|
# The /gateway source (phase 29) reads the one HOST_TARGETS agent that runs haproxy + nginx —
|
||||||
|
# agent 1.2.0's haproxy/nginx/sockets sections — on its own cadence. The default name already
|
||||||
|
# matches the HOST_TARGETS line above, so set these only to watch an ingress called something
|
||||||
|
# else, or to poll faster while developing the page.
|
||||||
|
# export GATEWAY_HOST=gateway
|
||||||
|
# export GATEWAY_INTERVAL_SECONDS=10
|
||||||
|
|
||||||
# The engines are dialled directly on their hostnames, like the agents.
|
# The engines are dialled directly on their hostnames, like the agents.
|
||||||
export GUFO_BASE_URL=http://deskwork.example.net:8731
|
export GUFO_BASE_URL=http://deskwork.example.net:8731
|
||||||
export VLLM_BASE_URL=http://driveripper.example.net:8000
|
export VLLM_BASE_URL=http://driveripper.example.net:8000
|
||||||
|
|||||||
@@ -73,6 +73,22 @@ DISKS_INTERVAL_SECONDS=300
|
|||||||
# fetch has to outlast HOST_TIMEOUT_SECONDS rather than inherit it.
|
# fetch has to outlast HOST_TIMEOUT_SECONDS rather than inherit it.
|
||||||
DISKS_TIMEOUT_SECONDS=30
|
DISKS_TIMEOUT_SECONDS=30
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Gateway dashboard (/gateway): one HOST_TARGETS name is the ingress
|
||||||
|
# The `gateway` source reads the SAME agent /snapshot the system page reads, projected onto
|
||||||
|
# the agent's haproxy/nginx config sections and its live /proc socket table (agent 1.2.0).
|
||||||
|
# There is no URL here: `GATEWAY_HOST` must be a name in HOST_TARGETS, and a name that is not
|
||||||
|
# there simply means "this deployment has no ingress to watch" — the source is unconfigured,
|
||||||
|
# never a red panel for a host nobody installed (same rule as /disks, which has no targets of
|
||||||
|
# its own either).
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
GATEWAY_HOST=gateway
|
||||||
|
# 30 s: a connection count is the shape of half a minute, not of ten seconds. The timeout is
|
||||||
|
# the plain agent fetch's own — these sections cost the agent two config reads and two /proc
|
||||||
|
# tables — with headroom because the same handler may be mid-smartctl pass for /disks.
|
||||||
|
GATEWAY_INTERVAL_SECONDS=30
|
||||||
|
GATEWAY_TIMEOUT_SECONDS=10
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# LLM dashboard: either engine is optional — unset means "not deployed here"
|
# LLM dashboard: either engine is optional — unset means "not deployed here"
|
||||||
# and its half of the page disappears instead of showing a dead panel.
|
# and its half of the page disappears instead of showing a dead panel.
|
||||||
|
|||||||
@@ -19,16 +19,17 @@ keep its numbers honest, and the state of the fleet it watches.
|
|||||||
|
|
||||||
## What this is
|
## What this is
|
||||||
|
|
||||||
Four dashboards for one homelab, behind one domain, in one process, backed by one SQLite
|
Five dashboards for one homelab, behind one domain, in one process, backed by one SQLite
|
||||||
file — plus an overview page that says which of the four you need:
|
file — plus an overview page that says which of the five you need:
|
||||||
|
|
||||||
| Route | Question it answers | Source |
|
| Route | Question it answers | Source |
|
||||||
| --------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
|
| ---------- | ------------------------------------------------------------------------------ | ------------------------------------------------- |
|
||||||
| `/` | Is anything wrong, and where do I look first? | all four, in one request |
|
| `/` | Is anything wrong, and where do I look first? | all five, in one request |
|
||||||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent per host |
|
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history | `agent/host_agent.py`, our own stdlib agent per host |
|
||||||
| `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) |
|
| `/disks` | Are any of my physical disks failing, and which one — health, temp, hours, wear | the same agents, `smartctl -i -A -H` per whole drive (never a partition) |
|
||||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` |
|
| `/llm` | Tokens/sec, active sessions, queue depth, KV pressure, latency percentiles | gufo `/metrics` + `/slots`, vLLM `/metrics` |
|
||||||
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
|
| `/ups` | Battery health, runtime left, power events, battery-test buttons | NUT `upsd` (our own wire-protocol client; 3051 here, not 3493) |
|
||||||
|
| `/gateway` | What is coming in through the router, and where does it go — frontends, `server_name`s, the SNI map, live pipes | the same agent (1.2.0) on the one host `GATEWAY_HOST` names: its `haproxy` + `nginx` config sections joined to its `/proc/net/tcp{,6}` sockets |
|
||||||
|
|
||||||
History is a first-class feature, not a sparkline afterthought: every number worth graphing
|
History is a first-class feature, not a sparkline afterthought: every number worth graphing
|
||||||
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.
|
is stored, rolled up (raw → 1m → 1h), and served from `/api/*/history`.
|
||||||
@@ -342,11 +343,12 @@ Responsibilities, one line each:
|
|||||||
publish: the same rule that scrubs a drive's serial (invariant 6). Those two tables are volatile by
|
publish: the same rule that scrubs a drive's serial (invariant 6). Those two tables are volatile by
|
||||||
nature, so `tests/test_agent.py` asserts their **shape**, the ports the recorded configs claim, and
|
nature, so `tests/test_agent.py` asserts their **shape**, the ports the recorded configs claim, and
|
||||||
the absence of a globally routable endpoint — never a particular row, which a re-run may lawfully
|
the absence of a globally routable endpoint — never a particular row, which a re-run may lawfully
|
||||||
change. What phase 28 stops at is the wire: `sources/host.py` passes **none** of the three into
|
change. What phase 28 stopped at was the wire: `sources/host.py` still passes **none** of the
|
||||||
`/api/system/snapshot` and `store.py` records no series for a proxy or a socket, so the sections
|
three into `/api/system/snapshot`, and `store.py` still records no series for a config — the join
|
||||||
live in the agent's own document and reach no page until phase 29's. What the dashboard does read
|
and the two gauges it publishes are phase 29's `sources/gateway.py`, which reads the same
|
||||||
today is their **error lines**, through `collectors.ADDON_ERROR_PREFIXES`, so a host with no proxy
|
`/snapshot` through its own `gateway` job. What the *system* page does read today is their
|
||||||
config stays out of `/system`'s `degraded` (invariant 3).
|
**error lines**, through `collectors.ADDON_ERROR_PREFIXES`, so a host with no proxy config stays
|
||||||
|
out of `/system`'s `degraded` (invariant 3).
|
||||||
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
|
- `config.py` — env only, no config file format, no CLI flags. `.env` is a *seed*; a real
|
||||||
env var always wins. Defaults are safe except the sources, which default to unset. The auth
|
env var always wins. Defaults are safe except the sources, which default to unset. The auth
|
||||||
block (phase 11) is `DASH_ADMIN_PASSWORD` (the **seed**, and `_env()` strips, so `VAR=` means
|
block (phase 11) is `DASH_ADMIN_PASSWORD` (the **seed**, and `_env()` strips, so `VAR=` means
|
||||||
@@ -355,19 +357,32 @@ Responsibilities, one line each:
|
|||||||
`http://127.0.0.1`; task 03 also treats `X-Forwarded-Proto: https` as secure), and
|
`http://127.0.0.1`; task 03 also treats `X-Forwarded-Proto: https` as secure), and
|
||||||
`DASH_PBKDF2_ITERATIONS` (default `auth.PBKDF2_ITERATIONS`, imported so the primitive and the
|
`DASH_PBKDF2_ITERATIONS` (default `auth.PBKDF2_ITERATIONS`, imported so the primitive and the
|
||||||
setting cannot drift) — that knob exists so the hermetic suite hashes in microseconds instead
|
setting cannot drift) — that knob exists so the hermetic suite hashes in microseconds instead
|
||||||
of 0.3 s per client and is not a production tuning parameter.
|
of 0.3 s per client and is not a production tuning parameter. The gateway block (phase 29) is
|
||||||
|
`GATEWAY_HOST` (default `gateway`, and **a name, never a URL**: `gateway_enabled` is true only
|
||||||
|
when a `HOST_TARGETS` entry has it, so the page is unconfigured rather than red for a
|
||||||
|
deployment with no ingress — the `nut_enabled` shape, and the reason `any_source_enabled`
|
||||||
|
includes it), `GATEWAY_INTERVAL_SECONDS` (30 — a connection count is the shape of half a
|
||||||
|
minute, not of ten seconds) and `GATEWAY_TIMEOUT_SECONDS` (10, the plain agent fetch with
|
||||||
|
headroom for a handler that may be mid-smartctl pass for `/disks`).
|
||||||
- `poller.py` — job loop, health records, stale/last-good semantics, transition events.
|
- `poller.py` — job loop, health records, stale/last-good semantics, transition events.
|
||||||
`/disks` is a **source of its own** (`disks:<host>`, `DISKS_INTERVAL_SECONDS` 300 s,
|
`/disks` is a **source of its own** (`disks:<host>`, `DISKS_INTERVAL_SECONDS` 300 s,
|
||||||
`DISKS_TIMEOUT_SECONDS` 30 s, events under the dashboard name `disks`) reading the same
|
`DISKS_TIMEOUT_SECONDS` 30 s, events under the dashboard name `disks`) reading the same
|
||||||
agents the `system` jobs read: the separate task is what keeps a SMART poll out of the
|
agents the `system` jobs read: the separate task is what keeps a SMART poll out of the
|
||||||
10-second utilisation loop and lets `/disks` age on its own instead of blinking `/system`.
|
10-second utilisation loop and lets `/disks` age on its own instead of blinking `/system`.
|
||||||
|
`gateway` is a **job of its own** for the same reason (phase 29 task 02) — `GATEWAY_INTERVAL_SECONDS`
|
||||||
|
30 s, `GATEWAY_TIMEOUT_SECONDS` 10 s, health and events under the dashboard name `gateway` — but
|
||||||
|
**one** job rather than one per host (the `nut` shape), dialling the `HOST_TARGETS` entry
|
||||||
|
`GATEWAY_HOST` names with its `label` set to that host's name, because the page is about the single
|
||||||
|
ingress and a record per machine would ask four hosts a question only one can answer. It exists
|
||||||
|
only when that name is a target: an unconfigured source has no task that could look red
|
||||||
|
(invariant 3), and it is handed no `RateEstimator` because a count is not a rate (rule 2).
|
||||||
The job list is built once (`app.build_hub()`) because it is wired twice — production and
|
The job list is built once (`app.build_hub()`) because it is wired twice — production and
|
||||||
the injected-store test path — and two hand-written copies drifting is how a source ends up
|
the injected-store test path — and two hand-written copies drifting is how a source ends up
|
||||||
polling in tests and not in the container. The
|
polling in tests and not in the container. The
|
||||||
success path records (payload → samples → events) and only *then* moves `last_ok`, so a
|
success path records (payload → samples → events) and only *then* moves `last_ok`, so a
|
||||||
refused SQLite write ages the source and alerts instead of leaving it current with charts
|
refused SQLite write ages the source and alerts instead of leaving it current with charts
|
||||||
that have silently stopped drawing.
|
that have silently stopped drawing.
|
||||||
Events are stored under the **dashboard** name (`system`/`disks`/`llm`/`ups`), not the sample
|
Events are stored under the **dashboard** name (`system`/`disks`/`gateway`/`llm`/`ups`), not the sample
|
||||||
source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever
|
source (`host`) — a page filtering `/api/events` by `host` gets an empty feed forever
|
||||||
(found on `/system` during phase 03 browser verification).
|
(found on `/system` during phase 03 browser verification).
|
||||||
- `collectors.py` — the fetch half. `AgentTls` (phase 15 task 03) is what makes an `https://`
|
- `collectors.py` — the fetch half. `AgentTls` (phase 15 task 03) is what makes an `https://`
|
||||||
@@ -390,8 +405,16 @@ Responsibilities, one line each:
|
|||||||
problem. `agent_tls_map()` builds one entry per configured host (the CA and client pair are
|
problem. `agent_tls_map()` builds one entry per configured host (the CA and client pair are
|
||||||
shared — the dashboard has one identity — only the revocation answer is per name, and an absent
|
shared — the dashboard has one identity — only the revocation answer is per name, and an absent
|
||||||
entry is *active*, never-issued being a different fact from revoked); `build_hub` takes that map
|
entry is *active*, never-issued being a different fact from revoked); `build_hub` takes that map
|
||||||
and hands the **same** entry to the `host:` and `disks:` jobs of one host, so a revocation
|
and hands the **same** entry to the `host:`, `disks:` and `gateway` jobs of one host, so a
|
||||||
fails both sources with one sentence and neither of them dials. `fetch_snapshot()` keeps
|
revocation fails every source reading that agent with one sentence and none of them dials
|
||||||
|
(`tests/test_api.py`'s revoked-gateway pin is the three-job version of the `host:`/`disks:` one).
|
||||||
|
`fetch_gateway_snapshot()` + `gateway_job()` (phase 29 task 02) are the third reader of one
|
||||||
|
`/snapshot`: the job is **one per deployment** (the `nut` shape, key `gateway`, `label` = the
|
||||||
|
`GATEWAY_HOST` name) on `GATEWAY_INTERVAL_SECONDS` with `GATEWAY_TIMEOUT_SECONDS`, it is handed
|
||||||
|
**no** `RateEstimator` (a connection count that goes through a Δ-window is a rate with a label and
|
||||||
|
no measurement behind it, rule 2), and `gateway_job` returns `None` when the name is not in
|
||||||
|
`HOST_TARGETS` — the `disks` configured gate, so an unconfigured source gets no task that could
|
||||||
|
look broken and `/api/health` says nothing about it. `fetch_snapshot()` keeps
|
||||||
`http://` targets on the shared client with their bearer token, unchanged from before the phase,
|
`http://` targets on the shared client with their bearer token, unchanged from before the phase,
|
||||||
so a rollout is per host.
|
so a rollout is per host.
|
||||||
`NutMonitor` owns per-UPS transition bookkeeping (its
|
`NutMonitor` owns per-UPS transition bookkeeping (its
|
||||||
@@ -447,6 +470,41 @@ Responsibilities, one line each:
|
|||||||
counts for the page header and drops a host with no payload from the sums while still
|
counts for the page header and drops a host with no payload from the sums while still
|
||||||
counting it in `hosts_total` — the `/llm` fleet-block rule, so the page never does
|
counting it in `hosts_total` — the `/llm` fleet-block rule, so the page never does
|
||||||
arithmetic and a dead agent never shrinks a fleet total.
|
arithmetic and a dead agent never shrinks a fleet total.
|
||||||
|
- `sources/gateway.py` — the same snapshot projected onto the **connection topology** of the one
|
||||||
|
host `GATEWAY_HOST` names (phase 29): `parse(snapshot, host)` joins agent 1.2.0's `haproxy`
|
||||||
|
frontends and `nginx` `listen`s into **inbound** pipes (one per listen port, carrying the
|
||||||
|
`server_name` domains and the live ESTABLISHED count + distinct client addresses) and haproxy
|
||||||
|
`server` lines, concrete `proxy_pass` targets and the `$var`→`map` entries into **outbound**
|
||||||
|
pipes, plus the `unmatched` tallies (`inbound` = connections on a listening port no config
|
||||||
|
names, `outbound` = connections to something no target claims, `other` = live listeners no
|
||||||
|
config claims — phase 28's "an unclaimed socket stays visible as one"). `count` semantics are
|
||||||
|
the whole honesty of the module: **no DNS** (the spy test pins `socket.getaddrinfo` is never
|
||||||
|
called — `ipaddress` decides "is this already an address", and it never asks), an exact
|
||||||
|
`(address, port)` match for a target written as an address (`by_port: false`), a port-only
|
||||||
|
match for a **name** and only when that port has exactly one configured host
|
||||||
|
(`by_port: true`), and **no attribution at all** when several names share a port (the real
|
||||||
|
`stream.conf` map's sixteen `*.reeselink.com:443`): those pipes carry `connections: null`, the
|
||||||
|
sockets go to `unmatched.outbound`, and `unmatched.groups` says once per port how many and
|
||||||
|
among which candidates — picking one would be rule 2, and repeating the port's total on
|
||||||
|
sixteen pipes reads as sixteen times the traffic. A pipe is `null` (never `0`) whenever the
|
||||||
|
socket table did not answer or the join could not decide, and publishes **no sample** then, so
|
||||||
|
a chart cannot draw a zero for an unknown. The classification is one exclusive pass
|
||||||
|
(`join_sockets`), which is what makes "inbound + outbound + unmatched = the socket table"
|
||||||
|
hold — the identity task 06 checks against a direct `/proc/net/tcp` read. `supported: false`
|
||||||
|
(a 1.1.0 agent, or `--no-proxy`) keeps every key with empty lists and null counts plus
|
||||||
|
`WHY_PREDATES`, `available: false`/`null` per section becomes a `why` sentence and the agent's
|
||||||
|
own line in `section_errors`, and nothing here carries a `severity` (`net.*` precedent: a busy
|
||||||
|
proxy is the job) or a `window_seconds` (a count is not a rate). Gauges: `gateway.in_conns@<port>`
|
||||||
|
and `gateway.out_conns@<host>:<port>` — the port **is** in the outbound name, because this
|
||||||
|
fleet's two haproxy Minecraft backends share `minecraft.reeselink.com` on 25565 and 25566 and
|
||||||
|
two gauges on one name would overwrite each other in silence. `parse_gateway_snapshot()` is the
|
||||||
|
`Result` the poller consumes; there are **no events** (a connection count has no transition
|
||||||
|
worth a line — a proxy that stops routing shows up as its host's own poll failing). The
|
||||||
|
never-answered arm is `stub(host, ts)` — the **same skeleton** `parse` fills in, plus
|
||||||
|
`unreachable: true`, `supported: null` and `WHY_NO_ANSWER` — because the route that serves a host
|
||||||
|
with no payload yet must not hand the page a shape with keys missing, and `WHY_NO_ANSWER` is not
|
||||||
|
`WHY_PREDATES`: an agent that has not replied has not said it has no sections (invariant 3's two
|
||||||
|
facts staying two sentences).
|
||||||
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
|
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
|
||||||
`VLLM_HISTOGRAMS`), measured Δ rates **diffed per label set** (`rate_key()` keys the
|
`VLLM_HISTOGRAMS`), measured Δ rates **diffed per label set** (`rate_key()` keys the
|
||||||
estimator on the resolved name plus its labels; the cell is the Σ of the per-series
|
estimator on the resolved name plus its labels; the cell is the Σ of the per-series
|
||||||
@@ -638,7 +696,18 @@ Responsibilities, one line each:
|
|||||||
`severity.py` rates no sector counter and this module may not have a threshold of its own, so
|
`severity.py` rates no sector counter and this module may not have a threshold of its own, so
|
||||||
grading a drive here would be the bug, not the fix (phase 09). Its `HREF` table does carry
|
grading a drive here would be the bug, not the fix (phase 09). Its `HREF` table does carry
|
||||||
`/disks`, so the one alert a disk job *can* raise — SMART failing while its host answers —
|
`/disks`, so the one alert a disk job *can* raise — SMART failing while its host answers —
|
||||||
links to the page that names the drive.
|
links to the page that names the drive, and since phase 29 `/gateway` for the same reason: a
|
||||||
|
`gateway` job failing **while its host answers** (the proxy configs or the socket table went
|
||||||
|
quiet inside a healthy agent) has no other voice. The echo suppression is now a table
|
||||||
|
(`AGENT_ECHO_DASHBOARDS` = `disks`, `gateway`) because three jobs read one agent, and it finds the
|
||||||
|
partner differently for each: `disks:<host>` names the host in its key, while the single `gateway`
|
||||||
|
job names it in its **label** (its key is one word, and `GATEWAY_HOST` is what an alert prints).
|
||||||
|
A missing `host:<name>` record suppresses nothing — "I cannot find the host's job" is not evidence
|
||||||
|
the host is fine. The gateway source has **no verdict rule and none is wanted**, the same shape as
|
||||||
|
the disk rule: its payload publishes no `severity`, `severity.THRESHOLDS` has no `gateway.*` row,
|
||||||
|
`alerts.build` takes no gateway payload at all, so a `gateway` alert can only ever be
|
||||||
|
reachability — stale past 3× its own 30 s interval or stopped answering — and 400 connections on
|
||||||
|
an ingress is the job, not a fault.
|
||||||
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning. The
|
- `store.py` — `sample(source,host,metric,res,ts)` + `event` + `meta`, rollups, pruning. The
|
||||||
`meta` table is the only durable key/value state in the app and is reached through three
|
`meta` table is the only durable key/value state in the app and is reached through three
|
||||||
named helpers — `get_meta`/`set_meta`/`delete_meta` (the last returns its rowcount, because
|
named helpers — `get_meta`/`set_meta`/`delete_meta` (the last returns its rowcount, because
|
||||||
@@ -654,7 +723,7 @@ Responsibilities, one line each:
|
|||||||
cannot remove a name, so `_note_series_write()` ends the cache on a `(source, host, metric)` the
|
cannot remove a name, so `_note_series_write()` ends the cache on a `(source, host, metric)` the
|
||||||
store has never written and leaves a poll of an existing series alone — invalidating on every
|
store has never written and leaves a poll of an existing series alone — invalidating on every
|
||||||
sample was task 02's defect, and it put the 572 ms scan back on nearly every `/api/metrics` call
|
sample was task 02's defect, and it put the 572 ms scan back on nearly every `/api/metrics` call
|
||||||
on a store nine pollers write to, behind a cache that never served; `delete_series()`/`maintain()`
|
on a store every poller writes to, behind a cache that never served; `delete_series()`/`maintain()`
|
||||||
invalidate unconditionally because a `DELETE` can take a name, `delete_series()` also forgetting
|
invalidate unconditionally because a `DELETE` can take a name, `delete_series()` also forgetting
|
||||||
the triples it emptied so a re-created series counts as new again), through a generation counter so
|
the triples it emptied so a re-created series counts as new again), through a generation counter so
|
||||||
a write racing a read cannot park a stale answer under a fresh timestamp, with the name cache
|
a write racing a read cannot park a stale answer under a fresh timestamp, with the name cache
|
||||||
@@ -756,7 +825,7 @@ Responsibilities, one line each:
|
|||||||
configured host **with drives to show** (disk rows + `smart` + `smartctl_available` +
|
configured host **with drives to show** (disk rows + `smart` + `smartctl_available` +
|
||||||
per-row `severity` + a `fleet` block summed by the API, with a configured-but-never-answered
|
per-row `severity` + a `fleet` block summed by the API, with a configured-but-never-answered
|
||||||
host as an `unreachable` stub whose counts are `None`, not `0`), and `/history` serves the
|
host as an `unreachable` stub whose counts are `None`, not `0`), and `/history` serves the
|
||||||
`disk.*@dev` series with unknown names answering as empty series like the other two
|
`disk.*@dev` series with unknown names answering as empty series like the other
|
||||||
history endpoints. The configured gate runs **before** anything that could reach the
|
history endpoints. The configured gate runs **before** anything that could reach the
|
||||||
network — `HOST_TARGETS` empty is `404 {"configured": false}` with no client, no fetch and
|
network — `HOST_TARGETS` empty is `404 {"configured": false}` with no client, no fetch and
|
||||||
no `disks` job in the hub at all (invariant 3), and `tests/test_api.py` proves it with
|
no `disks` job in the hub at all (invariant 3), and `tests/test_api.py` proves it with
|
||||||
@@ -778,11 +847,33 @@ Responsibilities, one line each:
|
|||||||
only POST that **dials an upstream** stays `/api/ups/command` (invariant 1, pinned by enumerating
|
only POST that **dials an upstream** stays `/api/ups/command` (invariant 1, pinned by enumerating
|
||||||
every served route's verbs and sorting the writes into fleet / session / certificate-authority
|
every served route's verbs and sorting the writes into fleet / session / certificate-authority
|
||||||
lists — the third is phase 15's, and it writes `meta` rows and nothing else).
|
lists — the third is phase 15's, and it writes `meta` rows and nothing else).
|
||||||
All four `history` routes reach the store through one door, `api._history_batch()` →
|
`/api/gateway/*` (phase 29 task 03) is the ingress pair, and it is the same two rules on a
|
||||||
|
single-host source. `/snapshot` answers `sources/gateway.parse`'s payload for the one
|
||||||
|
`GATEWAY_HOST` entry — the haproxy frontends and the nginx `listen`/`map` echo, the inbound pipes
|
||||||
|
(port, domains, live count, up to `MAX_CLIENTS` client addresses), the outbound pipes (`by_port`
|
||||||
|
naming which half of the match is a socket fact), each pipe's own `metric`, the `unmatched`
|
||||||
|
tallies, and `supported`/`sections`/`why` — with `states` appended and the payload's own `ts`
|
||||||
|
left alone, because that is when the agent took the reading and it is what the page ages. Three
|
||||||
|
arms reach it and none of them is red: `404 {"configured": false}` **in the route's first line**
|
||||||
|
when `GATEWAY_HOST` names no `HOST_TARGETS` entry (unset and wrong-name are both spy-pinned to
|
||||||
|
`httpx.AsyncClient.get` and `collectors.fetch_snapshot`, and neither arm has a `gateway` job to
|
||||||
|
age out either), `supported: false` for a 1.1.0/`--no-proxy agent, and `gateway_source.stub()`
|
||||||
|
for a configured host that has never answered — the payload's key set plus `unreachable: true`
|
||||||
|
and minus only the `agent_version` nobody published, every count `null` because an agent that has
|
||||||
|
not spoken has not reported an idle gateway (rule 2). Nothing in the payload carries a `severity`
|
||||||
|
or a `window_seconds`, at any depth (pinned by a recursive walk: a busy proxy is the job and no
|
||||||
|
Δ was ever measured). `/history` serves the `gateway.in_conns@<port>` /
|
||||||
|
`gateway.out_conns@<host>:<port>` gauges with `hosts=` defaulting to the ingress **alone** (the
|
||||||
|
name the samples are stored under — defaulting to every target would draw empty plots for
|
||||||
|
machines nobody asked to route), and it carries no configured gate because it dials nothing:
|
||||||
|
like `/api/disks/history`, an unconfigured source answers no hosts and no series. The pair adds
|
||||||
|
no write (the verb enumeration pins the two GETs and that the app's one upstream POST is still
|
||||||
|
`/api/ups/command`), and `tests/test_auth_api.py` lists both paths as session-required.
|
||||||
|
All five `history` routes reach the store through one door, `api._history_batch()` →
|
||||||
`store.history_many()` (phase 14 task 02). That is a cost change and nothing else: the series
|
`store.history_many()` (phase 14 task 02). That is a cost change and nothing else: the series
|
||||||
list is host-major and metric-minor in request order, each entry the same
|
list is host-major and metric-minor in request order, each entry the same
|
||||||
`{source, host, metric, range_seconds, points}` document the per-series loop built, with the
|
`{source, host, metric, range_seconds, points}` document the per-series loop built, with the
|
||||||
same empty-series answer for a name that has no rows. `tests/test_api.py` pins it for all four
|
same empty-series answer for a name that has no rows. `tests/test_api.py` pins it for all five
|
||||||
routes against points read straight out of the store with `query()` — a third path, in neither
|
routes against points read straight out of the store with `query()` — a third path, in neither
|
||||||
the loop nor the batch — so a future "optimisation" that moves a point, an order, or a key
|
the loop nor the batch — so a future "optimisation" that moves a point, an order, or a key
|
||||||
fails a test instead of redrawing a chart wrong.
|
fails a test instead of redrawing a chart wrong.
|
||||||
@@ -804,6 +895,25 @@ Responsibilities, one line each:
|
|||||||
job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in
|
job with no payload yet is listed as a stub (`unreachable: true`, every value `null`) in
|
||||||
`system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total`
|
`system.hosts`/`llm.engines` exactly as `/api/system/snapshot` does, so `hosts_total`
|
||||||
counts the machines there are and `hosts_live` the ones answering.
|
counts the machines there are and `hosts_live` the ones answering.
|
||||||
|
The **`gateway` block** (phase 29 task 02) is additive in the same way: `{configured, host, href,
|
||||||
|
supported, unreachable, inbound, outbound, map_targets, unmatched, unmatched_listeners, partial,
|
||||||
|
states}`. Task 02 wrote it for programmatic consumers (the page itself reads
|
||||||
|
`/api/gateway/snapshot`, task 03); task 05 gave it one page-side reader, the single line `/`
|
||||||
|
renders under the fleet table, so `href` joined the block — derived from `alerts.HREF`, the one
|
||||||
|
table that writes a dashboard's route, so the overview's link and an alert about the same source
|
||||||
|
cannot spell the page differently (`test_the_gateway_block_links_to_the_page_the_alert_row_links_to`,
|
||||||
|
and the page half reads only block keys: `test_the_overview_line_reads_the_gateway_block_it_claims`
|
||||||
|
resolves the served `GATEWAY_FIELDS` against a live document). `inbound`/`outbound` are the sums of the pipes' own counts and `map_targets` counts the stream
|
||||||
|
map's routing entries; every figure is `null` — never `0` — in the arms that did not measure
|
||||||
|
(unconfigured, configured-but-never-answered, a 1.1.0/`--no-proxy agent, a silent socket table),
|
||||||
|
and `partial: true` says a total had to leave an unknown pipe out (a port whose configured targets
|
||||||
|
cannot be told apart) rather than pretend to be the whole number. `unmatched` counts
|
||||||
|
*connections* and `unmatched_listeners` counts *ports*, so they are two fields and never one sum,
|
||||||
|
and nothing in the block carries a `severity`. `/api/config` grew the matching row (`enabled`,
|
||||||
|
`host`, `url`, `interval_seconds`, `timeout_seconds`), where `url` is the named entry's
|
||||||
|
`Target.display_url` and `null` when no target carries the name — invariant 6's boundary applies
|
||||||
|
to a name-only source exactly as it does to a URL-bearing one, because this body is what a browser
|
||||||
|
renders.
|
||||||
- `app.py` — assembly, and the **session gate** (`session_gate` middleware, invariant 12) whose
|
- `app.py` — assembly, and the **session gate** (`session_gate` middleware, invariant 12) whose
|
||||||
`PUBLIC_PATHS`/`PUBLIC_PREFIXES` are the whole anonymous surface. It answers in this order and
|
`PUBLIC_PATHS`/`PUBLIC_PREFIXES` are the whole anonymous surface. It answers in this order and
|
||||||
the order is the design: no `AuthState` → `503` for everything; no session and no such route →
|
the order is the design: no `AuthState` → `503` for everything; no session and no such route →
|
||||||
@@ -829,14 +939,18 @@ Responsibilities, one line each:
|
|||||||
directory the lifespan's `finally` removes. What hands that material to the loops is
|
directory the lifespan's `finally` removes. What hands that material to the loops is
|
||||||
`agent_tls_map(material, store, targets)` → `build_hub(..., tls=...)`, called in **both**
|
`agent_tls_map(material, store, targets)` → `build_hub(..., tls=...)`, called in **both**
|
||||||
branches the same way: no material (or no targets) means no TLS map, which is the plain
|
branches the same way: no material (or no targets) means no TLS map, which is the plain
|
||||||
`http://`/token path rather than an error, and one map covers both of a host's jobs. A corrupt stored certificate raises out of the same
|
`http://`/token path rather than an error, and one map covers all of a host's jobs — including the
|
||||||
|
`gateway` job, which is why one revoke in `/certs` silences the pipes with no restart. A corrupt stored certificate raises out of the same
|
||||||
place — a dashboard that cannot present its own client certificate would otherwise spend the
|
place — a dashboard that cannot present its own client certificate would otherwise spend the
|
||||||
fleet's TLS history reporting an upstream problem it caused itself.
|
fleet's TLS history reporting an upstream problem it caused itself.
|
||||||
`build_hub` also builds **two** `RateEstimator`s, because a window is only a claim about the
|
`build_hub` also builds **two** `RateEstimator`s, because a window is only a claim about the
|
||||||
polls that fill it (phase 23, `TODO.md:8`'s "ramp up/ramp down"): the `host` jobs get
|
polls that fill it (phase 23, `TODO.md:8`'s "ramp up/ramp down"): the `host` jobs get
|
||||||
`RateEstimator(span=settings.rate_window)` (10 s — what a 10 s cadence can fill) and the `llm`
|
`RateEstimator(span=settings.rate_window)` (10 s — what a 10 s cadence can fill) and the `llm`
|
||||||
jobs `RateEstimator(span=settings.llm_interval)` (2 s, one poll interval, so a slot that starts
|
jobs `RateEstimator(span=settings.llm_interval)` (2 s, one poll interval, so a slot that starts
|
||||||
reads steady on the next poll instead of climbing one fifth of the true rate per poll), and
|
reads steady on the next poll instead of climbing one fifth of the true rate per poll), and the
|
||||||
|
`gateway` job (phase 29) is handed **neither** — it is added to the hub between the `disks` and
|
||||||
|
`llm` jobs, only when `settings.gateway_enabled`, because a gauge needs no window and one job
|
||||||
|
cannot be right about two cadences' worth of them.,
|
||||||
there is no third knob — the LLM window **is** `LLM_INTERVAL_SECONDS`, which is the only way
|
there is no third knob — the LLM window **is** `LLM_INTERVAL_SECONDS`, which is the only way
|
||||||
the pair cannot drift. One estimator for both could only be right about one of them: a 2 s span
|
the pair cannot drift. One estimator for both could only be right about one of them: a 2 s span
|
||||||
over a 10 s poll sits the window-empty fallback against the `2 × span` ceiling and answers
|
over a 10 s poll sits the window-empty fallback against the `2 × span` ceiling and answers
|
||||||
@@ -844,9 +958,14 @@ Responsibilities, one line each:
|
|||||||
The estimator's own gates (the `MIN_WINDOW_SECONDS` floor, the ceiling, the 8× jump
|
The estimator's own gates (the `MIN_WINDOW_SECONDS` floor, the ceiling, the 8× jump
|
||||||
confirmation with its independent anchor) are unchanged and apply to the smaller window, whose
|
confirmation with its independent anchor) are unchanged and apply to the smaller window, whose
|
||||||
`Δ2.0s` label the payload carries with the value.
|
`Δ2.0s` label the payload carries with the value.
|
||||||
- `web/` — five dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`) plus the login page
|
- `web/` — six dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`, `/gateway`) plus the
|
||||||
(`/login`, `pages/login.html` + `static/page-login.js`), one stylesheet, `charts.js`. No
|
login page (`/login`, `pages/login.html` + `static/page-login.js`), one stylesheet, `charts.js`.
|
||||||
framework. `/login` renders the masthead's **brand link and theme toggle** and nothing else
|
No framework. The tab nav is those six and nothing else, in that order, on **all eight**
|
||||||
|
signed-in pages (phase 29 task 05): `tests/test_api.py::test_the_disks_tab_is_in_the_masthead_of_every_page`
|
||||||
|
compares the nav's hrefs to `TAB_ORDER` (which **is** `PAGES`) and asserts the `aria-current`
|
||||||
|
count per page — one, naming itself, on the six dashboards and zero on `/settings` and `/certs`.
|
||||||
|
`/gateway`'s own tab arrived with that task; until then the page existed behind a route with no
|
||||||
|
way to click to it. `/login` renders the masthead's **brand link and theme toggle** and nothing else
|
||||||
from it: no tabs (every tab bounces straight back here, and a nav of links that all redirect is
|
from it: no tabs (every tab bounces straight back here, and a nav of links that all redirect is
|
||||||
a lie about what is available), no status dots and no clock (there is nothing signed in for them
|
a lie about what is available), no status dots and no clock (there is nothing signed in for them
|
||||||
to say). It posts through `Dash.fetchJSON` like every other script, which is only safe because
|
to say). It posts through `Dash.fetchJSON` like every other script, which is only safe because
|
||||||
@@ -864,8 +983,8 @@ Responsibilities, one line each:
|
|||||||
login page is audited for orphan `id`s and CDN references too (`tests/test_auth_api.py`), fetched
|
login page is audited for orphan `id`s and CDN references too (`tests/test_auth_api.py`), fetched
|
||||||
unsigned because a signed-in `GET /login` is a `303` and would audit `/`'s markup under the wrong
|
unsigned because a signed-in `GET /login` is a `303` and would audit `/`'s markup under the wrong
|
||||||
name. `/settings` (`pages/settings.html` + `static/page-settings.js`) is a **page and not a
|
name. `/settings` (`pages/settings.html` + `static/page-settings.js`) is a **page and not a
|
||||||
sixth dashboard**: same `<head>` and masthead, the five tabs with **no** `aria-current` (it is
|
seventh dashboard**: same `<head>` and masthead, the six tabs with **no** `aria-current` (it is
|
||||||
not one of them), the same four `#nav-status` badges as every dashboard (phase 25 task 01 —
|
not one of them), the same five `#nav-status` badges as every dashboard (phase 25 task 01 —
|
||||||
`TODO.md:3`'s "hosts disks engines UPS disappear when in the certs/account page" was the
|
`TODO.md:3`'s "hosts disks engines UPS disappear when in the certs/account page" was the
|
||||||
missing container, and `Dash.initChrome()` fills it from `/api/health` with no JS change), and
|
missing container, and `Dash.initChrome()` fills it from `/api/health` with no JS change), and
|
||||||
it is reached from the `⚙ Account` link in
|
it is reached from the `⚙ Account` link in
|
||||||
@@ -884,19 +1003,19 @@ Responsibilities, one line each:
|
|||||||
`csrf` is now invalid — every later write, the UPS buttons included, 403s until a reload) and on
|
`csrf` is now invalid — every later write, the UPS buttons included, 403s until a reload) and on
|
||||||
any `401` bounce.
|
any `401` bounce.
|
||||||
`/certs` (`pages/certs.html` + `static/page-certs.js`) is the second of those admin pages, and it
|
`/certs` (`pages/certs.html` + `static/page-certs.js`) is the second of those admin pages, and it
|
||||||
is the same shape for the same reasons: five tabs with **no** `aria-current`, the four fleet
|
is the same shape for the same reasons: six tabs with **no** `aria-current`, the five fleet
|
||||||
badges in `#nav-status` like every other masthead (phase 25 task 01, same reasoning as
|
badges in `#nav-status` like every other masthead (phase 25 task 01, same reasoning as
|
||||||
`/settings` above), and both masthead links — `⚷ Certs` beside `⚙ Account` — exactly as the five
|
`/settings` above), and both masthead links — `⚷ Certs` beside `⚙ Account` — exactly as the six
|
||||||
dashboards render them. **Both links are on both admin pages** (phase 20 task 04, `TODO.md:4`),
|
dashboards render them. **Both links are on both admin pages** (phase 20 task 04, `TODO.md:4`),
|
||||||
which retires phase 15's "no self-link" precedent: chrome that changes shape depending on where you already are reads
|
which retires phase 15's "no self-link" precedent: chrome that changes shape depending on where you already are reads
|
||||||
as a broken button, while a link to the page you are on is a no-op that answers with the page you
|
as a broken button, while a link to the page you are on is a no-op that answers with the page you
|
||||||
were looking at (`data/logs/20-task04-admin-links.cjs` measures exactly that — one document
|
were looking at (`data/logs/20-task04-admin-links.cjs` measures exactly that — one document
|
||||||
response, no redirect, the same cluster afterwards). The two links therefore sit on all seven
|
response, no redirect, the same cluster afterwards). The two links therefore sit on all eight
|
||||||
signed-in pages (the five dashboards plus `/settings` and `/certs`) in one order
|
signed-in pages (the six dashboards plus `/settings` and `/certs`) in one order
|
||||||
(`⚷ Certs` → `⚙ Account` → theme toggle → clock — phase 25 task 03 moved the clock to the end so
|
(`⚷ Certs` → `⚙ Account` → theme toggle → clock — phase 25 task 03 moved the clock to the end so
|
||||||
it is the farthest-right item on a `flex-end` row), which is what lets
|
it is the farthest-right item on a `flex-end` row), which is what lets
|
||||||
the pinned audit be one loop over `MARKUP_PAGES` instead of two over different tuples; `/login`
|
the pinned audit be one loop over `MARKUP_PAGES` instead of two over different tuples; `/login`
|
||||||
is the eighth page and the one with neither. Everything on it is
|
is the ninth page and the one with neither. Everything on it is
|
||||||
`GET /api/certs`: the Root CA block, the display-only Dashboard client line (that certificate is
|
`GET /api/certs`: the Root CA block, the display-only Dashboard client line (that certificate is
|
||||||
minted with the root and has no route of its own), and one row per host with its status chip,
|
minted with the root and has no route of its own), and one row per host with its status chip,
|
||||||
serial, expiry (local timestamp *and* `Dash.duration` left), issued-vs-deployed `bundle_rev`,
|
serial, expiry (local timestamp *and* `Dash.duration` left), issued-vs-deployed `bundle_rev`,
|
||||||
@@ -926,7 +1045,12 @@ Responsibilities, one line each:
|
|||||||
`data/logs/27-task01-scroll.cjs`).
|
`data/logs/27-task01-scroll.cjs`).
|
||||||
`app.js`'s masthead status dots are one per dashboard job —
|
`app.js`'s masthead status dots are one per dashboard job —
|
||||||
including `disks`, because the same agent can answer the 10-second utilisation poll and
|
including `disks`, because the same agent can answer the 10-second utilisation poll and
|
||||||
have a wedged SMART pass, and one dot for both would be right about neither. `app.js` owns the
|
have a wedged SMART pass, and one dot for both would be right about neither, and including
|
||||||
|
`gateway` (phase 29 task 05), which is one dot rather than one per host because the ingress is
|
||||||
|
one job and a dot for a source that stays `idle` when `GATEWAY_HOST` names nothing is how an
|
||||||
|
unconfigured source reads grey rather than red. The badge table is a literal in `initChrome()`,
|
||||||
|
so a new dashboard's job means a new entry there — `NAV_STATUS_BADGES` in `tests/test_api.py`
|
||||||
|
is the pin that says so. `app.js` owns the
|
||||||
colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an
|
colour vocabulary: `Dash.severityFrom(cellOrSeverity)` turns a payload verdict into an
|
||||||
`ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and
|
`ok|warn|fail` class (`''` for anything unrated, so unknown never looks green) and
|
||||||
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
|
`Dash.hero(el, {label, value, unit, digits, severity, sub, bar})` renders/patches a
|
||||||
@@ -941,7 +1065,7 @@ Responsibilities, one line each:
|
|||||||
**The masthead is three rows at every width, the two control rows flush right** (phase 20 task
|
**The masthead is three rows at every width, the two control rows flush right** (phase 20 task
|
||||||
01 made the rows, `TODO.md:5-6`; phase 25 task 02 re-justified them, `TODO.md:4` — "I made a
|
01 made the rows, `TODO.md:5-6`; phase 25 task 02 re-justified them, `TODO.md:4` — "I made a
|
||||||
mistake earlier, I did want hosts disks engines UPS badges and certs/account/dark right
|
mistake earlier, I did want hosts disks engines UPS badges and certs/account/dark right
|
||||||
justified"): row 1 the brand, `#nav-toggle` and the five tabs; row 2 the `#nav-status`
|
justified"): row 1 the brand, `#nav-toggle` and the six tabs; row 2 the `#nav-status`
|
||||||
per-source dots; row 3
|
per-source dots; row 3
|
||||||
the `.masthead-right` controls (`⚷ Certs`, `⚙ Account`, theme, then the clock — phase 25 task 03
|
the `.masthead-right` controls (`⚷ Certs`, `⚙ Account`, theme, then the clock — phase 25 task 03
|
||||||
moved the clock last for `TODO.md:5`'s "the time should show up as the farthest right item";
|
moved the clock last for `TODO.md:5`'s "the time should show up as the farthest right item";
|
||||||
@@ -960,11 +1084,20 @@ Responsibilities, one line each:
|
|||||||
brand and the tabs on separate rows. `height: var(--header-h)` became **`min-height`** (three rows
|
brand and the tabs on separate rows. `height: var(--header-h)` became **`min-height`** (three rows
|
||||||
do not fit in 58 px and a fixed height clips the controls) and is declared **once** — the ≤640px
|
do not fit in 58 px and a fixed height clips the controls) and is declared **once** — the ≤640px
|
||||||
block no longer restates `height`/`min-height`/`flex-wrap`, and that pair drifting is exactly what
|
block no longer restates `height`/`min-height`/`flex-wrap`, and that pair drifting is exactly what
|
||||||
would clip the bar on a phone only. Every page that renders the bar now renders three rows —
|
would clip the bar on a phone only — except that **`#nav-status` wraps at ≤640px since phase 29
|
||||||
|
task 05**: five badges are 352 px of chips against this bar's 320 px content box, and a `nowrap`
|
||||||
|
item that may not shrink does not get smaller, it widens the *document* (measured 372 px — the
|
||||||
|
sideways scroll phase 18 forbids). The fifth chip takes a second line of the same right-aligned
|
||||||
|
row and the bar grows by one line box; no label shrinks and nothing is hidden. Phase 18's
|
||||||
|
`nowrap` was right for four (267 px) and the pin now demands the opposite, with
|
||||||
|
`data/logs/29-task05-tabs.cjs` measuring 360 px documents on all eight pages.
|
||||||
|
Every page that renders the bar now renders three rows —
|
||||||
`/certs` and `/settings` grew the dots row in phase 25 task 01, where `#nav-status` is pinned as
|
`/certs` and `/settings` grew the dots row in phase 25 task 01, where `#nav-status` is pinned as
|
||||||
the **first child** of `.masthead-right` on all seven signed-in pages (`app.js` is untouched:
|
the **first child** of `.masthead-right` on all eight signed-in pages (`app.js`'s *badge list* is
|
||||||
the container's existence is what makes `initChrome` build the four badges, which is why the fix
|
a literal, and phase 25's fix was markup because the container was what was missing; phase 29
|
||||||
was markup and not JS) — and `/login` remains the one page with no dots, rendering two rows
|
task 05 added the fifth entry — `{ key: 'gateway', label: 'gateway', href: '/gateway' }` — because
|
||||||
|
the ingress has a job of its own and a job with no badge is a source that blinks in silence) —
|
||||||
|
and `/login` remains the one page with no dots, rendering two rows
|
||||||
(brand, then the theme toggle). Pinned on the served bytes by phase 20's three tests in
|
(brand, then the theme toggle). Pinned on the served bytes by phase 20's three tests in
|
||||||
`tests/test_api.py` — the row declarations (task 02 renamed that one to
|
`tests/test_api.py` — the row declarations (task 02 renamed that one to
|
||||||
`test_the_masthead_is_three_rows_right_justified_at_every_width`), the mobile block's silence
|
`test_the_masthead_is_three_rows_right_justified_at_every_width`), the mobile block's silence
|
||||||
@@ -980,12 +1113,14 @@ Responsibilities, one line each:
|
|||||||
(the row-break assertions are the same ones phase 20 wrote, deliberately: the re-justify must
|
(the row-break assertions are the same ones phase 20 wrote, deliberately: the re-justify must
|
||||||
not cost the rows), and the measured half is `data/logs/25-task02-justify.cjs` — seven pages ×
|
not cost the rows), and the measured half is `data/logs/25-task02-justify.cjs` — seven pages ×
|
||||||
two widths, each demanding both rows compute `flex-end` and sit flush with the bar's right
|
two widths, each demanding both rows compute `flex-end` and sit flush with the bar's right
|
||||||
padding edge, the four badges present, three rows intact, the dropdown still under the whole
|
padding edge, the four badges present (a four-badge claim — `29-task05-tabs.cjs` is the pin that
|
||||||
|
the fifth, `gateway`, joined them), three rows intact, the dropdown still under the whole
|
||||||
bar and no sideways scroll, plus `/login`'s two-row bar with no badges
|
bar and no sideways scroll, plus `/login`'s two-row bar with no badges
|
||||||
(`data/logs/25-task02-justify.out`, screenshots `data/logs/25-task02-*.png`). **Phase 25 task 03
|
(`data/logs/25-task02-justify.out`, screenshots `data/logs/25-task02-*.png`). **Phase 25 task 03
|
||||||
then put the clock at the end of the control row** (`TODO.md:5` — "The time should show up as the
|
then put the clock at the end of the control row** (`TODO.md:5` — "The time should show up as the
|
||||||
farthest right item on the navbar"): the `<span class="clock" id="clock">` moved from the first
|
farthest right item on the navbar"): the `<span class="clock" id="clock">` moved from the first
|
||||||
control to the last one on all seven templates, so the row reads `⚷ Certs` → `⚙ Account` → theme
|
control to the last one on all seven templates then (eight since `/gateway` joined the nav), so
|
||||||
|
the row reads `⚷ Certs` → `⚙ Account` → theme
|
||||||
toggle → clock. It is a **markup** move and deliberately not a CSS `order:` declaration, because
|
toggle → clock. It is a **markup** move and deliberately not a CSS `order:` declaration, because
|
||||||
on a `flex-end` row the last child *is* the farthest-right item and an `order:` override would be
|
on a `flex-end` row the last child *is* the farthest-right item and an `order:` override would be
|
||||||
something the served-byte pin cannot see — `test_every_page_links_to_both_admin_pages` therefore
|
something the served-byte pin cannot see — `test_every_page_links_to_both_admin_pages` therefore
|
||||||
@@ -1015,7 +1150,8 @@ Responsibilities, one line each:
|
|||||||
off it after a scroll; screenshots `data/logs/masthead-unsticky*.png`). Note that a browser cannot
|
off it after a scroll; screenshots `data/logs/masthead-unsticky*.png`). Note that a browser cannot
|
||||||
tell `relative` from `relative; top: 0` — Chrome computes `top: 0px` for either — so the absence
|
tell `relative` from `relative; top: 0` — Chrome computes `top: 0px` for either — so the absence
|
||||||
of the offset is a served-CSS pin, not a measurement.
|
of the offset is a served-CSS pin, not a measurement.
|
||||||
**The tab nav collapses into a hamburger at ≤640px** (phase 18): the seven tabbed pages each
|
**The tab nav collapses into a hamburger at ≤640px** (phase 18): the eight tabbed pages (the six
|
||||||
|
dashboards plus `/settings` and `/certs`; `/gateway` joined in phase 29 task 05) each
|
||||||
carry `#nav-toggle` (`class="ghost nav-toggle"`, `type="button"`, `aria-controls="nav-tabs"`,
|
carry `#nav-toggle` (`class="ghost nav-toggle"`, `type="button"`, `aria-controls="nav-tabs"`,
|
||||||
`aria-label="Toggle navigation"`, `☰`) between `.brand` and `<nav class="tabs" id="nav-tabs">`,
|
`aria-label="Toggle navigation"`, `☰`) between `.brand` and `<nav class="tabs" id="nav-tabs">`,
|
||||||
and `/login` carries neither — it renders the brand link and the theme toggle only. Its base
|
and `/login` carries neither — it renders the brand link and the theme toggle only. Its base
|
||||||
@@ -1059,8 +1195,9 @@ Responsibilities, one line each:
|
|||||||
The `if (navToggle && appNav)` guard is the whole `/login` story: the shared script loads there
|
The `if (navToggle && appNav)` guard is the whole `/login` story: the shared script loads there
|
||||||
and does nothing. The `640px` literal lives in two files — the stylesheet block and that media
|
and does nothing. The `640px` literal lives in two files — the stylesheet block and that media
|
||||||
query — and has to move in both, or the menu opens into a hidden nav. The contract is
|
query — and has to move in both, or the menu opens into a hidden nav. The contract is
|
||||||
pinned on the **served** bytes by three `tests/test_api.py` pins — the button on all seven pages
|
pinned on the **served** bytes by three `tests/test_api.py` pins — the button on every page in
|
||||||
and nowhere on `/login`, and one test that **compares** the CSS media query with the JS
|
`MARKUP_PAGES` (eight since `/gateway` joined the nav) and nowhere on `/login`, and one test that
|
||||||
|
**compares** the CSS media query with the JS
|
||||||
`matchMedia` instead of grepping for `640px` twice, so a drift fails with both numbers in the
|
`matchMedia` instead of grepping for `640px` twice, so a drift fails with both numbers in the
|
||||||
message — and proven in a browser at 360px by `data/logs/nav-task04-verify.cjs` (the `/` +
|
message — and proven in a browser at 360px by `data/logs/nav-task04-verify.cjs` (the `/` +
|
||||||
`/system` interaction, 24 checks) and `data/logs/nav-final-pass.cjs` (the phase-completion
|
`/system` interaction, 24 checks) and `data/logs/nav-final-pass.cjs` (the phase-completion
|
||||||
@@ -1081,9 +1218,12 @@ Responsibilities, one line each:
|
|||||||
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
|
Metric naming: dotted (`cpu.usage_pct`, `mem.used_pct`, `disk.used_pct@/srv`,
|
||||||
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
|
`gpu.busy_pct@0`, `llm.decode_tokens_per_second`, `ups.charge_pct`), `@stream` for per-device
|
||||||
series — `/api/metrics` answers with exactly these names. The `sample.source` column is the
|
series — `/api/metrics` answers with exactly these names. The `sample.source` column is the
|
||||||
dashboard family — `host`, `llm`, `ups`, and `disks` for the SMART series
|
dashboard family — `host`, `llm`, `ups`, `disks` for the SMART series
|
||||||
(`disk.temp_c@nvme0n1`, `disk.power_on_hours@…`, `disk.reallocated@…`, `disk.pending@…`,
|
(`disk.temp_c@nvme0n1`, `disk.power_on_hours@…`, `disk.reallocated@…`, `disk.pending@…`,
|
||||||
`disk.uncorrectable@…`, `disk.percentage_used@…`). Those keep their own family instead of
|
`disk.uncorrectable@…`, `disk.percentage_used@…`), and `gateway` for the phase-29 connection
|
||||||
|
gauges (`gateway.in_conns@<listen-port>`, `gateway.out_conns@<host>:<port>` — the port belongs
|
||||||
|
in the outbound name because two services on one host are two facts and one name would make the
|
||||||
|
second insert overwrite the first). Those keep their own family instead of
|
||||||
sitting beside `host`'s `disk.used_pct@/mount` because the cadences differ thirty-fold and
|
sitting beside `host`'s `disk.used_pct@/mount` because the cadences differ thirty-fold and
|
||||||
`GET /api/metrics?source=disks` has to answer "what does *this page* have history for".
|
`GET /api/metrics?source=disks` has to answer "what does *this page* have history for".
|
||||||
That scheme is the history API's key — treat it as a public interface.
|
That scheme is the history API's key — treat it as a public interface.
|
||||||
@@ -1134,9 +1274,11 @@ That scheme is the history API's key — treat it as a public interface.
|
|||||||
`sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force`
|
`sudo deploy/install-agent.sh --bind 10.1.0.126 --port 9120 --as-root --caps CAP_SYS_RAWIO,CAP_SYS_ADMIN --force`
|
||||||
reproduces it and a re-run that would quietly replace a hand-fixed unit is now a refusal with a
|
reproduces it and a re-run that would quietly replace a hand-fixed unit is now a refusal with a
|
||||||
diff attached rather than a clobber (gap 10).
|
diff attached rather than a clobber (gap 10).
|
||||||
`/api/health` today reads **all nine sources `ok`**: `host:driveripper`, `host:deskwork`,
|
`/api/health` today reads **all ten sources `ok`**: `host:driveripper`, `host:deskwork`,
|
||||||
`host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`,
|
`host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `gateway`, `nut`,
|
||||||
`llm:vllm`.
|
`llm:gufo`, `llm:vllm`. The tenth is phase 29's: one job for the ingress host, dialling the same
|
||||||
|
`https://` agent as `host:gateway` with the same certificate entry, and re-verified live on the
|
||||||
|
dev server with no change to any pre-existing source.
|
||||||
**The fleet is mTLS since 2026-10-21 (phase 24 task 03), so the bearer-token transport is now
|
**The fleet is mTLS since 2026-10-21 (phase 24 task 03), so the bearer-token transport is now
|
||||||
history rather than the designed shape.** `gateway` runs `agent/host_agent.py` **1.2.0** (phase 28
|
history rather than the designed shape.** `gateway` runs `agent/host_agent.py` **1.2.0** (phase 28
|
||||||
task 05, 2026-10-22) and `deskwork`/`driveripper` still run **1.1.0**; the spread costs nothing,
|
task 05, 2026-10-22) and `deskwork`/`driveripper` still run **1.1.0**; the spread costs nothing,
|
||||||
@@ -1376,9 +1518,9 @@ uv run python scripts/dev-certs-export.py --url http://127.0.0.1:8090 \
|
|||||||
# GET /api/certs/<name>/bundle and writes
|
# GET /api/certs/<name>/bundle and writes
|
||||||
# data/certs/<name>.bundle at 0600 — mints nothing
|
# data/certs/<name>.bundle at 0600 — mints nothing
|
||||||
|
|
||||||
uv run pytest # hermetic: 1650 passed + 15 skipped (11 live,
|
uv run pytest # hermetic: 1746 passed + 16 skipped (12 live,
|
||||||
# 4 store_perf), no network
|
# 4 store_perf), no network
|
||||||
uv run pytest --cov # coverage gate (98.97%; floor 94%)
|
uv run pytest --cov # coverage gate (99.08%; floor 94%)
|
||||||
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
|
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
|
||||||
uv run pytest -m store_perf --store-perf # opt-in: the store hot-path timings over the seeded
|
uv run pytest -m store_perf --store-perf # opt-in: the store hot-path timings over the seeded
|
||||||
# store, priced against `store_perf.BUDGETS` (the cached
|
# store, priced against `store_perf.BUDGETS` (the cached
|
||||||
@@ -1497,6 +1639,14 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
|
|||||||
`test_the_real_drives_answer_every_number_the_disks_page_prints` asserts every number
|
`test_the_real_drives_answer_every_number_the_disks_page_prints` asserts every number
|
||||||
`/disks` prints comes out of real drive output — that test is what caught the parser
|
`/disks` prints comes out of real drive output — that test is what caught the parser
|
||||||
reading the end of an attribute row instead of its annotated RAW_VALUE column.
|
reading the end of an attribute row instead of its annotated RAW_VALUE column.
|
||||||
|
- **A join is pinned by the identity it must satisfy and by the one question it may never ask**
|
||||||
|
(phase 29). `tests/test_gateway_source.py` runs the captured gateway configs and the recorded
|
||||||
|
`/proc/net/tcp{,6}` tables through the **real** `host_agent` parsers, so the join is compared with
|
||||||
|
the agent's document and not with a copy of its field names, and then demands the arithmetic:
|
||||||
|
inbound + outbound + unmatched equals the socket table's own ESTABLISHED count, a port no config
|
||||||
|
claims stays visible in `unmatched.other`, several names sharing one port answer `null` rather than
|
||||||
|
picking a favourite, and `socket.getaddrinfo` is spied so the day the join resolves a name the run
|
||||||
|
fails — deciding "already an address, or a name?" is all that module may ask (rule 2).
|
||||||
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
|
- Anything that needs a failure mode — upsd refusing a command, an engine 500ing, a host
|
||||||
rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
|
rebooting mid-poll, a counter reset — is a fake in `tests/conftest.py`.
|
||||||
- **A verdict the operator sees is pinned on the surface that shows it, in both directions.**
|
- **A verdict the operator sees is pinned on the surface that shows it, in both directions.**
|
||||||
@@ -1676,7 +1826,7 @@ branch `main`); there is no CI on it, so `bash .agents/validate.sh` is the only
|
|||||||
existing series issues no `SELECT`, a write of a name nobody wrote ends the cache — and the live
|
existing series issues no `SELECT`, a write of a name nobody wrote ends the cache — and the live
|
||||||
version is `scripts/watch-store-perf.py`, which is where the defect showed up.
|
version is `scripts/watch-store-perf.py`, which is where the defect showed up.
|
||||||
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
|
- Coverage gate: 80% is the repo floor, the phase gate in `.agents/validate.sh` enforces
|
||||||
94% (`--cov-fail-under`), and ~98% is where we are; keep it there. New/modified Python
|
94% (`--cov-fail-under`), and ~99% is where we are; keep it there. New/modified Python
|
||||||
still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
|
still has to clear >90% on its own. API tests use `create_app()` with an in-memory DB,
|
||||||
never a real target, and never a real upstream: `/api/ups/command`'s send is proven against
|
never a real target, and never a real upstream: `/api/ups/command`'s send is proven against
|
||||||
a scripted `nut.NutClient` (`ScriptedNutClient` in `tests/test_api.py`) because the live
|
a scripted `nut.NutClient` (`ScriptedNutClient` in `tests/test_api.py`) because the live
|
||||||
@@ -1693,7 +1843,10 @@ one — `/system`'s three per-host slots, `/disks`'s two per-host plots, `/llm`'
|
|||||||
plots, `/ups`'s four battery plots — each pinned by `tests/test_api.py`, so a chart that ships
|
plots, `/ups`'s four battery plots — each pinned by `tests/test_api.py`, so a chart that ships
|
||||||
anonymous fails a test.
|
anonymous fails a test.
|
||||||
Window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups` and `/disks`; the API accepts
|
Window chips (5m…30d on `/system` and `/llm`, 1h…90d on `/ups` and `/disks`; the API accepts
|
||||||
5m…90d — SMART lands every 300 s, so a 5-minute window on `/disks` is one dot). A
|
5m…90d — SMART lands every 300 s, so a 5-minute window on `/disks` is one dot). `/gateway` has
|
||||||
|
**no** chips and one window: a connection trend is read as "the last day of this pipe", so each
|
||||||
|
card's plot is a fixed 24 h named by the `Charts` chip beside its toggle rather than a page control
|
||||||
|
that would re-fetch every card at once. A
|
||||||
panel that cannot get data says why in the panel — the operator should never have to open
|
panel that cannot get data says why in the panel — the operator should never have to open
|
||||||
devtools to learn that a source is unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
|
devtools to learn that a source is unconfigured. Escape anything upstream-provided through `DashCharts.escapeHtml`.
|
||||||
|
|
||||||
@@ -1726,6 +1879,14 @@ the table never implies last good numbers are live and never silently loses a ma
|
|||||||
No summary cards and no raw event feed survive on `/` — `tests/test_api.py` pins the
|
No summary cards and no raw event feed survive on `/` — `tests/test_api.py` pins the
|
||||||
two-panel shape against the served HTML.
|
two-panel shape against the served HTML.
|
||||||
|
|
||||||
|
Under the fleet table sits the **ingress's one line** (phase 29 task 05): `#fleet-gate`, filled by
|
||||||
|
`renderGateway()` from `/api/summary`'s `gateway` block with the block's own `href`, so the page
|
||||||
|
invents neither a route nor a number (`gateway 14 inbound · 4 outbound`, `n/a` for a count the
|
||||||
|
source did not measure, the shared `.stale` dim when the agent stopped answering, and the reason in
|
||||||
|
its `title` when it is `partial`). `configured: false` renders **nothing at all** — the `disks`
|
||||||
|
block's invariant-3 rule, no row and no red one to learn to ignore — and the row is a `<p>` inside
|
||||||
|
the fleet panel because the page is two panels, not three (the pin counts `<section>`s).
|
||||||
|
|
||||||
Nothing waits for a writer that never came: `tests/test_api.py` fails if a page keeps an
|
Nothing waits for a writer that never came: `tests/test_api.py` fails if a page keeps an
|
||||||
`id` that no page script, no shared script, no stylesheet rule and no `aria-*`/`for`
|
`id` that no page script, no shared script, no stylesheet rule and no `aria-*`/`for`
|
||||||
attribute references — the orphan a deleted renderer leaves behind, caught on served bytes
|
attribute references — the orphan a deleted renderer leaves behind, caught on served bytes
|
||||||
@@ -1877,6 +2038,41 @@ repaints only when charge / verdict / state change (`drawGlyph()` keys a `data-d
|
|||||||
charge float jitters every `NUT_INTERVAL_SECONDS`, and rebuilding the `<svg>` on that re-announces
|
charge float jitters every `NUT_INTERVAL_SECONDS`, and rebuilding the `<svg>` on that re-announces
|
||||||
an accessible name for a difference no eye can see.
|
an accessible name for a difference no eye can see.
|
||||||
|
|
||||||
|
**The `/gateway` page draws pipes, and a pipe's only claim is a count** (phase 29): four cards keyed
|
||||||
|
on the payload's own `source` word (`haproxy`, `nginx`, `map`) plus `unmatched`, each shipping
|
||||||
|
`hidden` and appearing on the poll that has pipes for it — a gateway with no nginx in it spends no
|
||||||
|
panel saying so, and a section that did not answer at all is the page's one `why` line rather than a
|
||||||
|
card repeating it in a smaller font. A row is inbound (`:<listen port>`, its `server_name`s as the
|
||||||
|
domains, up to `MAX_CLIENTS` client addresses as the footnote) or outbound (the target **as the
|
||||||
|
config wrote it**, with the SNI name the stream map matched on in front of it when exactly one key
|
||||||
|
routes there), and its count is `countText()` — a `null` from the join prints `n/a` and never `0`
|
||||||
|
(rule 2). The pipe is hand-rolled SVG (`role="img"` + its own `aria-label` naming port/target,
|
||||||
|
domains, clients and the count), it **animates only while the count is ≥ 1**, and it is **never
|
||||||
|
coloured by a verdict**: `severity.py` has no `gateway.*` row, the payload carries no `severity` at
|
||||||
|
any depth, and `tests/test_api.py::test_no_gateway_count_is_coloured_by_a_verdict` fails the day a
|
||||||
|
`severity`/`ok|warn|fail` token appears in the page or the script (a busy ingress is the job, the
|
||||||
|
`net.*` and GPU precedents). A target matched on its socket **port** alone carries a `by port` chip,
|
||||||
|
because that count is the port's and not the target's (`by_port`, and the reason a shared `:443`
|
||||||
|
fan-out prints `n/a` instead of repeating one number sixteen times). The page head's five tallies are
|
||||||
|
the socket table's own totals (`listeners`, `established`) beside the totals the pipes carry (`pipes
|
||||||
|
in`, `pipes out`) plus `unmatched`, so the page prints its reconciliation rather than asserting it,
|
||||||
|
and the `unmatched` card's three cells stay three (`other` counts **ports**, the other two
|
||||||
|
**connections**). A label gets its **line**: `.pipe-label` wraps and `.pipe-name` is based on `max-content`, so a short
|
||||||
|
`:22` sits beside its count and a long target takes the whole column with the `by port` chip, the
|
||||||
|
config's own `server` name and the count beneath it (task 06 measured the alternative — one `nowrap`
|
||||||
|
line, seven characters of a hostname — and `data/logs/29-task06-card-haproxy.png` is the result). What
|
||||||
|
still does not fit a ~150px column is ellipsised and not lost: `renderPipeRow` puts the whole string
|
||||||
|
on the element as `title` and `pipeAria` carries the whole endpoint, which is what
|
||||||
|
`test_a_clipped_pipe_label_keeps_its_full_text_on_the_element` pins along with the two declarations;
|
||||||
|
the residual and the options for it are gap 13. Charts follow `/system`: one
|
||||||
|
collapsed `Charts` `<details>` per card, built and
|
||||||
|
painted only while open, fed by **one** `/api/gateway/history` request per card per refresh naming
|
||||||
|
that card's own gauges, the legend naming pipes because the card heading already says which config
|
||||||
|
family it is. Every degradation arm uses the payload's own words — the `why` sentence for a
|
||||||
|
pre-1.2.0/`--no-proxy` agent or a silent section, the `unreachable` dim for a host that has never
|
||||||
|
answered, and `#gate-empty` (naming `GATEWAY_HOST` and agent **1.2.0**) for `404 {configured: false}`
|
||||||
|
— and a failed poll re-draws nothing, so the last topology stays on screen.
|
||||||
|
|
||||||
An event row names its subject **once**. Every feed (`/system`, `/disks`, `/llm`, `/ups`) renders the
|
An event row names its subject **once**. Every feed (`/system`, `/disks`, `/llm`, `/ups`) renders the
|
||||||
message through `Dash.eventText(event)`, which drops the `UpstreamError: ` wrapper and a
|
message through `Dash.eventText(event)`, which drops the `UpstreamError: ` wrapper and a
|
||||||
*leading* duplicated subject the way `alerts._clean_error()` does for the overview — `poller.py`
|
*leading* duplicated subject the way `alerts._clean_error()` does for the overview — `poller.py`
|
||||||
@@ -2660,8 +2856,9 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
|||||||
Task 02 wrote that audit: `data/logs/27-task02-mobile-audit.cjs`
|
Task 02 wrote that audit: `data/logs/27-task02-mobile-audit.cjs`
|
||||||
(`NODE_PATH=<playwright node_modules> node …`, `DASH_BASE`/`DASH_PW`, transcript
|
(`NODE_PATH=<playwright node_modules> node …`, `DASH_BASE`/`DASH_PW`, transcript
|
||||||
`27-task02-mobile-audit.out`, screenshots `data/logs/mobile-27-<page>-<width>.png`) sweeps
|
`27-task02-mobile-audit.out`, screenshots `data/logs/mobile-27-<page>-<width>.png`) sweeps
|
||||||
**every page the app serves** — the seven signed-in ones plus `/login` unsigned, eight in
|
**every page the app serves** — every signed-in page plus `/login` reached unsigned (nine of
|
||||||
total, not the nine the phase file counted — at 360 px and 1280 px, and per page demands: no
|
them since phase 29 task 05 put `/gateway` in the list; the sweep is exactly the kind of page a
|
||||||
|
hand-written four-page loop would have missed) — at 360 px and 1280 px, and per page demands: no
|
||||||
sideways document scroll, no `table.data` clipped (its `overflow-x` must be `auto`/`scroll`
|
sideways document scroll, no `table.data` clipped (its `overflow-x` must be `auto`/`scroll`
|
||||||
when its content is wider, and its tail column measured reachable), no element past the
|
when its content is wider, and its tail column measured reachable), no element past the
|
||||||
viewport except inside a horizontal scroll container, the three-row masthead with its controls
|
viewport except inside a horizontal scroll container, the three-row masthead with its controls
|
||||||
@@ -2722,7 +2919,10 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
|||||||
0 FAIL / 0 FINDING** (9 skips, each one a check that width cannot answer — the clock, hidden
|
0 FAIL / 0 FINDING** (9 skips, each one a check that width cannot answer — the clock, hidden
|
||||||
below 900 px) on all eight served pages at both widths, exit 0
|
below 900 px) on all eight served pages at both widths, exit 0
|
||||||
(`data/logs/27-task02-mobile-audit.out`, screenshots `mobile-27-*.png`; the first run is kept
|
(`data/logs/27-task02-mobile-audit.out`, screenshots `mobile-27-*.png`; the first run is kept
|
||||||
as `27-task02-first-run.out`), and `data/logs/27-task03-control.cjs` proves the new check is
|
as `27-task02-first-run.out`). Phase 29's completion pass re-ran the same sweep now that
|
||||||
|
`/gateway` is in it — **371 PASS / 0 FAIL / 0 FINDING**, 10 skips, nine served pages, exit 0
|
||||||
|
(`data/logs/29-final-mobile-audit.out`) — so the two declarations above still hold on a page
|
||||||
|
this audit was written before, and `data/logs/27-task03-control.cjs` proves the new check is
|
||||||
not vacuous by neutralising both declarations and re-creating the whole first-run findings list
|
not vacuous by neutralising both declarations and re-creating the whole first-run findings list
|
||||||
(3/4/2 header offenders at 360 px, 8 stranded disclosure tables at 1280 px). The served-byte
|
(3/4/2 header offenders at 360 px, 8 stranded disclosure tables at 1280 px). The served-byte
|
||||||
pins are `test_a_card_header_wraps_its_facts_instead_of_losing_its_tail` and
|
pins are `test_a_card_header_wraps_its_facts_instead_of_losing_its_tail` and
|
||||||
@@ -2734,7 +2934,7 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
|||||||
beside it, `20_masthead_nav_polish` has landed in full (now complete): the bar is three
|
beside it, `20_masthead_nav_polish` has landed in full (now complete): the bar is three
|
||||||
rows at every width (flush right on the two control rows since phase 25 task 02), it is no
|
rows at every width (flush right on the two control rows since phase 25 task 02), it is no
|
||||||
longer sticky, the phone menu slides, and both
|
longer sticky, the phone menu slides, and both
|
||||||
`⚙ Account` and `⚷ Certs` render on all seven signed-in pages including the two they name
|
`⚙ Account` and `⚷ Certs` render on all eight signed-in pages including the two they name
|
||||||
(task 04, reversing phase 15's "no self-link" rule at the operator's request). The other,
|
(task 04, reversing phase 15's "no self-link" rule at the operator's request). The other,
|
||||||
`24_dev_mtls_rollout` (never accepting a non-mTLS agent) is closed: its fleet half landed on
|
`24_dev_mtls_rollout` (never accepting a non-mTLS agent) is closed: its fleet half landed on
|
||||||
2026-10-21 and task 04 re-read the end state read-only the same day — gap 11. No `http://`
|
2026-10-21 and task 04 re-read the end state read-only the same day — gap 11. No `http://`
|
||||||
@@ -2742,6 +2942,40 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
|||||||
seed one (`tests/test_deploy_scripts.py` keeps it that way). What remains of phase 24 is
|
seed one (`tests/test_deploy_scripts.py` keeps it that way). What remains of phase 24 is
|
||||||
ordinary rotation hygiene, not a missing step.
|
ordinary rotation hygiene, not a missing step.
|
||||||
|
|
||||||
|
13. **`/gateway`'s long pipe labels are ellipsised at a desktop width — a column-width question,
|
||||||
|
not a lost fact** (found and half-fixed by phase 29 task 06's live pass,
|
||||||
|
`data/logs/29-task06-live.out`; the picture is `data/logs/29-task06-card-haproxy.png`).
|
||||||
|
|
||||||
|
What the pass found first: at 1440px the outbound name showed **seven characters**
|
||||||
|
(`reeseapps-…`) for a 40-character `server_name → target` string. `.pipe-label` was one `nowrap`
|
||||||
|
flex row, so the name shrank to whatever the `by port` chip, the config's own `server` name and
|
||||||
|
the `flex: none` count left. Wrapping the *text* inside that box was measured and rejected — it
|
||||||
|
tore one hostname over eleven lines of a word each. What shipped instead is the label getting
|
||||||
|
the line to itself: `flex-wrap: wrap` on `.pipe-label` beside `flex: 1 1 max-content` on
|
||||||
|
`.pipe-name`, so a short label (`:22`) keeps its count on the same line as before and a long one
|
||||||
|
is by itself wider than the column, takes the whole column, and lets the chip, the footnote and
|
||||||
|
the count wrap beneath it. That is ~22 characters instead of 7, the config's own `server` name
|
||||||
|
now reads in full, and nothing was truncated that was not truncated before.
|
||||||
|
|
||||||
|
**What is left** is arithmetic, not a bug: **28 of 42** labels still show an ellipsis because
|
||||||
|
`.pipe-columns` is `minmax(0, 1fr) auto minmax(0, 1fr)` inside one of three cards across a
|
||||||
|
1440px viewport, which is a ~150px column, and `reeseapps-brain-of-code.reeselink.com:443` is 41
|
||||||
|
characters. No fact is lost — `renderPipeRow` puts the whole string on the element as `title` and
|
||||||
|
`pipeAria` carries the whole endpoint, which is what
|
||||||
|
`tests/test_api.py::test_a_clipped_pipe_label_keeps_its_full_text_on_the_element` pins, and the
|
||||||
|
two declarations above are pinned beside it so the label cannot be squeezed back to seven
|
||||||
|
characters. In order of cost: (a) give the outbound column more of the card than the inbound one
|
||||||
|
(inbound labels are `:22`-shaped, so `minmax(0, .45fr) auto minmax(0, 1.55fr)` buys ~90px); (b)
|
||||||
|
two cards per row for the fan-out, which is the honest layout for a 17-row map but leaves the
|
||||||
|
third of the viewport empty; (c) accept it — the stream map's rows are read as a set, the
|
||||||
|
left-hand SNI name is the part that fits, and the count is the number the row is about. The
|
||||||
|
≤640px arm already stacks the two columns into one, where the label gets the whole card and
|
||||||
|
**16 of 42** clip instead of 28 (a stream-map string here runs to 70 characters, so even a 312px
|
||||||
|
column is not enough for the longest few) — measured in the same pass, which prints both counts
|
||||||
|
and demands a `title` on every clipped one. Whoever picks this up re-runs `data/logs/29-task06-live.cjs`: its
|
||||||
|
FINDING lines are the two counts, and the run fails if a clipped label ever stops carrying its
|
||||||
|
full text.
|
||||||
|
|
||||||
## Locked decisions (and why)
|
## Locked decisions (and why)
|
||||||
|
|
||||||
- **SQLite, not Prometheus/Grafana.** Handful of boxes, 1–10s resolution, one file the
|
- **SQLite, not Prometheus/Grafana.** Handful of boxes, 1–10s resolution, one file the
|
||||||
|
|||||||
@@ -1,15 +1,16 @@
|
|||||||
# Homelab Dashboards
|
# Homelab Dashboards
|
||||||
|
|
||||||
Four dashboards and the overview page that points at them — one domain, one process, one SQLite
|
Five dashboards and the overview page that points at them — one domain, one process, one SQLite
|
||||||
file:
|
file:
|
||||||
|
|
||||||
| Page | What it answers | Source |
|
| Page | What it answers | Source |
|
||||||
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
|
| ------------ | ------------------------------------------------------------------------------- | -------------------------------------------------- |
|
||||||
| `/` | Is anything wrong, and which dashboard do I need? | all four, in one request (`/api/summary`) |
|
| `/` | Is anything wrong, and which dashboard do I need? | all five, in one request (`/api/summary`) |
|
||||||
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
|
| `/system` | What are my machines doing — CPU, memory, disk, GPU, network, and the history of all of it | `agent/host_agent.py`, our own stdlib agent per host |
|
||||||
| `/disks` | Are any of my **physical** disks failing, and which one — health, temperature, power-on hours, wear, sector alarms | the same agents, running `smartctl -i -A -H` per whole drive (never a partition or a mount); a host whose agent has no `smartctl` is not a host on this page |
|
| `/disks` | Are any of my **physical** disks failing, and which one — health, temperature, power-on hours, wear, sector alarms | the same agents, running `smartctl -i -A -H` per whole drive (never a partition or a mount); a host whose agent has no `smartctl` is not a host on this page |
|
||||||
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
|
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
|
||||||
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
|
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
|
||||||
|
| `/gateway` | Which domains are coming in on which port, and where the traffic goes — the ingress's listeners, `server_name`s and SNI map joined to its live connections | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |
|
||||||
|
|
||||||
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
|
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
|
||||||
runtime; every asset is served by this app.
|
runtime; every asset is served by this app.
|
||||||
@@ -155,8 +156,8 @@ what the config says: every `frontend` block with its name, each `bind` address:
|
|||||||
written (`:::22` is the address `::` on port `22`) and the backend names it routes to
|
written (`:::22` is the address `::` on port `22`) and the backend names it routes to
|
||||||
(`default_backend`, plus every named `use_backend`), and every `backend` block with each `server`'s
|
(`default_backend`, plus every named `use_backend`), and every `backend` block with each `server`'s
|
||||||
name, target and port — `port: null` when the line names none, never `0`. It interprets nothing:
|
name, target and port — `port: null` when the line names none, never `0`. It interprets nothing:
|
||||||
the routing between the two lists is the **names**, so joining them is the page's job (the gateway
|
the routing between the two lists is the **names**, so joining them is the gateway page's job
|
||||||
page is phase 29) and the agent never evaluates a rule. `available` answers three questions the way `smartctl_available` does —
|
(`sources/gateway.py`) and the agent never evaluates a rule. `available` answers three questions the way `smartctl_available` does —
|
||||||
`true` read and parsed, `false` no config at that path (this host runs no haproxy), `null` a file
|
`true` read and parsed, `false` no config at that path (this host runs no haproxy), `null` a file
|
||||||
is there but could not be read or understood — and each non-`true` answer costs exactly **one**
|
is there but could not be read or understood — and each non-`true` answer costs exactly **one**
|
||||||
`errors` line for the pass. Like the SMART lines, those lines never make the *system* page amber:
|
`errors` line for the pass. Like the SMART lines, those lines never make the *system* page amber:
|
||||||
@@ -201,10 +202,13 @@ kernel without IPv6 has no `tcp6` file, which costs nothing and no line, while a
|
|||||||
there and denied costs one. `available` is `true` when either table answered and `null` when
|
there and denied costs one. `available` is `true` when either table answered and `null` when
|
||||||
neither did, with one line saying why; there is no `false`, because a table that did not answer is
|
neither did, with one line saying why; there is no `false`, because a table that did not answer is
|
||||||
"no answer" and not "this host has no sockets". Nothing here is joined to a config: a socket does
|
"no answer" and not "this host has no sockets". Nothing here is joined to a config: a socket does
|
||||||
not know its domain or its backend, so phase 29's gateway page will be what joins these ports to
|
not know its domain or its backend, so the **gateway page** is what joins these ports to the
|
||||||
the `haproxy` and `nginx` sections — where a socket no frontend claims stays visible as one. Until
|
`haproxy` and `nginx` sections, and a socket no config claims stays visible as one. The join lives
|
||||||
that page exists the three sections live only in the agent's own `/snapshot` document: no `/api`
|
in `sources/gateway.py`, never in the agent: `/snapshot` keeps the three sections as the agent read
|
||||||
payload grows them, no `sample` row is stored for them, and nothing charts them.
|
them, `/api/gateway/snapshot` serves the joined topology, and only the counts that join produces are
|
||||||
|
stored — as the `gateway.in_conns@<port>` / `gateway.out_conns@<host>:<port>` gauges the page
|
||||||
|
charts. Nothing in `/api/system/snapshot` grows a proxy or socket field, and no `sample` row is
|
||||||
|
written for a config.
|
||||||
|
|
||||||
All three parsers are pinned to captures rather than to descriptions, and
|
All three parsers are pinned to captures rather than to descriptions, and
|
||||||
`./scripts/capture-gateway-config.sh` re-records them over the `GATEWAY_SSH` alias (default
|
`./scripts/capture-gateway-config.sh` re-records them over the `GATEWAY_SSH` alias (default
|
||||||
@@ -459,8 +463,9 @@ cannot match against a denylist.
|
|||||||
One admin, one password, and the server **refuses to start** when it has neither a stored
|
One admin, one password, and the server **refuses to start** when it has neither a stored
|
||||||
verifier nor `DASH_ADMIN_PASSWORD` — the domain is public and the fleet's hostnames and UPS
|
verifier nor `DASH_ADMIN_PASSWORD` — the domain is public and the fleet's hostnames and UPS
|
||||||
controls sit behind this login, so there is no way to run it open by accident. There is no
|
controls sit behind this login, so there is no way to run it open by accident. There is no
|
||||||
second account and no per-user view: the five pages (`/`, `/system`, `/disks`, `/llm`, `/ups`),
|
second account and no per-user view: the six dashboard pages (`/`, `/system`, `/disks`, `/llm`,
|
||||||
the two admin pages (`/settings`, `/certs`) and every data route belong to that one operator.
|
`/ups`, `/gateway`), the two admin pages (`/settings`, `/certs`) and every data route belong to
|
||||||
|
that one operator.
|
||||||
|
|
||||||
A session is an **HMAC-signed cookie** (`dash_session`), not a row in a table: it carries the
|
A session is an **HMAC-signed cookie** (`dash_session`), not a row in a table: it carries the
|
||||||
signed-in name and its own expiry, is valid for `DASH_SESSION_HOURS` (default `720` = 30 days),
|
signed-in name and its own expiry, is valid for `DASH_SESSION_HOURS` (default `720` = 30 days),
|
||||||
@@ -671,10 +676,11 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
|
|||||||
| `RATE_WINDOW_SECONDS` | the **host's** Δ window (default 10) — bytes/sec and jiffies→CPU %, measured over what a 10 s poll can fill; each value carries the window it actually got, and `/system`'s footnote prints that, not this default. A window the poll loop cannot honour — the ~0.2 s catch-up after a fetch slower than the interval, or a base older than 2× the window — answers one `n/a` instead of a spike, and a rate 8× above the last one is held until a window measured from a **different** sample confirms it (phase 13). The LLM rates do **not** use it: they are measured over `LLM_INTERVAL_SECONDS` (phase 23) |
|
| `RATE_WINDOW_SECONDS` | the **host's** Δ window (default 10) — bytes/sec and jiffies→CPU %, measured over what a 10 s poll can fill; each value carries the window it actually got, and `/system`'s footnote prints that, not this default. A window the poll loop cannot honour — the ~0.2 s catch-up after a fetch slower than the interval, or a base older than 2× the window — answers one `n/a` instead of a spike, and a rate 8× above the last one is held until a window measured from a **different** sample confirms it (phase 13). The LLM rates do **not** use it: they are measured over `LLM_INTERVAL_SECONDS` (phase 23) |
|
||||||
| `LLM_INTERVAL_SECONDS` | the `/llm` poll cadence (default 2 s) **and** the Δ window behind every token rate — one estimator per measuring cadence, so a slot that starts reads steady within two polls instead of ramping over five. Slowing the poll slows the window with it; there is no separate LLM window knob, and a window narrower than the interval would answer `n/a` every poll |
|
| `LLM_INTERVAL_SECONDS` | the `/llm` poll cadence (default 2 s) **and** the Δ window behind every token rate — one estimator per measuring cadence, so a slot that starts reads steady within two polls instead of ramping over five. Slowing the poll slows the window with it; there is no separate LLM window knob, and a window narrower than the interval would answer `n/a` every poll |
|
||||||
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
|
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
|
||||||
| `HOST_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often the host agents and upsd are polled (10 / 5 s) |
|
| `HOST_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often the host agents and upsd are polled (10 / 5 s). `GATEWAY_INTERVAL_SECONDS` (30) and `GATEWAY_TIMEOUT_SECONDS` (10) are the gateway source's own pair: a connection count is the shape of half a minute, and the fetch is the plain agent one with headroom for a handler that may be mid-smartctl pass |
|
||||||
| `DISKS_INTERVAL_SECONDS` | how often the disk-health source re-reads each agent for SMART (default 300 s — SMART is not a per-second signal). No targets of its own: the `disks:<host>` jobs exist exactly when `HOST_TARGETS` does |
|
| `DISKS_INTERVAL_SECONDS` | how often the disk-health source re-reads each agent for SMART (default 300 s — SMART is not a per-second signal). No targets of its own: the `disks:<host>` jobs exist exactly when `HOST_TARGETS` does |
|
||||||
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
|
| `HOST_TIMEOUT_SECONDS` / `LLM_TIMEOUT_SECONDS` / `NUT_TIMEOUT_SECONDS` | per-source fetch timeout — one slow source never holds up the others |
|
||||||
| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and a wedged SMART pass must not read as a dead host |
|
| `DISKS_TIMEOUT_SECONDS` | the disks fetch's own budget (default 30 s), because the agent runs `smartctl` inside its snapshot handler and a wedged SMART pass must not read as a dead host |
|
||||||
|
| `GATEWAY_HOST` | which `HOST_TARGETS` name is the ingress the `/gateway` page watches (default `gateway`). It has no URL of its own — the source reads the same agent `/snapshot` the system page reads, projected onto agent 1.2.0's `haproxy`/`nginx`/`sockets` sections — so a name that is not in `HOST_TARGETS` means "no ingress to watch", and the source is unconfigured rather than red. The page refetches itself every 15 s in the browser and has no `POLL_MS_*` knob, because the payload only moves when that one job re-reads the ingress |
|
||||||
| `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server |
|
| `GUFO_API_KEY` / `VLLM_API_KEY` | only if an engine is behind a bearer token; they never leave the server |
|
||||||
| `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
|
| `NUT_HOST` / `NUT_PORT` | upsd address — **whatever port it really listens on**, not always 3493. Reads are unauthenticated in stock NUT |
|
||||||
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
|
| `NUT_USER` / `NUT_PASSWORD` | only needed for instant commands (battery tests) |
|
||||||
@@ -686,7 +692,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
|
|||||||
| `DASH_SESSION_HOURS` | session lifetime (default `720` = 30 days) |
|
| `DASH_SESSION_HOURS` | session lifetime (default `720` = 30 days) |
|
||||||
| `DASH_COOKIE_SECURE` | `1` marks the session cookie `Secure`; off because dev is `http://127.0.0.1`, and a request that arrived over `X-Forwarded-Proto: https` is treated as secure either way |
|
| `DASH_COOKIE_SECURE` | `1` marks the session cookie `Secure`; off because dev is `http://127.0.0.1`, and a request that arrived over `X-Forwarded-Proto: https` is treated as secure either way |
|
||||||
| `DASH_PBKDF2_ITERATIONS` | verifier cost (default `600000`). The knob exists for the test suite; leave it alone in a deployment. Raising it on a real box does take effect: the next successful sign-in re-hashes a cheaper stored verifier and keeps the signing key |
|
| `DASH_PBKDF2_ITERATIONS` | verifier cost (default `600000`). The knob exists for the test suite; leave it alone in a deployment. Raising it on a real box does take effect: the next successful sign-in re-hashes a cheaper stored verifier and keeps the signing key |
|
||||||
| `POLL_MS_SYSTEM` / `POLL_MS_LLM` / `POLL_MS_UPS` | how often the *browser* refetches each page (10 s / 2 s / 5 s) — shown in the page footer. `/disks` has no `POLL_MS_*` of its own and polls at a fixed 30 s: its payload only moves when the `disks` job re-reads the agents. A backgrounded tab idles its polls and fires one on return; nothing an operator presses stops them (phase 21) |
|
| `POLL_MS_SYSTEM` / `POLL_MS_LLM` / `POLL_MS_UPS` | how often the *browser* refetches each page (10 s / 2 s / 5 s) — shown in the page footer. `/disks` and `/gateway` have no `POLL_MS_*` of their own and poll at a fixed 30 s / 15 s: the first payload only moves when the `disks` job re-reads the agents, the second when the one `gateway` job re-reads the ingress. A backgrounded tab idles its polls and fires one on return; nothing an operator presses stops them (phase 21) |
|
||||||
| `DASH_RELOAD` | `1` makes `uv run dashboards` watch the source and restart on a change — development only, and every poll loop restarts with it |
|
| `DASH_RELOAD` | `1` makes `uv run dashboards` watch the source and restart on a change — development only, and every poll loop restarts with it |
|
||||||
| `DEBUGPY` / `DEBUGPY_PORT` | `DEBUGPY=1` opens a debugpy listener (default 5678) without blocking startup |
|
| `DEBUGPY` / `DEBUGPY_PORT` | `DEBUGPY=1` opens a debugpy listener (default 5678) without blocking startup |
|
||||||
| `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning |
|
| `RETAIN_RAW_HOURS` / `RETAIN_MINUTE_DAYS` / `RETAIN_HOUR_DAYS` | history rollup and pruning |
|
||||||
@@ -700,12 +706,13 @@ installed.
|
|||||||
```
|
```
|
||||||
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
|
agent/host_agent.py (each host) gufo / vLLM upsd (NUT)
|
||||||
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
|
│ GET /snapshot │ GET /metrics │ LIST UPS / LIST VAR / LIST CMD
|
||||||
│ (twice: system 10s, disks 5m) │ + /slots (gufo) │
|
│ (three readers: system 10s, │ + /slots (gufo) │
|
||||||
|
│ disks 5m, gateway 30s) │ │
|
||||||
▼ ▼ ▼
|
▼ ▼ ▼
|
||||||
src/dashboards/collectors.py ────────────────────────────────────────────┐
|
src/dashboards/collectors.py ────────────────────────────────────────────┐
|
||||||
│ parse + Δcounter/Δt (rates.py) │
|
│ parse + Δcounter/Δt (rates.py) │
|
||||||
▼ ▼
|
▼ ▼
|
||||||
sources/host.py · disks.py · llm.py · nut.py poller.py (one task per source)
|
sources/host.py · disks.py · gateway.py · llm.py · nut.py poller.py (one task per source)
|
||||||
│ samples events │
|
│ samples events │
|
||||||
▼ ▼
|
▼ ▼
|
||||||
store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event + meta ── api.py → web/
|
store.py ── SQLite (WAL): sample(source,host,metric,res,ts) + event + meta ── api.py → web/
|
||||||
@@ -720,9 +727,14 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
|||||||
`/api/health`, 5-minute cadence, its own fetch timeout) reading the *same* agent
|
`/api/health`, 5-minute cadence, its own fetch timeout) reading the *same* agent
|
||||||
`/snapshot` the system page reads — a separate task is what keeps a SMART poll out of the
|
`/snapshot` the system page reads — a separate task is what keeps a SMART poll out of the
|
||||||
10-second utilisation poll and lets drive history go stale without blinking `/system`, and
|
10-second utilisation poll and lets drive history go stale without blinking `/system`, and
|
||||||
`alerts.py` is why one dead agent is still one line on the overview rather than two. And it never quietly *stops*: the task is that source's only
|
`alerts.py` is why one dead agent is still one line on the overview rather than two. The gateway
|
||||||
poller, so a SQLite write that gets refused is reported (`ok: false`, the real reason, the
|
page's job (phase 29) is the **third** reader of that same `/snapshot`, and one job for the whole
|
||||||
numbers aging, one line in the event log) instead of freezing the source under a green dot.
|
deployment rather than one per host — `GATEWAY_HOST` names the single ingress, and a job per host
|
||||||
|
would ask four machines a question only one can answer — on its own 30 s cadence with no
|
||||||
|
`RateEstimator`, because a connection count is not a rate. And it never quietly *stops*: the task
|
||||||
|
is that source's only poller, so a SQLite write that gets refused is reported (`ok: false`, the
|
||||||
|
real reason, the numbers aging, one line in the event log) instead of freezing the source under a
|
||||||
|
green dot.
|
||||||
* `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
|
* `store.py` — raw samples roll up to 1-minute and 1-hour averages and the raw rows are
|
||||||
pruned, so a year of history is a few hundred MB rather than a disk emergency. Nothing a
|
pruned, so a year of history is a few hundred MB rather than a disk emergency. Nothing a
|
||||||
per-request path asks is answered by a whole-table scan **on every call** (phase 14): the two
|
per-request path asks is answered by a whole-table scan **on every call** (phase 14): the two
|
||||||
@@ -730,8 +742,8 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
|||||||
row counts behind `/api/health` — come from a 60 s / 30 s cache, so the scan is paid once per
|
row counts behind `/api/health` — come from a 60 s / 30 s cache, so the scan is paid once per
|
||||||
window (its cost is the honest remainder in the known-gaps list) instead of once per request. `series_names`' is ended only by a write that can
|
window (its cost is the honest remainder in the known-gaps list) instead of once per request. `series_names`' is ended only by a write that can
|
||||||
change it: a `(source, host, metric)` the store has never written, or any `DELETE`. A poll of an
|
change it: a `(source, host, metric)` the store has never written, or any `DELETE`. A poll of an
|
||||||
existing series leaves it alone, which is what makes the cache serve at all on a store nine pollers
|
existing series leaves it alone, which is what makes the cache serve at all on a store every
|
||||||
write to (task 04's live watch is the proof: 535 ms per call before, 1.8 ms median after), and the
|
poller writes to (task 04's live watch is the proof: 535 ms per call before, 1.8 ms median after), and the
|
||||||
counts age on the clock. A chart's resolution comes from one capped compound statement instead of
|
counts age on the clock. A chart's resolution comes from one capped compound statement instead of
|
||||||
three full `COUNT`s per series. `last_maintain` and `size_bytes` are deliberately never cached —
|
three full `COUNT`s per series. `last_maintain` and `size_bytes` are deliberately never cached —
|
||||||
the first is how an operator tells a stalled rollup loop from a healthy one.
|
the first is how an operator tells a stalled rollup loop from a healthy one.
|
||||||
@@ -801,12 +813,17 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
|||||||
that stopped answering is `fail`; one that is stale but still serving its last good value
|
that stopped answering is `fail`; one that is stale but still serving its last good value
|
||||||
is `warn`; an **unconfigured** source never alerts; a dead source is one line rather than
|
is `warn`; an **unconfigured** source never alerts; a dead source is one line rather than
|
||||||
one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a
|
one per stale metric; the list dedupes per (source, metric) and is capped at 12 with a
|
||||||
marker that says how many it hid.
|
marker that says how many it hid. A job that dials an agent another job already named adds no
|
||||||
|
line either (`AGENT_ECHO_DASHBOARDS`: `disks` and, since phase 29, `gateway` — three jobs read
|
||||||
|
one `/snapshot`, so one dead box is one overview row), while a job failing *while its host
|
||||||
|
answers* keeps its own voice and links to the page that explains it (`/disks` for a wedged SMART
|
||||||
|
pass, `/gateway` for a proxy or socket section that went quiet). Staleness is measured in each
|
||||||
|
source's own interval, so the gateway page goes amber after 90 quiet seconds and not after 30.
|
||||||
* `api.py` — the JSON every page reads: `/api/config`, `/api/health`, `/api/events`,
|
* `api.py` — the JSON every page reads: `/api/config`, `/api/health`, `/api/events`,
|
||||||
`/api/history`, `/api/metrics`, `/api/summary`, one `snapshot` + `history` pair per
|
`/api/history`, `/api/metrics`, `/api/summary`, one `snapshot` + `history` pair per
|
||||||
dashboard (`/api/system/*`, `/api/disks/*`, `/api/llm/*`, `/api/ups/*` — including
|
dashboard (`/api/system/*`, `/api/disks/*`, `/api/gateway/*`, `/api/llm/*`, `/api/ups/*` —
|
||||||
`/api/ups/commands`, which answers what may be sent to each UPS right now and why not when it
|
including `/api/ups/commands`, which answers what may be sent to each UPS right now and why not
|
||||||
may not), and the certificate authority's own surface (`/api/certs`,
|
when it may not), and the certificate authority's own surface (`/api/certs`,
|
||||||
`/api/certs/<name>/bundle`, `POST …/rotate`, `POST
|
`/api/certs/<name>/bundle`, `POST …/rotate`, `POST
|
||||||
…/revoke`, `POST /api/certs/root/rotate`). GETs everywhere except `POST /api/ups/command`,
|
…/revoke`, `POST /api/certs/root/rotate`). GETs everywhere except `POST /api/ups/command`,
|
||||||
which stays the **only** route that dials anything: the `/api/certs/*` writes are `meta` row
|
which stays the **only** route that dials anything: the `/api/certs/*` writes are `meta` row
|
||||||
@@ -818,7 +835,7 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
|||||||
no env config answers `404 {"configured": false}` **without dialling** (so a page says
|
no env config answers `404 {"configured": false}` **without dialling** (so a page says
|
||||||
"not configured" instead of going red), and a history query for a series that does not
|
"not configured" instead of going red), and a history query for a series that does not
|
||||||
exist answers an empty series instead of an error (so a card asks for every plot it could
|
exist answers an empty series instead of an error (so a card asks for every plot it could
|
||||||
draw and reads `n/a` for the ones the drive never reported). The four `history` routes take
|
draw and reads `n/a` for the ones the drive never reported). The five `history` routes take
|
||||||
one batched door into the store (`api._history_batch` → `store.history_many`), so a page's
|
one batched door into the store (`api._history_batch` → `store.history_many`), so a page's
|
||||||
chart request is one grouped resolution query plus one index seek per series rather than four
|
chart request is one grouped resolution query plus one index seek per series rather than four
|
||||||
queries per series — and the JSON it answers with is the same document the per-series loop
|
queries per series — and the JSON it answers with is the same document the per-series loop
|
||||||
@@ -842,13 +859,35 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
|||||||
drive the hardware called `FAILED` without a second call); it is additive, exactly as `alerts`
|
drive the hardware called `FAILED` without a second call); it is additive, exactly as `alerts`
|
||||||
was, and the key paths of both are pinned with and without targets configured. Since phase 22
|
was, and the key paths of both are pinned with and without targets configured. Since phase 22
|
||||||
that block lists the **same hosts** `/api/disks/snapshot` does, and `fleet.hosts_total` is
|
that block lists the **same hosts** `/api/disks/snapshot` does, and `fleet.hosts_total` is
|
||||||
that row count, so `/` and `/disks` can never describe two different fleets.
|
that row count, so `/` and `/disks` can never describe two different fleets. `/api/summary` grew a
|
||||||
* `web/` — five static dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`) plus the three
|
`gateway` block the same way in phase 29 (`configured`, `host`, `supported`, `unreachable`,
|
||||||
admin pages (`/login`, `/settings`, `/certs`), one stylesheet, hand-rolled
|
`inbound`, `outbound`, `map_targets`, `unmatched`, `unmatched_listeners`, `partial`, `states`):
|
||||||
|
the ingress's own tallies, `null` and never `0` in every arm that did not measure, `partial: true`
|
||||||
|
when a total had to leave an unattributable pipe out, and no `severity` anywhere — a connection
|
||||||
|
count is not a graded number. `/api/config` carries the matching row, including
|
||||||
|
`gateway.interval_seconds`/`timeout_seconds` and the named target's redacted `url`.
|
||||||
|
`/api/gateway/*` (phase 29 task 03) is that page's pair. `/snapshot` is one host's whole
|
||||||
|
topology — the haproxy frontends and the nginx `listen`/`map` echo the pipes were built from,
|
||||||
|
the inbound pipes (port, its domains, its live connection count, up to 32 client addresses), the
|
||||||
|
outbound pipes (target `host:port`, `by_port` naming which half of the match is a socket fact,
|
||||||
|
the count, the config that named it) each carrying the `metric` it is stored under, the
|
||||||
|
`unmatched` tallies, and `supported`/`sections`/`why` for the degradation arms — and the counts
|
||||||
|
reconcile: inbound + outbound + unmatched == `sockets.established`, so the page's numbers are
|
||||||
|
provable against a direct `/proc/net/tcp` read. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
|
||||||
|
a `404 {"configured": false}` answered before any client is built (both arms spy-pinned), a host
|
||||||
|
that has never answered is the `unreachable` stub whose every count is `null`, and a 1.1.0 agent
|
||||||
|
is `supported: false` plus its one `why` sentence — three arms, no red among them (invariant 3).
|
||||||
|
`/history` serves the `gateway.in_conns@<port>` and `gateway.out_conns@<host>:<port>` gauges —
|
||||||
|
the port in the outbound name because two services on one host are two facts — with `hosts=`
|
||||||
|
defaulting to the ingress alone, unknown names answering as empty series, and no dial either way
|
||||||
|
because history reads only SQLite.
|
||||||
|
* `web/` — six static dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`, `/gateway`) plus
|
||||||
|
the three admin pages (`/login`, `/settings`, `/certs`), one stylesheet, hand-rolled
|
||||||
SVG charts (`charts.js`), and a masthead status dot per dashboard job on **every** signed-in
|
SVG charts (`charts.js`), and a masthead status dot per dashboard job on **every** signed-in
|
||||||
page — the two admin pages included (`disks` has its own:
|
page — the two admin pages included (`disks` has its own:
|
||||||
the same agent can answer the 10-second utilisation poll and have a wedged SMART pass).
|
the same agent can answer the 10-second utilisation poll and have a wedged SMART pass, and
|
||||||
The masthead is **three rows at every width, the two control rows flush right**: the brand and the five dashboard
|
`gateway` has one because its job is its own poll of one ingress).
|
||||||
|
The masthead is **three rows at every width, the two control rows flush right**: the brand and the six dashboard
|
||||||
tabs, then the per-source indicators on their own row, then the `⚷ Certs` / `⚙ Account` /
|
tabs, then the per-source indicators on their own row, then the `⚷ Certs` / `⚙ Account` /
|
||||||
`☾ Dark` / clock controls on a row under those, in that order, so the **time is the farthest-right
|
`☾ Dark` / clock controls on a row under those, in that order, so the **time is the farthest-right
|
||||||
thing on the bar** (phase 25: the badges and the cluster were re-justified right, then the clock
|
thing on the bar** (phase 25: the badges and the cluster were re-justified right, then the clock
|
||||||
@@ -858,13 +897,13 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
|||||||
indicators, and so two rows, and a control is never hidden to make a
|
indicators, and so two rows, and a control is never hidden to make a
|
||||||
narrower width work. There is no control that stops the polling (phase 21): the bar used to carry
|
narrower width work. There is no control that stops the polling (phase 21): the bar used to carry
|
||||||
a `⏸ Pause` button with a `Space` shortcut, and it left the numbers on screen looking current with
|
a `⏸ Pause` button with a `Space` shortcut, and it left the numbers on screen looking current with
|
||||||
no `stale` label on them. Both admin links are on all seven
|
no `stale` label on them. Both admin links are on all eight
|
||||||
signed-in pages — the five dashboards plus `/settings` and `/certs`, the two they name included —
|
signed-in pages — the six dashboards plus `/settings` and `/certs`, the two they name included —
|
||||||
and not on `/login`, the eighth page and the only anonymous one: the cluster is the same wherever
|
and not on `/login`, the ninth page and the only anonymous one: the cluster is the same wherever
|
||||||
you are, and a link to the page you are already on is a harmless no-op rather than chrome that
|
you are, and a link to the page you are already on is a harmless no-op rather than chrome that
|
||||||
vanishes. The bar is **not sticky**: it scrolls away
|
vanishes. The bar is **not sticky**: it scrolls away
|
||||||
with the page, so every pixel of height goes to the data (it stays a positioned element, which is
|
with the page, so every pixel of height goes to the data (it stays a positioned element, which is
|
||||||
what keeps the mobile dropdown anchored under it). At ≤640px the five tab links leave the bar
|
what keeps the mobile dropdown anchored under it). At ≤640px the six tab links leave the bar
|
||||||
and come back as a dropdown under the whole of it, opened by the `☰` button (`/login` renders no
|
and come back as a dropdown under the whole of it, opened by the `☰` button (`/login` renders no
|
||||||
tabs, so it has no button either), and that menu **slides**: 180 ms of fade plus an 8 px rise on
|
tabs, so it has no button either), and that menu **slides**: 180 ms of fade plus an 8 px rise on
|
||||||
the way out and the same on the way back in, with nothing hidden from a keyboard or a screen
|
the way out and the same on the way back in, with nothing hidden from a keyboard or a screen
|
||||||
@@ -888,6 +927,9 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
|||||||
list reads as an explicit "all clear", and a live problem moves the panel above the
|
list reads as an explicit "all clear", and a live problem moves the panel above the
|
||||||
table. A host that is not answering keeps its row — dimmed, every number `n/a` — so the
|
table. A host that is not answering keeps its row — dimmed, every number `n/a` — so the
|
||||||
fleet never quietly gets smaller and last good values are never mistaken for live ones.
|
fleet never quietly gets smaller and last good values are never mistaken for live ones.
|
||||||
|
Under that table is the ingress's **one line** (`gateway · 14 inbound · 4 outbound`, linked to
|
||||||
|
`/gateway`): the counts are `/api/summary`'s own sums, a count the source did not measure prints
|
||||||
|
`n/a` rather than `0`, and a deployment with no `GATEWAY_HOST` renders no line at all.
|
||||||
A big colour-coded number is `Dash.hero(el, {...})` on the
|
A big colour-coded number is `Dash.hero(el, {...})` on the
|
||||||
`.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
|
`.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
|
||||||
`Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
|
`Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
|
||||||
@@ -994,6 +1036,26 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
|
|||||||
`No test initiated` or no variable at all — says the same "run a battery test" sentence as
|
`No test initiated` or no variable at all — says the same "run a battery test" sentence as
|
||||||
`unknown`. Only a word the map has never seen stays a warning, printed verbatim so it can
|
`unknown`. Only a word the map has never seen stays a warning, printed verbatim so it can
|
||||||
be added.
|
be added.
|
||||||
|
`/gateway` answers the topology question `/system` cannot: which sockets on the ingress face
|
||||||
|
inbound, where each one's traffic goes, and how many connections are live on each. It is four
|
||||||
|
cards keyed on what the **config** said a pipe came from — `haproxy`, `nginx`, the `stream map`,
|
||||||
|
and `unmatched` — each hidden until a poll actually has pipes for it, so a router with no nginx
|
||||||
|
in it spends no panel saying so. An inbound row is a listen port with its `server_name`s and the
|
||||||
|
client addresses on it; an outbound row is the target **exactly as the config wrote it** (the SNI
|
||||||
|
name the stream map matched on in front of it when one key routes there), and a target the join
|
||||||
|
could only match on its socket **port** carries a `by port` chip, because that count belongs to
|
||||||
|
the port and not to that service. A pipe animates while it carries at least one live connection
|
||||||
|
and rests at zero, and **nothing on the page is coloured by a verdict** — a busy ingress is the
|
||||||
|
job, so `severity.py` rates no `gateway.*` metric and no number gets a green/amber/red tint. A
|
||||||
|
count the join did not decide prints `n/a` and never `0`, which is why the fan-out on a port many
|
||||||
|
domains share is `n/a` rather than one number repeated as if it were sixteen. The header's five
|
||||||
|
tallies print the reconciliation instead of asserting it: what the kernel's socket table reported
|
||||||
|
(`listeners`, `established`) beside what the pipes carry (`pipes in`, `pipes out`) and what no
|
||||||
|
config claims (`unmatched` — ports and connections kept as separate cells). Each card's trend is
|
||||||
|
behind a collapsed **Charts** toggle like `/system`'s, one `/api/gateway/history` request per
|
||||||
|
card per refresh. With no `GATEWAY_HOST` the page is the setup sentence rather than a red panel,
|
||||||
|
and an ingress running an agent older than **1.2.0** says so in the agent's own words with every
|
||||||
|
count `n/a` and nothing coloured.
|
||||||
The four event feeds (`/system`, `/disks`, `/llm`, `/ups`) are a **paged window** over the `event`
|
The four event feeds (`/system`, `/disks`, `/llm`, `/ups`) are a **paged window** over the `event`
|
||||||
table rather than a printout of it: the list is a bounded box that scrolls, 50 events to a
|
table rather than a printout of it: the list is a bounded box that scrolls, 50 events to a
|
||||||
page, with `1–50 of 200` between `‹ prev` and `next ›` (disabled at the ends) underneath.
|
page, with `1–50 of 200` between `‹ prev` and `next ›` (disabled at the ends) underneath.
|
||||||
@@ -1171,6 +1233,19 @@ quietly becoming a fallback chain. A busy
|
|||||||
slot's `prompt` is the user's in-flight request text, so the capture script blanks it and
|
slot's `prompt` is the user's in-flight request text, so the capture script blanks it and
|
||||||
the page never reads it.
|
the page never reads it.
|
||||||
|
|
||||||
|
`tests/test_gateway_source.py` is where the ingress join's honesty lives, and it pins that module
|
||||||
|
from both ends. The input is the **captured** gateway — `gateway_haproxy.cfg`, the nginx main config
|
||||||
|
plus its included documents, and the two `/proc/net/tcp{,6}` captures — run through the real
|
||||||
|
`host_agent` parsers, so the test compares the join against the agent's own document rather than
|
||||||
|
against a copy of its field names. The assertions are the join's: the pipes carry exactly the ports,
|
||||||
|
`server_name`s and targets those documents name; `inbound + outbound + unmatched` equals the socket
|
||||||
|
table's own ESTABLISHED count, so no connection is counted twice or dropped; a listening port no
|
||||||
|
config claims stays visible as `unmatched.other`; several names sharing one port produce `null`
|
||||||
|
counts and a `unmatched.groups` line rather than one number repeated on sixteen pipes; and **no join
|
||||||
|
ever asks a resolver** — `socket.getaddrinfo` is spied on and fails the test, because deciding
|
||||||
|
"already an address, or a name?" is the only question that module may ask, and a DNS lookup in the
|
||||||
|
poll loop would be rule 2 with a round trip attached.
|
||||||
|
|
||||||
`tests/rate_window_contract.py` is the same device for one number rather than a field table: the Δ
|
`tests/rate_window_contract.py` is the same device for one number rather than a field table: the Δ
|
||||||
window each source measures its rates over — `RATE_WINDOW_SECONDS` for the host agents,
|
window each source measures its rates over — `RATE_WINDOW_SECONDS` for the host agents,
|
||||||
`LLM_INTERVAL_SECONDS` for the engines, one estimator per measuring cadence (phase 23). It holds
|
`LLM_INTERVAL_SECONDS` for the engines, one estimator per measuring cadence (phase 23). It holds
|
||||||
|
|||||||
+50
-12
@@ -26,7 +26,14 @@ The rules, and the reason each exists:
|
|||||||
operator can see which line was crossed, and the numbers still live in one file.
|
operator can see which line was crossed, and the numbers still live in one file.
|
||||||
* **A dead source is one alert.** Its last good payload is still in the hub, so its stale
|
* **A dead source is one alert.** Its last good payload is still in the hub, so its stale
|
||||||
threshold breaches would pile three more amber lines on top of the red one that already
|
threshold breaches would pile three more amber lines on top of the red one that already
|
||||||
names it. Payload-derived alerts are skipped for a source that is not answering.
|
names it. Payload-derived alerts are skipped for a source that is not answering. A job that
|
||||||
|
dials an agent another job already named (`AGENT_ECHO_DASHBOARDS`) adds no line either.
|
||||||
|
* **No verdict of its own, ever.** Nothing here compares a number to a threshold: the host and
|
||||||
|
LLM rules read `severity` dicts and the UPS rule reads `health.checks[]`. That is why a source
|
||||||
|
whose payload carries no verdicts can only ever alert on reachability — the `gateway` page's
|
||||||
|
connection counts are such a source (`severity.py` rates none, the `net.*` precedent), and a
|
||||||
|
busy proxy is the job rather than a fault. There is no `gateway` payload rule and none is
|
||||||
|
wanted: adding one would put a threshold in this module, which the rule above forbids.
|
||||||
* **Bounded and deduped.** One alert per (source, metric) — the `id` is that pair — and at
|
* **Bounded and deduped.** One alert per (source, metric) — the `id` is that pair — and at
|
||||||
most :data:`MAX_ALERTS` of them, the last being a marker that says how many it hid. A
|
most :data:`MAX_ALERTS` of them, the last being a marker that says how many it hid. A
|
||||||
40-disk fleet must not turn the landing page into a log tail.
|
40-disk fleet must not turn the landing page into a log tail.
|
||||||
@@ -59,7 +66,8 @@ _RANK = {FAIL: 0, WARN: 1}
|
|||||||
#: Where each dashboard lives, so every alert can say where to look. `disks` joined with the
|
#: Where each dashboard lives, so every alert can say where to look. `disks` joined with the
|
||||||
#: `/disks` page itself (phase 09 task 04): before that route existed a disk alert pointed at
|
#: `/disks` page itself (phase 09 task 04): before that route existed a disk alert pointed at
|
||||||
#: `/`, because a link to a page that answers 404 is worse than no link.
|
#: `/`, because a link to a page that answers 404 is worse than no link.
|
||||||
HREF = {"system": "/system", "disks": "/disks", "llm": "/llm", "ups": "/ups"}
|
HREF = {"system": "/system", "disks": "/disks", "gateway": "/gateway", "llm": "/llm",
|
||||||
|
"ups": "/ups"}
|
||||||
DEFAULT_HREF = "/"
|
DEFAULT_HREF = "/"
|
||||||
|
|
||||||
#: A source whose last good value is older than this many of its own poll intervals is
|
#: A source whose last good value is older than this many of its own poll intervals is
|
||||||
@@ -167,20 +175,49 @@ def _interval(settings: Settings, dashboard: str) -> float:
|
|||||||
# wrong interval would amber the landing page every 5 minutes for a source that is
|
# wrong interval would amber the landing page every 5 minutes for a source that is
|
||||||
# behaving exactly as configured.
|
# behaving exactly as configured.
|
||||||
return float(settings.disks_interval)
|
return float(settings.disks_interval)
|
||||||
|
if dashboard == "gateway":
|
||||||
|
# 30 s by default: the pipes are drawn from a 30-second poll, so amber after 90 seconds of
|
||||||
|
# silence means "what you are looking at is two polls old" and nothing else. Judging it by
|
||||||
|
# the 10 s system interval would amber `/` for a source polling exactly as configured.
|
||||||
|
return float(settings.gateway_interval)
|
||||||
return 60.0
|
return 60.0
|
||||||
|
|
||||||
|
|
||||||
def _system_partner_outage(states: Sequence[Mapping[str, Any]], key: str) -> bool:
|
#: Dashboards whose job dials the **same** agent `/snapshot` the `system` job of that host dials.
|
||||||
"""Is the `system` job for the same host down too? (used for `disks` job keys)
|
#: Their failure is the consequence of an agent being down rather than a second fault, so while the
|
||||||
|
#: host's own job is also failing they add no line (`AGENTS.md`: one dead source, one line).
|
||||||
|
#: `gateway` joined in phase 29 for exactly the reason `disks` did in phase 09: the third reader of
|
||||||
|
#: one agent must not turn one outage into three overview rows.
|
||||||
|
AGENT_ECHO_DASHBOARDS = ("disks", "gateway")
|
||||||
|
|
||||||
The disk-health job reads the same agent `/snapshot` the system dashboard does, so its
|
|
||||||
own failure is the *consequence* of an agent being down, never a second fault. The
|
def _partner_host(dashboard: str, key: str, label: str) -> str:
|
||||||
system alert already names the machine once (`AGENTS.md`: one dead source, one line),
|
"""Which host's `system` job reads the same agent as this one — the echo rule's lookup.
|
||||||
so this suppresses the echo and nothing else: a `disks` job that fails while its host
|
|
||||||
answers — SMART wedged inside the agent's snapshot handler, say — is still one alert,
|
Only called for :data:`AGENT_ECHO_DASHBOARDS`, which is why both arms answer a name. The
|
||||||
because that failure has no other voice.
|
`disks:<host>` job carries its host in its key; the `gateway` job is one per deployment (the
|
||||||
|
`nut` shape) and carries its host in its **label**, because `GATEWAY_HOST`'s name is what the
|
||||||
|
alert prints and the key is the single word `gateway`, which says nothing about a machine.
|
||||||
|
"""
|
||||||
|
if dashboard == "gateway":
|
||||||
|
return label
|
||||||
|
return key.split(":", 1)[1] if ":" in key else key
|
||||||
|
|
||||||
|
|
||||||
|
def _system_partner_outage(states: Sequence[Mapping[str, Any]], host: str) -> bool:
|
||||||
|
"""Is the `system` job for that host down too?
|
||||||
|
|
||||||
|
The disk-health and gateway jobs read the same agent `/snapshot` the system dashboard does, so
|
||||||
|
their own failure is the *consequence* of an agent being down, never a second fault. The system
|
||||||
|
alert already names the machine once, so this suppresses the echo and nothing else: a `disks`
|
||||||
|
job that fails while its host answers — SMART wedged inside the agent's snapshot handler, say —
|
||||||
|
is still one alert, because that failure has no other voice. The same is true of a `gateway`
|
||||||
|
job whose host answers but whose proxy sections stopped parsing: only that job can say it.
|
||||||
|
|
||||||
|
A missing partner answers `False`, not `True`: "I cannot find the host's job" is not evidence
|
||||||
|
that the host is fine, and silencing this job on that guess would hide the one fault it has a
|
||||||
|
voice for.
|
||||||
"""
|
"""
|
||||||
host = key.split(":", 1)[1] if ":" in key else key
|
|
||||||
partner = next((state for state in states
|
partner = next((state for state in states
|
||||||
if state.get("dashboard") == "system"
|
if state.get("dashboard") == "system"
|
||||||
and str(state.get("key") or "") == f"host:{host}"), None)
|
and str(state.get("key") or "") == f"host:{host}"), None)
|
||||||
@@ -307,7 +344,8 @@ def _source_alerts(sink: _Sink, settings: Settings, states: Sequence[Mapping[str
|
|||||||
href = HREF.get(dashboard, DEFAULT_HREF)
|
href = HREF.get(dashboard, DEFAULT_HREF)
|
||||||
key = str(state.get("key") or label)
|
key = str(state.get("key") or label)
|
||||||
if not state.get("ok"):
|
if not state.get("ok"):
|
||||||
if dashboard == "disks" and _system_partner_outage(states, key):
|
if dashboard in AGENT_ECHO_DASHBOARDS and _system_partner_outage(
|
||||||
|
states, _partner_host(dashboard, key, label)):
|
||||||
continue # the host agent is already named by its own alert; one line
|
continue # the host agent is already named by its own alert; one line
|
||||||
# the alert starts with the name, so the name inside the message is removed —
|
# the alert starts with the name, so the name inside the message is removed —
|
||||||
# "driveripper: driveripper: ..." reads like two problems instead of one
|
# "driveripper: driveripper: ..." reads like two problems instead of one
|
||||||
|
|||||||
+204
-4
@@ -33,10 +33,12 @@ from .collectors import NutMonitor
|
|||||||
from .config import Settings, strip_userinfo
|
from .config import Settings, strip_userinfo
|
||||||
from .poller import Hub, SourceState
|
from .poller import Hub, SourceState
|
||||||
from .severity import classify
|
from .severity import classify
|
||||||
|
from .sources import gateway as gateway_source
|
||||||
from .sources import nut
|
from .sources import nut
|
||||||
from .sources.disks import DASHBOARD as DISKS_DASHBOARD
|
from .sources.disks import DASHBOARD as DISKS_DASHBOARD
|
||||||
from .sources.disks import SOURCE as DISKS_SOURCE
|
from .sources.disks import SOURCE as DISKS_SOURCE
|
||||||
from .sources.disks import fleet_disk_summary
|
from .sources.disks import fleet_disk_summary
|
||||||
|
from .sources.gateway import DASHBOARD as GATEWAY_DASHBOARD
|
||||||
from .sources.host import fleet_summary
|
from .sources.host import fleet_summary
|
||||||
from .sources.llm import fleet_totals
|
from .sources.llm import fleet_totals
|
||||||
from .store import Event, SampleStore
|
from .store import Event, SampleStore
|
||||||
@@ -196,6 +198,102 @@ def _ups_severity(card: dict[str, Any]) -> dict[str, str | None]:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _gateway_url(settings: Settings) -> str | None:
|
||||||
|
"""The `display_url` of the `HOST_TARGETS` entry `GATEWAY_HOST` names, or `None`.
|
||||||
|
|
||||||
|
`display_url` rather than `url` for the reason every other row on `/api/config` has one: that
|
||||||
|
body goes to the browser, and a token written into a target's userinfo must stop at the backend
|
||||||
|
(`AGENTS.md` invariant 6). `None` when no target carries the name — which is the same fact
|
||||||
|
`enabled: false` states, printed as the URL the page would have dialled.
|
||||||
|
"""
|
||||||
|
return next((target.display_url for target in settings.host_targets
|
||||||
|
if target.name == settings.gateway_host), None)
|
||||||
|
|
||||||
|
|
||||||
|
def _pipe_totals(pipes: Any, counting: bool) -> tuple[int | None, bool]:
|
||||||
|
"""Σ of a pipe list's connection counts, and whether any pipe's count was unknown.
|
||||||
|
|
||||||
|
`counting` is the payload's own answer to "did the kernel's socket table reply". When it did
|
||||||
|
not, every count is `null` and so is the total, with nothing left to call partial: a `0` there
|
||||||
|
would print an idle gateway over a source that never measured one (`AGENTS.md` rule 2). When it
|
||||||
|
did, the sum covers the pipes that carry a number and `partial` says at least one other did not
|
||||||
|
(a port whose configured targets cannot be told apart, `sources/gateway.py`) — so a total is
|
||||||
|
never quietly a subtotal, and `partial` only ever qualifies a total that exists.
|
||||||
|
"""
|
||||||
|
rows = [pipe for pipe in (pipes or []) if isinstance(pipe, Mapping)]
|
||||||
|
if not counting:
|
||||||
|
return None, False
|
||||||
|
values = [pipe.get("connections") for pipe in rows]
|
||||||
|
known = [value for value in values
|
||||||
|
if isinstance(value, int) and not isinstance(value, bool)]
|
||||||
|
return sum(known), len(known) != len(values)
|
||||||
|
|
||||||
|
|
||||||
|
def _gateway_summary(payload: Mapping[str, Any] | None, settings: Settings) -> dict[str, Any]:
|
||||||
|
"""`/api/summary`'s gateway block: the ingress's totals, or a stub of unknowns.
|
||||||
|
|
||||||
|
Additive in phase 29. The page charts from `/api/gateway/snapshot`, which has one entry per
|
||||||
|
pipe; this block is the **tally** a programmatic consumer (and the overview's one line, task
|
||||||
|
05) reads without walking the pipes. Three arms, and they are three different sentences:
|
||||||
|
|
||||||
|
* **unconfigured** (`GATEWAY_HOST` is not a `HOST_TARGETS` name) — `configured: false`, every
|
||||||
|
count `null`, `states: []`. There is no job, so there is nothing to age (invariant 3).
|
||||||
|
* **configured, never answered** — the stub `system.hosts` and `/api/disks` use: the source is
|
||||||
|
in the deployment, it has no numbers yet, and `unreachable: true` says which of the two facts
|
||||||
|
it is. `host` still names the machine, because that is a config fact and not a measurement.
|
||||||
|
* **answering** — the payload's own tallies, summed without inventing anything: `inbound` /
|
||||||
|
`outbound` are the sums of the pipes that have a count, `partial` says one pipe did not, and
|
||||||
|
`map_targets` counts the stream `map`'s routing entries.
|
||||||
|
|
||||||
|
Nothing here carries a `severity`: a connection count is not a graded number (`severity.py`
|
||||||
|
rates none, the `net.*` precedent), and an `unmatched` socket is a report and not a fault.
|
||||||
|
|
||||||
|
`href` is the row's link, from `alerts.py`'s one table of dashboard routes, so the overview's
|
||||||
|
line and an alert about the same source cannot spell the page differently (the rule
|
||||||
|
`test_overview_alert_rows_link_to_the_dashboard_that_explains_them` holds for an alert row).
|
||||||
|
It travels in the unconfigured arm too — a link nobody renders costs nothing, and a block that
|
||||||
|
grew the key only when configured would make the page test for two shapes.
|
||||||
|
"""
|
||||||
|
block: dict[str, Any] = {
|
||||||
|
"configured": settings.gateway_enabled,
|
||||||
|
"host": settings.gateway_host if settings.gateway_enabled else None,
|
||||||
|
"href": alert_rules.HREF[GATEWAY_DASHBOARD],
|
||||||
|
"supported": None,
|
||||||
|
"unreachable": True,
|
||||||
|
"inbound": None,
|
||||||
|
"outbound": None,
|
||||||
|
"map_targets": None,
|
||||||
|
"unmatched": None,
|
||||||
|
# `unmatched` is connections no configured pipe claimed; `unmatched_listeners` is a
|
||||||
|
# *listener* count (a port this host serves that neither config names). Two units, so two
|
||||||
|
# fields: adding them into one number would be an aggregate of unlike things.
|
||||||
|
"unmatched_listeners": None,
|
||||||
|
"partial": False,
|
||||||
|
}
|
||||||
|
if payload is None:
|
||||||
|
return block
|
||||||
|
block["unreachable"] = False
|
||||||
|
block["supported"] = bool(payload.get("supported"))
|
||||||
|
block["host"] = str(payload.get("host") or block["host"] or settings.gateway_host)
|
||||||
|
if not block["supported"]:
|
||||||
|
# A 1.1.0 agent or `--no-proxy`: nothing was read, so "0 map targets" would be a claim
|
||||||
|
# about a config nobody looked at. `supported: false` plus nulls is the honest pair.
|
||||||
|
return block
|
||||||
|
sockets = payload.get("sockets") or {}
|
||||||
|
counting = (isinstance(sockets.get("established"), int)
|
||||||
|
and not isinstance(sockets.get("established"), bool))
|
||||||
|
block["map_targets"] = len((payload.get("nginx") or {}).get("map_targets") or [])
|
||||||
|
inbound, partial_in = _pipe_totals(payload.get("inbound"), counting)
|
||||||
|
outbound, partial_out = _pipe_totals(payload.get("outbound"), counting)
|
||||||
|
block["inbound"], block["outbound"] = inbound, outbound
|
||||||
|
block["partial"] = partial_in or partial_out
|
||||||
|
unmatched = payload.get("unmatched") or {}
|
||||||
|
block["unmatched"] = (int(unmatched.get("inbound") or 0) + int(unmatched.get("outbound") or 0)
|
||||||
|
if counting else None)
|
||||||
|
block["unmatched_listeners"] = unmatched.get("other") if counting else None
|
||||||
|
return block
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# The admin session: login, status, logout
|
# The admin session: login, status, logout
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
@@ -563,6 +661,20 @@ def common_router() -> APIRouter:
|
|||||||
"interval_seconds": settings.disks_interval,
|
"interval_seconds": settings.disks_interval,
|
||||||
"timeout_seconds": settings.disks_timeout,
|
"timeout_seconds": settings.disks_timeout,
|
||||||
},
|
},
|
||||||
|
"gateway": {
|
||||||
|
# The ingress page has no upstream of its own either: it is configured exactly when
|
||||||
|
# the name `GATEWAY_HOST` holds is one of the agents being dialled, so an operator
|
||||||
|
# who monitors no host by that name gets `enabled: false` and a page that says "not
|
||||||
|
# configured" rather than a red panel (invariant 3). `url` is that entry's
|
||||||
|
# `display_url` — the same redaction boundary every other row keeps, because this
|
||||||
|
# route's body goes to the browser and a `HOST_TARGETS` userinfo token must not
|
||||||
|
# travel in it (invariant 6) — and `null` when no target carries the name.
|
||||||
|
"enabled": settings.gateway_enabled,
|
||||||
|
"host": settings.gateway_host,
|
||||||
|
"url": _gateway_url(settings),
|
||||||
|
"interval_seconds": settings.gateway_interval,
|
||||||
|
"timeout_seconds": settings.gateway_timeout,
|
||||||
|
},
|
||||||
"ups": {
|
"ups": {
|
||||||
"enabled": settings.nut_enabled,
|
"enabled": settings.nut_enabled,
|
||||||
"server": f"{settings.nut_host}:{settings.nut_port}" if settings.nut_enabled else None,
|
"server": f"{settings.nut_host}:{settings.nut_port}" if settings.nut_enabled else None,
|
||||||
@@ -619,10 +731,10 @@ def common_router() -> APIRouter:
|
|||||||
Everything else here is for **programmatic use**: this is the cheapest whole-fleet
|
Everything else here is for **programmatic use**: this is the cheapest whole-fleet
|
||||||
probe that exists, so the live tests, an external check and a notifier later read
|
probe that exists, so the live tests, an external check and a notifier later read
|
||||||
`llm.engines`, `ups.ups` and the `states`/`recent_events` no panel paints any more.
|
`llm.engines`, `ups.ups` and the `states`/`recent_events` no panel paints any more.
|
||||||
Keys are additive-only — `alerts` was added in phase 02 and `disks` in phase 09; the
|
Keys are additive-only — `alerts` was added in phase 02, `disks` in phase 09 and `gateway`
|
||||||
`disks` block is for programmatic consumers too (which drives are down to `n/a`, which
|
in phase 29; the `disks` block is for programmatic consumers too (which drives are down to
|
||||||
ones the drive itself called `FAILED`), and nothing was renamed, dropped, or given a new
|
`n/a`, which ones the drive itself called `FAILED`), and nothing was renamed, dropped, or
|
||||||
meaning. `tests/test_api.py` pins each path below against a future cleanup.
|
given a new meaning. `tests/test_api.py` pins each path below against a future cleanup.
|
||||||
"""
|
"""
|
||||||
# A configured job that has never answered gets the same stub `/api/*/snapshot`
|
# A configured job that has never answered gets the same stub `/api/*/snapshot`
|
||||||
# hands out: a host in `HOST_TARGETS` with no payload yet is still in the fleet, and
|
# hands out: a host in `HOST_TARGETS` with no payload yet is still in the fleet, and
|
||||||
@@ -652,6 +764,10 @@ def common_router() -> APIRouter:
|
|||||||
# (`_disk_visible`). `disk_entries` stays unfiltered as the "are there disk jobs at all"
|
# (`_disk_visible`). `disk_entries` stays unfiltered as the "are there disk jobs at all"
|
||||||
# question below, so an all-hidden fleet answers a zero fleet exactly as the snapshot does.
|
# question below, so an all-hidden fleet answers a zero fleet exactly as the snapshot does.
|
||||||
disks_hosts = [entry for entry in disk_entries if _disk_visible(entry)]
|
disks_hosts = [entry for entry in disk_entries if _disk_visible(entry)]
|
||||||
|
# the ingress's last good pipes, or `None` when the source is unconfigured (no job) or has
|
||||||
|
# never answered — `_gateway_summary` turns that second case into the stub
|
||||||
|
gateway_entry = ctx.hub.jobs.get(gateway_source.SOURCE)
|
||||||
|
gateway_payload = gateway_entry.state.payload if gateway_entry is not None else None
|
||||||
ups_payload = None
|
ups_payload = None
|
||||||
nut_job = ctx.hub.jobs.get("nut")
|
nut_job = ctx.hub.jobs.get("nut")
|
||||||
if nut_job and nut_job.state.payload:
|
if nut_job and nut_job.state.payload:
|
||||||
@@ -716,6 +832,12 @@ def common_router() -> APIRouter:
|
|||||||
"unreachable": bool(h.get("unreachable"))}
|
"unreachable": bool(h.get("unreachable"))}
|
||||||
for h in disks_hosts],
|
for h in disks_hosts],
|
||||||
"states": ctx.hub.states(DISKS_DASHBOARD)},
|
"states": ctx.hub.states(DISKS_DASHBOARD)},
|
||||||
|
# additive in phase 29: the ingress's own totals, from the one `gateway` job. Same
|
||||||
|
# three-way shape as every block above — unconfigured, configured-but-never-answered
|
||||||
|
# (a stub of `null`s, never `0`), and answering — and no `severity` anywhere, because
|
||||||
|
# a connection count is not a graded number and an unclaimed socket is a report.
|
||||||
|
"gateway": {**_gateway_summary(gateway_payload, ctx.settings),
|
||||||
|
"states": ctx.hub.states(GATEWAY_DASHBOARD)},
|
||||||
"ups": {"enabled": ctx.settings.nut_enabled,
|
"ups": {"enabled": ctx.settings.nut_enabled,
|
||||||
"simulated": bool(ups_payload and ups_payload.get("simulated")),
|
"simulated": bool(ups_payload and ups_payload.get("simulated")),
|
||||||
"ups": [{"name": card["name"], "overall": card["health"]["overall"],
|
"ups": [{"name": card["name"], "overall": card["health"]["overall"],
|
||||||
@@ -902,6 +1024,84 @@ def disks_router() -> APIRouter:
|
|||||||
return router
|
return router
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Gateway dashboard (the ingress's pipes)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def gateway_router() -> APIRouter:
|
||||||
|
"""`/api/gateway/*` — GETs only, like every other dashboard's surface.
|
||||||
|
|
||||||
|
A fifth data page grows no write (invariant 1's shape): reading two config files and a kernel
|
||||||
|
socket table justifies no mutation, and the app's only route that dials an upstream stays
|
||||||
|
`POST /api/ups/command`. `tests/test_api.py` enumerates the served verbs to keep that true.
|
||||||
|
"""
|
||||||
|
router = APIRouter(prefix="/api/gateway", tags=[GATEWAY_DASHBOARD])
|
||||||
|
|
||||||
|
@router.get("/snapshot")
|
||||||
|
async def snapshot(ctx: AppContext = Depends(get_context)) -> dict[str, Any]:
|
||||||
|
"""The one ingress host's pipes: its configs, its live counts, and what nothing claimed.
|
||||||
|
|
||||||
|
The payload is `sources/gateway.parse`'s, verbatim and unread by this module: `haproxy`/
|
||||||
|
`nginx` are the config the agent read, `inbound` one entry per configured listen port (its
|
||||||
|
domains, its live connection count, up to `MAX_CLIENTS` client addresses), `outbound` one
|
||||||
|
entry per configured target (with `by_port` naming which half of the match is a socket
|
||||||
|
fact and which is the config's), `unmatched` the connections and listeners no config
|
||||||
|
names, and each pipe's own `metric` so a chart asks for the series it *is*. Three arms
|
||||||
|
reach the page and they are three different sentences: `404 {"configured": false}` when
|
||||||
|
`GATEWAY_HOST` names no target (answered below, before anything that could open a
|
||||||
|
socket — invariant 3), `supported: false` for an agent that predates the sections, and
|
||||||
|
`unreachable: true` for a host that has never answered — the stub, whose every count is
|
||||||
|
`null` because an agent that has not spoken has not reported an idle gateway (rule 2).
|
||||||
|
|
||||||
|
Nothing here carries a `severity` and nothing carries a `window_seconds`: a connection
|
||||||
|
count is neither a graded number nor a measurement over a window, so `/gateway` colours
|
||||||
|
nothing and prints no `Δ` (the `net.*` precedent).
|
||||||
|
"""
|
||||||
|
if not ctx.settings.gateway_enabled:
|
||||||
|
# First line of the route, before any client, fetch, or store read: a deployment that
|
||||||
|
# monitors no host by this name has nothing to dial, and a red panel for it would
|
||||||
|
# train the operator to ignore red on the pipes that do exist (invariant 3).
|
||||||
|
raise HTTPException(404, detail={"configured": False,
|
||||||
|
"detail": "no gateway host configured — set "
|
||||||
|
"GATEWAY_HOST to a HOST_TARGETS name"})
|
||||||
|
job = ctx.hub.jobs.get(gateway_source.SOURCE)
|
||||||
|
payload = job.state.payload if job is not None else None
|
||||||
|
if payload is None:
|
||||||
|
# configured and dialling, nothing in yet: the stub keeps the page's row (a dead agent
|
||||||
|
# must not read as "never configured") and every count unknown
|
||||||
|
payload = gateway_source.stub(ctx.settings.gateway_host,
|
||||||
|
ts=job.state.last_attempt if job is not None else None)
|
||||||
|
# `states` last and additive: the payload's own `ts` is when the *agent* took the reading,
|
||||||
|
# which is what the page ages, and a route-level timestamp would overwrite it.
|
||||||
|
return {**payload, "states": ctx.hub.states(GATEWAY_DASHBOARD)}
|
||||||
|
|
||||||
|
@router.get("/history")
|
||||||
|
def history(metrics: str = Query("gateway.in_conns"), hosts: str = Query(""),
|
||||||
|
range: str = Query("24h"), points: int = Query(600, ge=10, le=4000),
|
||||||
|
ctx: AppContext = Depends(get_context)) -> dict[str, Any]:
|
||||||
|
"""The `gateway.in_conns@<port>` / `gateway.out_conns@<host>:<port>` gauges.
|
||||||
|
|
||||||
|
`metrics` are full series names (comma-separated) and an unknown one answers as an
|
||||||
|
**empty** series, not a 404 — the contract every other `history` route has, so a card asks
|
||||||
|
for every pipe it could draw in one request and a target with no history draws an empty
|
||||||
|
plot. `hosts` is comma-separated or empty for the one host this source reads (the name
|
||||||
|
`GATEWAY_HOST` holds, which is also the `host` column the gauges are stored under). The
|
||||||
|
outbound name carries the **port** because two services on one host are two facts — one
|
||||||
|
name for `minecraft…:25565` and `…:25566` would make the second insert overwrite the
|
||||||
|
first. History reads only SQLite, so nothing dials here whichever way the config goes;
|
||||||
|
the answer comes through `_history_batch` → `store.history_many`, the one batched door the
|
||||||
|
other four `history` routes use (phase 14 task 02).
|
||||||
|
"""
|
||||||
|
span = parse_range(range)
|
||||||
|
wanted_metrics = [item.strip() for item in metrics.split(",") if item.strip()]
|
||||||
|
wanted_hosts = [item.strip() for item in hosts.split(",") if item.strip()] or (
|
||||||
|
[ctx.settings.gateway_host] if ctx.settings.gateway_enabled else [])
|
||||||
|
return {"hosts": wanted_hosts, "range_seconds": span,
|
||||||
|
"series": _history_batch(ctx.store, gateway_source.SOURCE, wanted_hosts,
|
||||||
|
wanted_metrics, span, points)}
|
||||||
|
|
||||||
|
return router
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# LLM dashboard
|
# LLM dashboard
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
|
|||||||
+31
-12
@@ -39,6 +39,7 @@ from .api import (
|
|||||||
certs_router,
|
certs_router,
|
||||||
common_router,
|
common_router,
|
||||||
disks_router,
|
disks_router,
|
||||||
|
gateway_router,
|
||||||
llm_router,
|
llm_router,
|
||||||
refusal,
|
refusal,
|
||||||
system_router,
|
system_router,
|
||||||
@@ -52,6 +53,7 @@ from .collectors import (
|
|||||||
UpstreamError,
|
UpstreamError,
|
||||||
agent_tls_map,
|
agent_tls_map,
|
||||||
disks_jobs,
|
disks_jobs,
|
||||||
|
gateway_job,
|
||||||
host_jobs,
|
host_jobs,
|
||||||
llm_jobs,
|
llm_jobs,
|
||||||
nut_job,
|
nut_job,
|
||||||
@@ -69,13 +71,18 @@ WEB = Path(__file__).resolve().parent / "web"
|
|||||||
# `/settings` is a page and not a dashboard: it is in `PAGES` (so it is a route) and *not* in
|
# `/settings` is a page and not a dashboard: it is in `PAGES` (so it is a route) and *not* in
|
||||||
# `PUBLIC_PATHS` (so the gate guards it with no extra code — the route-enumeration test in
|
# `PUBLIC_PATHS` (so the gate guards it with no extra code — the route-enumeration test in
|
||||||
# `tests/test_auth_api.py` proves that, and `tests/test_api.py` proves the anonymous refusal for
|
# `tests/test_auth_api.py` proves that, and `tests/test_api.py` proves the anonymous refusal for
|
||||||
# this page specifically). It carries no `aria-current` tab because it is not one of the five
|
# this page specifically). It carries no `aria-current` tab because it is not one of the six
|
||||||
# dashboards (`/`, `/system`, `/disks`, `/llm`, `/ups`).
|
# dashboards (`/`, `/system`, `/disks`, `/llm`, `/ups`, `/gateway`).
|
||||||
# `/certs` (phase 15 task 05) is the second of those admin pages, for the same reason: the
|
# `/certs` (phase 15 task 05) is the second of those admin pages, for the same reason: the
|
||||||
# certificate authority is not a fifth dashboard, it is the thing you administer past them.
|
# certificate authority is not a dashboard, it is the thing you administer past them.
|
||||||
|
# `/gateway` (phase 29) is the sixth dashboard, so it is in the tab nav with `aria-current` of its
|
||||||
|
# own (task 05 — the pinned tab audit in `tests/test_api.py` walks this dict's six dashboards).
|
||||||
|
# Being in this dict is what puts it behind the session gate — `PUBLIC_PATHS` is untouched, and
|
||||||
|
# `tests/test_auth_api.py` proves the refusal from the served route table rather than from a list
|
||||||
|
# written here.
|
||||||
PAGES = {"/": "index.html", "/system": "system.html", "/disks": "disks.html",
|
PAGES = {"/": "index.html", "/system": "system.html", "/disks": "disks.html",
|
||||||
"/llm": "llm.html", "/ups": "ups.html", "/login": "login.html",
|
"/llm": "llm.html", "/ups": "ups.html", "/gateway": "gateway.html",
|
||||||
"/settings": "settings.html", "/certs": "certs.html"}
|
"/login": "login.html", "/settings": "settings.html", "/certs": "certs.html"}
|
||||||
VERSION = "0.1.0"
|
VERSION = "0.1.0"
|
||||||
|
|
||||||
# Everything else requires a signed-in admin. This list is the whole attack surface, so
|
# Everything else requires a signed-in admin. This list is the whole attack surface, so
|
||||||
@@ -110,14 +117,15 @@ def build_hub(settings: Settings, store: SampleStore, client: httpx.AsyncClient,
|
|||||||
polling in tests and not in the container.
|
polling in tests and not in the container.
|
||||||
|
|
||||||
`tls` is the per-host certificate material (`agent_tls_map`, built from `build_certs`), and
|
`tls` is the per-host certificate material (`agent_tls_map`, built from `build_certs`), and
|
||||||
it is a parameter rather than something this function derives because the two `host:`/`disks:`
|
it is a parameter rather than something this function derives because the `host:`, `disks:` and
|
||||||
jobs for one host must hold **the same** entry: the same identity on the wire and the same
|
`gateway` jobs of one host must hold **the same** entry: the same identity on the wire and the
|
||||||
revocation answer, so a revoked agent fails both of its sources with one sentence and neither
|
same revocation answer, so a revoked agent fails every source that reads it with one sentence
|
||||||
of them opens a socket.
|
and none of them opens a socket.
|
||||||
|
|
||||||
Order is host → disks → llm → ups so far only for readability: two jobs carry the same
|
Order is host → disks → gateway → llm → ups so far only for readability: three jobs can carry
|
||||||
host label (`host:gw`, `disks:gw`), so anything that looks a job up by label alone gets
|
the same host label (`host:gw`, `disks:gw`, and `gateway` for the ingress), so anything that
|
||||||
the system record — `alerts.py` filters by dashboard instead and does not depend on it.
|
looks a job up by label alone gets whichever registered first — `alerts.py` filters by
|
||||||
|
dashboard instead and does not depend on it.
|
||||||
|
|
||||||
There are **two** rate estimators, one per measuring cadence (phase 23). A window is only
|
There are **two** rate estimators, one per measuring cadence (phase 23). A window is only
|
||||||
a claim about the polls that fill it: the `host` jobs run every `HOST_INTERVAL_SECONDS`
|
a claim about the polls that fill it: the `host` jobs run every `HOST_INTERVAL_SECONDS`
|
||||||
@@ -130,6 +138,10 @@ def build_hub(settings: Settings, store: SampleStore, client: httpx.AsyncClient,
|
|||||||
phase-13 gates (the `MIN_WINDOW_SECONDS` floor, the ceiling, the 8x jump confirmation) are
|
phase-13 gates (the `MIN_WINDOW_SECONDS` floor, the ceiling, the 8x jump confirmation) are
|
||||||
the estimator's own and therefore apply to the smaller window unchanged; the `Δ<window>`
|
the estimator's own and therefore apply to the smaller window unchanged; the `Δ<window>`
|
||||||
label always prints the window actually used, so the smaller claim is legible.
|
label always prints the window actually used, so the smaller claim is legible.
|
||||||
|
|
||||||
|
Neither estimator reaches the `gateway` job. Its numbers are counts read at one instant, and a
|
||||||
|
count that goes through a Δ-window becomes a rate with a window label and no measurement
|
||||||
|
behind it (`sources/gateway.py`, AGENTS.md rule 2), so that job is handed no estimator at all.
|
||||||
"""
|
"""
|
||||||
host_rates = RateEstimator(span=settings.rate_window)
|
host_rates = RateEstimator(span=settings.rate_window)
|
||||||
llm_rates = RateEstimator(span=settings.llm_interval)
|
llm_rates = RateEstimator(span=settings.llm_interval)
|
||||||
@@ -140,6 +152,12 @@ def build_hub(settings: Settings, store: SampleStore, client: httpx.AsyncClient,
|
|||||||
for job in disks_jobs(list(settings.host_targets), client, DisksCollector(),
|
for job in disks_jobs(list(settings.host_targets), client, DisksCollector(),
|
||||||
settings.disks_interval, timeout=settings.disks_timeout, tls=tls):
|
settings.disks_interval, timeout=settings.disks_timeout, tls=tls):
|
||||||
hub.add(job)
|
hub.add(job)
|
||||||
|
# A third reader of the same `/snapshot`, for the one host `GATEWAY_HOST` names (phase 29).
|
||||||
|
# `None` when that name is not a target: the source is unconfigured rather than red, so it
|
||||||
|
# gets no task at all (the `disks` gate, invariant 3) and `/api/gateway/*` 404s without a dial.
|
||||||
|
gateway = gateway_job(settings, client, tls=tls)
|
||||||
|
if gateway is not None:
|
||||||
|
hub.add(gateway)
|
||||||
for job in llm_jobs(settings, client, llm_rates):
|
for job in llm_jobs(settings, client, llm_rates):
|
||||||
hub.add(job)
|
hub.add(job)
|
||||||
nut_monitor: NutMonitor | None = None
|
nut_monitor: NutMonitor | None = None
|
||||||
@@ -359,6 +377,7 @@ def create_app(settings: Settings | None = None, store: SampleStore | None = Non
|
|||||||
app.include_router(common_router())
|
app.include_router(common_router())
|
||||||
app.include_router(system_router())
|
app.include_router(system_router())
|
||||||
app.include_router(disks_router())
|
app.include_router(disks_router())
|
||||||
|
app.include_router(gateway_router())
|
||||||
app.include_router(llm_router())
|
app.include_router(llm_router())
|
||||||
app.include_router(ups_router())
|
app.include_router(ups_router())
|
||||||
|
|
||||||
|
|||||||
@@ -17,10 +17,11 @@ from typing import Any
|
|||||||
import httpx
|
import httpx
|
||||||
|
|
||||||
from . import certs
|
from . import certs
|
||||||
from .config import Target, strip_userinfo
|
from .config import Settings, Target, strip_userinfo
|
||||||
from .poller import Job, Result
|
from .poller import Job, Result
|
||||||
from .rates import RateEstimator
|
from .rates import RateEstimator
|
||||||
from .sources import disks as disks_source
|
from .sources import disks as disks_source
|
||||||
|
from .sources import gateway as gateway_source
|
||||||
from .sources import llm as llm_source
|
from .sources import llm as llm_source
|
||||||
from .sources import nut
|
from .sources import nut
|
||||||
from .sources.host import HostCollector, parse_host_snapshot
|
from .sources.host import HostCollector, parse_host_snapshot
|
||||||
@@ -273,6 +274,25 @@ async def fetch_disk_snapshot(client: httpx.AsyncClient, target: Target,
|
|||||||
return disks_source.parse_disks_snapshot(target.name, body, disks_collector)
|
return disks_source.parse_disks_snapshot(target.name, body, disks_collector)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Gateway topology (the /gateway dashboard)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
async def fetch_gateway_snapshot(client: httpx.AsyncClient, target: Target,
|
||||||
|
*, timeout: float | None = None,
|
||||||
|
tls: AgentTls | None = None) -> Result:
|
||||||
|
"""The same `/snapshot`, projected onto the connection topology of one ingress host.
|
||||||
|
|
||||||
|
A third read of an endpoint `system` already polls, for the same reason `disks` is a second
|
||||||
|
one: the cadences differ (a connection count is a 30 s shape, utilisation a 10 s one) and the
|
||||||
|
page has to age on its own instead of blinking `/system` (invariant 4). The fetch itself is
|
||||||
|
the plain one — the snapshot handler does no extra work for these sections beyond reading two
|
||||||
|
configs and two `/proc` tables — so `timeout` is headroom for a handler that may be mid-`smartctl`
|
||||||
|
pass for the other page, not for this one.
|
||||||
|
"""
|
||||||
|
body = await fetch_snapshot(client, target, timeout=timeout, tls=tls)
|
||||||
|
return gateway_source.parse_gateway_snapshot(target.name, body)
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# LLM engines
|
# LLM engines
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
@@ -493,6 +513,33 @@ def disks_jobs(targets: list[Target], client: httpx.AsyncClient,
|
|||||||
]
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def gateway_job(settings: Settings, client: httpx.AsyncClient, *,
|
||||||
|
tls: Mapping[str, AgentTls] | None = None) -> Job | None:
|
||||||
|
"""The **one** `gateway` job, for the host `GATEWAY_HOST` names — or `None`.
|
||||||
|
|
||||||
|
One job rather than one per host (the `nut` shape): the page is about the single ingress, and
|
||||||
|
a job per host would put a `gateway` record against every machine in the fleet for a question
|
||||||
|
only one of them answers. `label` is the host name, because that is what an alert and an event
|
||||||
|
print, and it is what `alerts.py` uses to find the `host:<name>` job dialling the same agent
|
||||||
|
(the echo rule: a dead agent is one line, not three).
|
||||||
|
|
||||||
|
`None` when the name is not in `HOST_TARGETS` — the `disks` job's configured gate, invariant
|
||||||
|
3: an unconfigured source gets **no task at all**, so `/api/health` carries no red record for a
|
||||||
|
page nobody installed and nothing dials. `tls` is the same per-host entry the `host:`/`disks:`
|
||||||
|
jobs of that host hold, so revoking the gateway's certificate fails all three with one sentence
|
||||||
|
and none of them opens a socket.
|
||||||
|
"""
|
||||||
|
target = next((item for item in settings.host_targets
|
||||||
|
if item.name == settings.gateway_host), None)
|
||||||
|
if target is None:
|
||||||
|
return None
|
||||||
|
entry = (tls or {}).get(target.name)
|
||||||
|
return Job(gateway_source.SOURCE, gateway_source.DASHBOARD, target.name,
|
||||||
|
settings.gateway_interval,
|
||||||
|
lambda: fetch_gateway_snapshot(client, target,
|
||||||
|
timeout=settings.gateway_timeout, tls=entry))
|
||||||
|
|
||||||
|
|
||||||
def llm_jobs(settings: Any, client: httpx.AsyncClient, rates: RateEstimator) -> list[Job]:
|
def llm_jobs(settings: Any, client: httpx.AsyncClient, rates: RateEstimator) -> list[Job]:
|
||||||
jobs: list[Job] = []
|
jobs: list[Job] = []
|
||||||
if settings.gufo_url:
|
if settings.gufo_url:
|
||||||
|
|||||||
@@ -189,6 +189,23 @@ class Settings:
|
|||||||
disks_interval: float = _num("DISKS_INTERVAL_SECONDS", 300)
|
disks_interval: float = _num("DISKS_INTERVAL_SECONDS", 300)
|
||||||
disks_timeout: float = _num("DISKS_TIMEOUT_SECONDS", 30)
|
disks_timeout: float = _num("DISKS_TIMEOUT_SECONDS", 30)
|
||||||
|
|
||||||
|
# --- gateway dashboard -------------------------------------------------------
|
||||||
|
# Which `HOST_TARGETS` name is the ingress. It has no URL of its own: the `gateway` job
|
||||||
|
# dials the same `/snapshot` the `host:<name>` job dials, on its own cadence and with its
|
||||||
|
# own health record (the `disks` precedent) — so "not configured" is a *name* that is not
|
||||||
|
# in the target list, and the page 404s without dialling anything (invariant 3).
|
||||||
|
# The default is the fleet's real name, so an operator who monitors a host called `gateway`
|
||||||
|
# gets the page by installing agent 1.2.0 on it; anyone else sets this or leaves the
|
||||||
|
# host_targets list without that name.
|
||||||
|
gateway_host: str = _env("GATEWAY_HOST", "gateway")
|
||||||
|
# 30 s: a connection count is neither a per-second signal nor a drive's SMART — it is the
|
||||||
|
# shape of traffic over half a minute, which is what the pipes animate. The timeout is the
|
||||||
|
# plain agent fetch's (the snapshot handler does no work for these sections beyond reading
|
||||||
|
# two configs and two /proc tables), with headroom because the same handler may be mid-
|
||||||
|
# smartctl pass for a different page.
|
||||||
|
gateway_interval: float = _num("GATEWAY_INTERVAL_SECONDS", 30)
|
||||||
|
gateway_timeout: float = _num("GATEWAY_TIMEOUT_SECONDS", 10)
|
||||||
|
|
||||||
# --- llm dashboard ------------------------------------------------------------
|
# --- llm dashboard ------------------------------------------------------------
|
||||||
gufo_url: str = _base_url("GUFO_BASE_URL")
|
gufo_url: str = _base_url("GUFO_BASE_URL")
|
||||||
gufo_api_key: str = _env("GUFO_API_KEY")
|
gufo_api_key: str = _env("GUFO_API_KEY")
|
||||||
@@ -229,6 +246,17 @@ class Settings:
|
|||||||
def nut_enabled(self) -> bool:
|
def nut_enabled(self) -> bool:
|
||||||
return bool(self.nut_host)
|
return bool(self.nut_host)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def gateway_enabled(self) -> bool:
|
||||||
|
"""Is `GATEWAY_HOST` one of the hosts this deployment dials?
|
||||||
|
|
||||||
|
The `nut_enabled` shape: a name with no target behind it is *unconfigured*, not broken,
|
||||||
|
so `/gateway` says "not configured" and `/api/gateway/*` answers 404 without a dial.
|
||||||
|
Case-sensitive on purpose — `HOST_TARGETS` names key the history DB and `parse_targets`
|
||||||
|
restricts them to lowercase, so a capitalised setting is a typo and not a match.
|
||||||
|
"""
|
||||||
|
return self.gateway_host in {target.name for target in self.host_targets}
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def actions_allowed(self) -> bool:
|
def actions_allowed(self) -> bool:
|
||||||
"""Coarse switch. Whether a *specific* command can run also depends on the user
|
"""Coarse switch. Whether a *specific* command can run also depends on the user
|
||||||
@@ -239,7 +267,8 @@ class Settings:
|
|||||||
|
|
||||||
@property
|
@property
|
||||||
def any_source_enabled(self) -> bool:
|
def any_source_enabled(self) -> bool:
|
||||||
return bool(self.host_targets) or bool(self.gufo_url or self.vllm_url) or self.nut_enabled
|
return (bool(self.host_targets) or bool(self.gufo_url or self.vllm_url)
|
||||||
|
or self.nut_enabled or self.gateway_enabled)
|
||||||
|
|
||||||
|
|
||||||
def load() -> Settings:
|
def load() -> Settings:
|
||||||
|
|||||||
@@ -41,8 +41,8 @@ class Result:
|
|||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class SourceState:
|
class SourceState:
|
||||||
key: str # "host:driveripper", "disks:driveripper", "llm:gufo", "nut"
|
key: str # "host:driveripper", "disks:driveripper", "gateway", "llm:gufo", "nut"
|
||||||
dashboard: str # "system" | "disks" | "llm" | "ups"
|
dashboard: str # "system" | "disks" | "gateway" | "llm" | "ups"
|
||||||
label: str
|
label: str
|
||||||
configured: bool = True
|
configured: bool = True
|
||||||
ok: bool = False
|
ok: bool = False
|
||||||
|
|||||||
@@ -0,0 +1,764 @@
|
|||||||
|
"""Gateway source: the connection topology of the host `GATEWAY_HOST` names.
|
||||||
|
|
||||||
|
`/system` answers "what is this machine doing". This answers the one question the router
|
||||||
|
exists for: **what is coming in through it and where does it go out**. It reads the three
|
||||||
|
phase-28 sections of one agent `/snapshot` — `haproxy` (frontends + binds, backends +
|
||||||
|
servers), `nginx` (server blocks with `listen`/`server_name`/`proxy_pass`, `map` blocks) and
|
||||||
|
`sockets` (`/proc/net/tcp{,6}`'s LISTEN + ESTABLISHED) — and joins them: the **inbound**
|
||||||
|
pipes (one per configured listen port, with the domains that port serves and the live client
|
||||||
|
addresses) and the **outbound** pipes (one per configured target, with its live count).
|
||||||
|
|
||||||
|
What this module owns, and what it deliberately does not:
|
||||||
|
|
||||||
|
* **No measurement.** Every number here is a count read at one instant: there is no
|
||||||
|
`RateEstimator`, no `window_seconds`, no `Δ<window>` label (AGENTS.md rule 2 — a count is a
|
||||||
|
count). The gauges are `gateway.in_conns@<port>` and `gateway.out_conns@<host>:<port>`,
|
||||||
|
under their own `gateway` sample family so `GET /api/metrics?source=gateway` answers what
|
||||||
|
this page has history for and a 30-second connection count never mixes with the 10-second
|
||||||
|
system ones. They roll up raw → 1 m → 1 h like every other gauge (invariant 9).
|
||||||
|
* **No verdict.** `severity.py` rates no connection count and this payload carries no
|
||||||
|
`severity` dict on one — the `net.*` precedent: 400 connections on a reverse proxy is the
|
||||||
|
job, and 0 on an idle one is healthy. Pages colour nothing here.
|
||||||
|
* **No DNS, ever.** The agent never resolves a name and neither does this module
|
||||||
|
(`tests/test_gateway_source.py` spies `socket.getaddrinfo` and pins that it is not called
|
||||||
|
once). That single fact decides the whole outbound join, below.
|
||||||
|
|
||||||
|
The join, and where its honesty lives
|
||||||
|
------------------------------------
|
||||||
|
|
||||||
|
**Inbound** is unambiguous: a socket's **local port** is the port the config listens on, and
|
||||||
|
its **remote address** is the client. A socket whose local port no configured frontend names
|
||||||
|
is counted in `unmatched.inbound` — reported, never dropped, because "the gateway is carrying
|
||||||
|
traffic on a port no config mentions" is exactly the fact the operator needs.
|
||||||
|
|
||||||
|
**Outbound** cannot be unambiguous, and pretends otherwise nowhere. The config names targets
|
||||||
|
as **domains** (`nextcloud.reeselink.com:11000`) while `/proc` reports **addresses**
|
||||||
|
(`10.1.1.164:11000`), and resolving one to the other is the one thing this stack never does.
|
||||||
|
So a socket is attributed to at most one pipe, in this order:
|
||||||
|
|
||||||
|
1. an **exact** `(address, port)` match against a target written as an IP literal
|
||||||
|
(`by_port: false`) — a real, unambiguous join;
|
||||||
|
2. a **port-only** match against a target written as a **name**, and only when that port
|
||||||
|
belongs to exactly **one** configured host — the socket fact is real, the host is the
|
||||||
|
config's, and the pipe carries `by_port: true` to say which half is which. A target written
|
||||||
|
as an address never reaches this arm: a miss on the exact match above is a different
|
||||||
|
endpoint, not this one;
|
||||||
|
3. a port shared by **several** configured targets — the case the real `stream.conf` map is
|
||||||
|
(sixteen `*.reeselink.com:443` entries). Choosing one of them would be rule 2 with a
|
||||||
|
pipe's clothes on, so nothing is chosen: the socket lands in `unmatched.outbound` and the
|
||||||
|
group is reported once in `unmatched.groups` as `{port, connections, hosts, host_count}`,
|
||||||
|
which is the honest sentence — "N connections on port P, whose configured targets are
|
||||||
|
these M names, unresolved". Those pipes carry `connections: null`, not a share and not a
|
||||||
|
repeat of the port's total (a number drawn on sixteen pipes reads as sixteen times the
|
||||||
|
traffic).
|
||||||
|
|
||||||
|
A target that names **no port** (`server x host`) is never joinable at all, so its count is
|
||||||
|
`null` even when the socket table answered: `0` there would claim a measurement nobody took.
|
||||||
|
|
||||||
|
A pipe's count is therefore `null` — `n/a`, never `0` — whenever the socket table did not
|
||||||
|
answer or the join could not decide, and `0` only when the sockets did answer and this pipe
|
||||||
|
genuinely has none.
|
||||||
|
|
||||||
|
The `$var` → `map` fan-out
|
||||||
|
--------------------------
|
||||||
|
|
||||||
|
A stream `proxy_pass $targetBackend` has no target in its own right (nginx picks one per
|
||||||
|
connection from the map), so it contributes **no literal pipe**. The map whose `variable`
|
||||||
|
matches is what carries the fan-out: its entries become `nginx.map_targets`
|
||||||
|
(`{key: the SNI domain, host, port}`) *and* one outbound pipe per entry, so the page can draw
|
||||||
|
sixteen domains leaving the gateway and label each with the domain that reaches it. A map no
|
||||||
|
`proxy_pass` references (`$connection_upgrade`, whose values are the header words
|
||||||
|
`upgrade`/`close`) contributes nothing: it is not a routing table.
|
||||||
|
|
||||||
|
Series naming
|
||||||
|
-------------
|
||||||
|
|
||||||
|
Outbound gauges are keyed `gateway.out_conns@<host>:<port>`, with the port in the name. The
|
||||||
|
phase's first sketch keyed them on the host alone, and the live gateway proves that cannot
|
||||||
|
work: its two haproxy Minecraft backends are `minecraft.reeselink.com:25565` and
|
||||||
|
`minecraft.reeselink.com:25566`, two pipes whose counts are different facts about different
|
||||||
|
sockets. One name for both is one `INSERT OR REPLACE` overwriting the other in silence —
|
||||||
|
the aggregation failure rule 2 exists to catch. The port is also what makes the name stable
|
||||||
|
when the same host later gains or loses a second service.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import ipaddress
|
||||||
|
import time
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from ..poller import Result
|
||||||
|
from ..store import Sample
|
||||||
|
|
||||||
|
#: The sample family and the dashboard name — one word on purpose, so
|
||||||
|
#: `/api/events?source=gateway` and `hub.states("gateway")` agree (the phase-03 lesson).
|
||||||
|
SOURCE = "gateway"
|
||||||
|
DASHBOARD = "gateway"
|
||||||
|
|
||||||
|
#: The three phase-28 agent sections this projection reads. Their **presence** is what
|
||||||
|
#: distinguishes "this agent knows about gateway topology" from a 1.1.0 agent or one started
|
||||||
|
#: with `--no-proxy` (both of which publish none of the three keys), and that distinction is
|
||||||
|
#: the difference between "no pipes configured" and "this host cannot answer".
|
||||||
|
SECTIONS = ("haproxy", "nginx", "sockets")
|
||||||
|
|
||||||
|
#: The agent's own error-line prefixes for those sections. `collectors.ADDON_ERROR_PREFIXES`
|
||||||
|
#: keeps the same lines out of `/system`'s `degraded`; here they are the payload's `why`,
|
||||||
|
#: which is why they are carried through verbatim rather than paraphrased.
|
||||||
|
SECTION_ERROR_PREFIXES = ("haproxy", "nginx", "sockets")
|
||||||
|
|
||||||
|
#: How many distinct client addresses one inbound pipe carries. `connections` stays the true
|
||||||
|
#: socket count; the list is what a page lists beside the pipe, and a DDoS minute must not
|
||||||
|
#: turn the payload into a client dump.
|
||||||
|
MAX_CLIENTS = 32
|
||||||
|
|
||||||
|
#: Same bound for one ambiguous port group's candidate host list (the `stream.conf` map's 16
|
||||||
|
#: entries fit; `host_count` is the real total either way).
|
||||||
|
MAX_GROUP_HOSTS = 20
|
||||||
|
|
||||||
|
#: The sentence the unsupported arm prints. Two operator facts in one line — the agent is too
|
||||||
|
#: old, or it was told not to look — because both are fixed by reinstalling the agent, one
|
||||||
|
#: with and once without `--no-proxy`.
|
||||||
|
WHY_PREDATES = ("this agent reports no proxy or socket sections — it predates gateway support "
|
||||||
|
"(1.2.0) or runs with --no-proxy")
|
||||||
|
|
||||||
|
#: The sentence the never-answered arm prints, and the reason it is **not** `WHY_PREDATES`: an
|
||||||
|
#: agent that has not replied has not said it has no sections, and "install 1.2.0" is advice to
|
||||||
|
#: give a host that may already be running it (invariant 3 — the two facts stay two sentences).
|
||||||
|
WHY_NO_ANSWER = "the agent has not answered yet — no snapshot, so no config and no sockets"
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# the small readers: nothing here interprets, everything here tolerates
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def _text(value: Any) -> str | None:
|
||||||
|
"""A non-empty string, else ``None``."""
|
||||||
|
if isinstance(value, str) and value.strip():
|
||||||
|
return value.strip()
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _port(value: Any) -> int | None:
|
||||||
|
"""A port number, or ``None``.
|
||||||
|
|
||||||
|
The agent publishes `null` for a bind or server that names no port and never a `0`, so a
|
||||||
|
non-number here is drift and gets the same answer. `0` is rejected rather than treated as
|
||||||
|
a port: "some port" and "port zero" are different claims, and joining on the latter would
|
||||||
|
match nothing real while looking like a fact.
|
||||||
|
"""
|
||||||
|
if isinstance(value, bool) or not isinstance(value, int | float):
|
||||||
|
return None
|
||||||
|
number = int(value)
|
||||||
|
return number if number > 0 else None
|
||||||
|
|
||||||
|
|
||||||
|
def _is_ip_literal(value: str | None) -> bool:
|
||||||
|
"""Is this target host already an address? `ipaddress` parses text and never dials.
|
||||||
|
|
||||||
|
That is the whole reason it is used instead of a resolver: the outbound join has to know
|
||||||
|
whether a name still *needs* resolving, and a module that asks the question must not
|
||||||
|
answer it (invariant 7's "never guess" applied to a join).
|
||||||
|
"""
|
||||||
|
if not value:
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
ipaddress.ip_address(value)
|
||||||
|
except ValueError:
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _mappings(value: Any) -> list[Mapping[str, Any]]:
|
||||||
|
"""The list of mappings at `value`, ignoring anything that is not one."""
|
||||||
|
if not isinstance(value, Sequence) or isinstance(value, str | bytes):
|
||||||
|
return []
|
||||||
|
return [item for item in value if isinstance(item, Mapping)]
|
||||||
|
|
||||||
|
|
||||||
|
def _section(snapshot: Mapping[str, Any], key: str) -> Mapping[str, Any] | None:
|
||||||
|
"""One of the three sections, or ``None`` when the agent did not publish it."""
|
||||||
|
value = snapshot.get(key)
|
||||||
|
return value if isinstance(value, Mapping) else None
|
||||||
|
|
||||||
|
|
||||||
|
def _available(section: Mapping[str, Any] | None) -> bool | None:
|
||||||
|
"""A section's own three-way `available`, or ``None`` when there was no section at all.
|
||||||
|
|
||||||
|
`False` (haproxy/nginx: no file at that path) and `None` (it could not be read, or the
|
||||||
|
sockets tables did not answer) are both invariants-3 arms: the host is not broken, this
|
||||||
|
one source of information is not there.
|
||||||
|
"""
|
||||||
|
if section is None:
|
||||||
|
return None
|
||||||
|
flag = section.get("available")
|
||||||
|
return flag if isinstance(flag, bool) else None
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# support / degradation: the arms that must never read as red
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def supported(snapshot: Mapping[str, Any]) -> bool:
|
||||||
|
"""Does this agent publish the proxy/socket sections at all?
|
||||||
|
|
||||||
|
Presence of **any** one of the three counts, so a hypothetical agent that publishes
|
||||||
|
`sockets` but not the configs is "supported, with a half missing" (which `sections` and
|
||||||
|
`why` then name) rather than "predates support". A 1.1.0 agent and a `--no-proxy` agent
|
||||||
|
both publish none, which is the single arm this answers `False` for.
|
||||||
|
"""
|
||||||
|
return any(key in snapshot for key in SECTIONS)
|
||||||
|
|
||||||
|
|
||||||
|
def section_status(snapshot: Mapping[str, Any]) -> dict[str, bool | None]:
|
||||||
|
"""One entry per section: the agent's `available`, or `None` when it published neither."""
|
||||||
|
return {key: _available(_section(snapshot, key)) for key in SECTIONS}
|
||||||
|
|
||||||
|
|
||||||
|
def section_errors(snapshot: Mapping[str, Any]) -> list[str]:
|
||||||
|
"""The agent's own section lines out of `errors` (verbatim, the `smart_errors` idiom)."""
|
||||||
|
errors = snapshot.get("errors")
|
||||||
|
lines: list[str] = []
|
||||||
|
for item in errors if isinstance(errors, Sequence) and not isinstance(errors, str | bytes) else ():
|
||||||
|
text = str(item)
|
||||||
|
head = text.strip().lower()
|
||||||
|
if any(head.startswith(f"{prefix}:") for prefix in SECTION_ERROR_PREFIXES):
|
||||||
|
lines.append(text)
|
||||||
|
return lines
|
||||||
|
|
||||||
|
|
||||||
|
def why_lines(snapshot: Mapping[str, Any]) -> list[str]:
|
||||||
|
"""Why this page has no pipes, one sentence per section that did not answer.
|
||||||
|
|
||||||
|
The unsupported arm gets one sentence (`WHY_PREDATES`) and nothing else — there is no
|
||||||
|
section-level detail to give when the agent has no idea the feature exists. Otherwise a
|
||||||
|
line names the section and its arm, in the agent's vocabulary of arms, and never a verdict
|
||||||
|
about the host: an absent haproxy config is "this host runs no haproxy", which is
|
||||||
|
invariant 3's unconfigured case printed in grey, not a fault.
|
||||||
|
"""
|
||||||
|
if not supported(snapshot):
|
||||||
|
return [WHY_PREDATES]
|
||||||
|
lines: list[str] = []
|
||||||
|
for key in SECTIONS:
|
||||||
|
flag = _available(_section(snapshot, key))
|
||||||
|
if flag is True:
|
||||||
|
continue
|
||||||
|
if _section(snapshot, key) is None:
|
||||||
|
lines.append(f"{key}: this agent published no {key} section")
|
||||||
|
elif flag is False:
|
||||||
|
lines.append(f"{key}: no config file at the path the agent reads — this host runs "
|
||||||
|
f"no {key} (or its config lives elsewhere)")
|
||||||
|
elif key == "sockets":
|
||||||
|
lines.append("sockets: neither /proc/net/tcp nor /proc/net/tcp6 answered")
|
||||||
|
else:
|
||||||
|
lines.append(f"{key}: its config is there but could not be read or parsed")
|
||||||
|
return lines
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# the config halves: haproxy, nginx, and the sockets table
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def _names(value: Any) -> list[str]:
|
||||||
|
"""The non-empty strings of a list, in order, deduplicated — backend and server names."""
|
||||||
|
if not isinstance(value, Sequence) or isinstance(value, str | bytes):
|
||||||
|
return []
|
||||||
|
found: list[str] = []
|
||||||
|
for item in value:
|
||||||
|
text = _text(item)
|
||||||
|
if text and text not in found:
|
||||||
|
found.append(text)
|
||||||
|
return found
|
||||||
|
|
||||||
|
|
||||||
|
def haproxy_frontends(section: Mapping[str, Any] | None) -> list[dict[str, Any]]:
|
||||||
|
"""The agent's frontend rows, projected to `{name, port, backend, routes}`.
|
||||||
|
|
||||||
|
One row **per bind port** — a frontend with `bind :::80` and `bind :::443` is two pipes,
|
||||||
|
because a pipe is a port. `backend` is the frontend's own `default_backend`; `routes`
|
||||||
|
names every backend it can reach (`default_backend` plus every `use_backend` name),
|
||||||
|
because a config that routes by a fetch expression picks per request and reporting only
|
||||||
|
the default would be wrong for it. A bind with no port (a unix socket) yields no row:
|
||||||
|
there is no port to draw or join, and inventing one is rule 2.
|
||||||
|
"""
|
||||||
|
rows: list[dict[str, Any]] = []
|
||||||
|
for frontend in _mappings((section or {}).get("frontends")):
|
||||||
|
name = _text(frontend.get("name"))
|
||||||
|
default = _text(frontend.get("default_backend"))
|
||||||
|
# a frontend's `backends` is a list of **names** (`default_backend` plus every
|
||||||
|
# `use_backend`, with a fetch expression dropped by the agent because it names nothing),
|
||||||
|
# so the join downstream stays name → backend block and interprets no rule.
|
||||||
|
routes = _names(frontend.get("backends"))
|
||||||
|
if default and default not in routes:
|
||||||
|
routes.insert(0, default)
|
||||||
|
for bind in _mappings(frontend.get("binds")):
|
||||||
|
port = _port(bind.get("port"))
|
||||||
|
if port is None:
|
||||||
|
continue
|
||||||
|
rows.append({"name": name, "port": port, "backend": default, "routes": routes})
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def haproxy_targets(section: Mapping[str, Any] | None) -> list[dict[str, Any]]:
|
||||||
|
"""Every `server` line's `{name, host, port}` — the outbound half of a haproxy config."""
|
||||||
|
rows: list[dict[str, Any]] = []
|
||||||
|
for backend in _mappings((section or {}).get("backends")):
|
||||||
|
for server in _mappings(backend.get("servers")):
|
||||||
|
host = _text(server.get("target"))
|
||||||
|
if not host:
|
||||||
|
continue
|
||||||
|
rows.append({"name": _text(server.get("name")), "host": host,
|
||||||
|
"port": _port(server.get("port"))})
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def nginx_listen_ports(section: Mapping[str, Any] | None) -> list[tuple[int, list[str]]]:
|
||||||
|
"""`(port, server_names)` per nginx `listen`, in config order.
|
||||||
|
|
||||||
|
`server_names` is kept **verbatim**, the catch-all `_` included — it is what the config
|
||||||
|
says, and a page that wants to call it "any name" can say so without this module editing
|
||||||
|
the file's own text.
|
||||||
|
"""
|
||||||
|
rows: list[tuple[int, list[str]]] = []
|
||||||
|
for server in _mappings((section or {}).get("servers")):
|
||||||
|
names = _names(server.get("server_names"))
|
||||||
|
for listen in _mappings(server.get("listen")):
|
||||||
|
port = _port(listen.get("port"))
|
||||||
|
if port is not None:
|
||||||
|
rows.append((port, names))
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def nginx_targets(section: Mapping[str, Any] | None) -> list[dict[str, Any]]:
|
||||||
|
"""Every `proxy_pass` that names an address: `{name: null, host, port}`.
|
||||||
|
|
||||||
|
A `proxy_pass $var` has `target: null` by the agent's design and contributes no row here;
|
||||||
|
`nginx_map_targets()` is where its fan-out lives.
|
||||||
|
"""
|
||||||
|
rows: list[dict[str, Any]] = []
|
||||||
|
for server in _mappings((section or {}).get("servers")):
|
||||||
|
for proxy in _mappings(server.get("proxy_passes")):
|
||||||
|
target = _target_of(proxy.get("target"))
|
||||||
|
if target is None:
|
||||||
|
continue
|
||||||
|
rows.append({"name": None, "host": target[0], "port": target[1]})
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def proxy_variables(section: Mapping[str, Any] | None) -> set[str]:
|
||||||
|
"""The variable names a `proxy_pass` passes to, without their `$` (the agent drops it once)."""
|
||||||
|
names: set[str] = set()
|
||||||
|
for server in _mappings((section or {}).get("servers")):
|
||||||
|
for proxy in _mappings(server.get("proxy_passes")):
|
||||||
|
variable = _text(proxy.get("variable"))
|
||||||
|
if variable:
|
||||||
|
names.add(variable.lstrip("$"))
|
||||||
|
return names
|
||||||
|
|
||||||
|
|
||||||
|
def nginx_map_targets(section: Mapping[str, Any] | None) -> list[dict[str, Any]]:
|
||||||
|
"""The `map` entries that route: `{key, host, port}`, for maps a `proxy_pass` actually uses.
|
||||||
|
|
||||||
|
`key` is the SNI domain the map matches on, so the page can say "this domain goes here".
|
||||||
|
Two refusals, both rule 2: a map no `proxy_pass` names contributes nothing (the
|
||||||
|
`$connection_upgrade` map's values are the header words `upgrade`/`close`, and calling one
|
||||||
|
a host would be inventing a fact to fill a field), and an entry whose target is `null`
|
||||||
|
contributes nothing either.
|
||||||
|
"""
|
||||||
|
if section is None:
|
||||||
|
return []
|
||||||
|
wanted = proxy_variables(section)
|
||||||
|
rows: list[dict[str, Any]] = []
|
||||||
|
for block in _mappings(section.get("maps")):
|
||||||
|
variable = _text(block.get("variable"))
|
||||||
|
if not variable or variable.lstrip("$") not in wanted:
|
||||||
|
continue
|
||||||
|
for entry in _mappings(block.get("entries")):
|
||||||
|
key = _text(entry.get("key"))
|
||||||
|
target = _target_of(entry.get("target"))
|
||||||
|
if not key or target is None:
|
||||||
|
continue
|
||||||
|
rows.append({"key": key, "host": target[0], "port": target[1]})
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def _target_of(node: Any) -> tuple[str, int | None] | None:
|
||||||
|
"""A config `{host, port}` block as `(host, port)`, or ``None`` when it is not one.
|
||||||
|
|
||||||
|
One reader for the three places a target appears (a haproxy `server`, an nginx
|
||||||
|
`proxy_pass`, a `map` entry), so "no address written" is one answer everywhere: `proxy_pass
|
||||||
|
$var` has `target: null` by the agent's design and must produce no pipe in all of them.
|
||||||
|
"""
|
||||||
|
if not isinstance(node, Mapping):
|
||||||
|
return None
|
||||||
|
host = _text(node.get("host"))
|
||||||
|
if not host:
|
||||||
|
return None
|
||||||
|
return host, _port(node.get("port"))
|
||||||
|
|
||||||
|
|
||||||
|
def socket_rows(section: Mapping[str, Any] | None) -> tuple[list[Mapping[str, Any]],
|
||||||
|
list[Mapping[str, Any]]]:
|
||||||
|
"""`(listens, established)` from the sockets section — both empty when it did not answer."""
|
||||||
|
if section is None or _available(section) is not True:
|
||||||
|
return [], []
|
||||||
|
return _mappings(section.get("listens")), _mappings(section.get("established"))
|
||||||
|
|
||||||
|
|
||||||
|
def live_listen_ports(listening: Sequence[Mapping[str, Any]]) -> set[int]:
|
||||||
|
"""The ports this host is actually serving, from the kernel's LISTEN rows.
|
||||||
|
|
||||||
|
The set decides the direction of a socket nothing matched (a port here means this host is
|
||||||
|
the one serving it, which makes the connection an inbound one the configs do not name) and
|
||||||
|
is the denominator of `unmatched.other`. Duplicated addresses of one port collapse here:
|
||||||
|
a port is what the configs speak.
|
||||||
|
"""
|
||||||
|
return {port for port in (_port(item.get("port")) for item in listening) if port is not None}
|
||||||
|
|
||||||
|
|
||||||
|
def _remote(socket: Mapping[str, Any]) -> tuple[str | None, int | None]:
|
||||||
|
remote = socket.get("remote")
|
||||||
|
if not isinstance(remote, Mapping):
|
||||||
|
return None, None
|
||||||
|
return _text(remote.get("addr")), _port(remote.get("port"))
|
||||||
|
|
||||||
|
|
||||||
|
def _local_port(socket: Mapping[str, Any]) -> int | None:
|
||||||
|
"""The port this host is serving on. The local **address** is deliberately not read: the
|
||||||
|
join is by port, and a listen address (`0.0.0.0` vs the host's own) distinguishes nothing
|
||||||
|
a frontend config names anyway.
|
||||||
|
"""
|
||||||
|
local = socket.get("local")
|
||||||
|
if not isinstance(local, Mapping):
|
||||||
|
return None
|
||||||
|
return _port(local.get("port"))
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# the join
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def _outbound_key(host: str, port: int | None) -> tuple[str, int | None]:
|
||||||
|
return (host, port)
|
||||||
|
|
||||||
|
|
||||||
|
def _outbound_pipe(host: str, port: int | None, name: str | None, source: str | None,
|
||||||
|
*, counting: bool) -> dict[str, Any]:
|
||||||
|
"""One configured target, with the live count joined onto it later.
|
||||||
|
|
||||||
|
`metric` travels in the pipe so a page asks `/api/gateway/history` for the series this
|
||||||
|
pipe *is* instead of rebuilding the name (the same reason `/disks` cards and `/llm` cells
|
||||||
|
carry their own keys). `by_port` says how the join will have to work: a target written as
|
||||||
|
an address matches exactly, a target written as a name can only match on its port, and
|
||||||
|
the payload must never let the reader confuse the two.
|
||||||
|
"""
|
||||||
|
suffix = f"{host}:{port}" if port is not None else host
|
||||||
|
return {
|
||||||
|
"target": {"host": host, "port": port, "name": name,
|
||||||
|
"by_port": port is not None and not _is_ip_literal(host)},
|
||||||
|
# a target that names no port can never be joined at all, so its count stays unknown even
|
||||||
|
# when the socket table answered — `0` there would claim a measurement nobody took.
|
||||||
|
"connections": 0 if (counting and port is not None) else None,
|
||||||
|
"source": source,
|
||||||
|
"metric": f"{SOURCE}.out_conns@{suffix}",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _bump(pipe: dict[str, Any]) -> None:
|
||||||
|
pipe["connections"] = int(pipe["connections"] or 0) + 1
|
||||||
|
|
||||||
|
|
||||||
|
def join_sockets(inbound: Mapping[int, dict[str, Any]],
|
||||||
|
outbound: Mapping[tuple[str, int | None], dict[str, Any]],
|
||||||
|
established: Sequence[Mapping[str, Any]],
|
||||||
|
live_listen_ports: set[int]) -> dict[str, Any]:
|
||||||
|
"""One pass over the ESTABLISHED table, attributing each socket **at most once**.
|
||||||
|
|
||||||
|
Mutates the pipes' `connections` and `clients` and returns
|
||||||
|
`{inbound, outbound, ambiguous}`: the connections no configured frontend names, the
|
||||||
|
connections no specific configured target claims (the ambiguous ones included), and the
|
||||||
|
per-port tally of the ambiguous fan-out. Decision order and why each arm is honest are in
|
||||||
|
the module docstring; the property that matters to the page is the partition —
|
||||||
|
inbound pipes + outbound pipes + `inbound` + `outbound` == the socket table, so the counts
|
||||||
|
on screen reconcile against a direct `/proc/net/tcp` read (phase 29 task 06 checks exactly
|
||||||
|
that), and no socket is ever drawn down two pipes.
|
||||||
|
|
||||||
|
The port-only arm counts **domain** targets only, and a port whose configured hosts are
|
||||||
|
more than one is never resolved by picking: `/proc` does not know which name the socket
|
||||||
|
was dialled to and neither does anything downstream of it.
|
||||||
|
"""
|
||||||
|
hosts_by_port: dict[int, set[str]] = {}
|
||||||
|
for host, port in outbound:
|
||||||
|
if port is not None:
|
||||||
|
hosts_by_port.setdefault(port, set()).add(host)
|
||||||
|
|
||||||
|
ambiguous: dict[int, int] = {}
|
||||||
|
clients: dict[int, list[str]] = {}
|
||||||
|
unmatched_in = 0
|
||||||
|
unmatched_out = 0
|
||||||
|
for socket in established:
|
||||||
|
local_port = _local_port(socket)
|
||||||
|
remote_addr, remote_port = _remote(socket)
|
||||||
|
pipe = inbound.get(local_port) if local_port is not None else None
|
||||||
|
if pipe is not None:
|
||||||
|
_bump(pipe)
|
||||||
|
if remote_addr is not None:
|
||||||
|
seen = clients.setdefault(int(pipe["port"]), [])
|
||||||
|
if remote_addr not in seen:
|
||||||
|
seen.append(remote_addr)
|
||||||
|
continue
|
||||||
|
exact = (outbound.get(_outbound_key(remote_addr, remote_port))
|
||||||
|
if remote_addr is not None else None)
|
||||||
|
if exact is not None:
|
||||||
|
_bump(exact)
|
||||||
|
continue
|
||||||
|
candidates = hosts_by_port.get(remote_port) if remote_port is not None else None
|
||||||
|
if candidates and remote_port is not None:
|
||||||
|
if len(candidates) > 1:
|
||||||
|
ambiguous[remote_port] = ambiguous.get(remote_port, 0) + 1
|
||||||
|
unmatched_out += 1
|
||||||
|
continue
|
||||||
|
sole = candidates.copy().pop()
|
||||||
|
# a target written as an address was already tried exactly above; a miss there is
|
||||||
|
# a different endpoint, not this one, so only a *name* may be matched by port.
|
||||||
|
named = (outbound.get(_outbound_key(sole, remote_port))
|
||||||
|
if not _is_ip_literal(sole) else None)
|
||||||
|
if named is not None:
|
||||||
|
_bump(named)
|
||||||
|
continue
|
||||||
|
if local_port is not None and local_port in live_listen_ports:
|
||||||
|
# this host is *serving* the socket, on a port neither config names
|
||||||
|
unmatched_in += 1
|
||||||
|
else:
|
||||||
|
unmatched_out += 1
|
||||||
|
for port, seen in clients.items():
|
||||||
|
inbound[port]["clients"] = sorted(seen)[:MAX_CLIENTS]
|
||||||
|
return {"inbound": unmatched_in, "outbound": unmatched_out, "ambiguous": ambiguous}
|
||||||
|
|
||||||
|
|
||||||
|
def _groups(ambiguous: Mapping[int, int],
|
||||||
|
pipes: Mapping[tuple[str, int | None], dict[str, Any]]) -> list[dict[str, Any]]:
|
||||||
|
"""The ambiguous port groups, one entry per port with live connections on it."""
|
||||||
|
groups: list[dict[str, Any]] = []
|
||||||
|
for port, count in sorted(ambiguous.items()):
|
||||||
|
hosts = sorted({host for host, target in pipes if target == port})
|
||||||
|
groups.append({"port": port, "connections": count, "hosts": hosts[:MAX_GROUP_HOSTS],
|
||||||
|
"host_count": len(hosts)})
|
||||||
|
return groups
|
||||||
|
|
||||||
|
|
||||||
|
def _skeleton(host: str) -> dict[str, Any]:
|
||||||
|
"""The payload with every list empty and every count `None`.
|
||||||
|
|
||||||
|
One declaration of the key set, used by `parse` (which then fills it in) and by `stub` (which
|
||||||
|
leaves it empty), so the two arms cannot drift apart in the one way that bites a page: a key
|
||||||
|
the page indexes existing in one arm and not the other.
|
||||||
|
"""
|
||||||
|
return {
|
||||||
|
"configured": True, # the route's gate decided this; the payload says so too
|
||||||
|
"supported": None, # the caller decides: `parse` a verdict, `stub` an unknown
|
||||||
|
"host": host,
|
||||||
|
"ts": None,
|
||||||
|
"sections": dict.fromkeys(SECTIONS, None),
|
||||||
|
"why": None,
|
||||||
|
"section_errors": [],
|
||||||
|
"haproxy": {"frontends": []},
|
||||||
|
"nginx": {"servers": [], "map_targets": []},
|
||||||
|
"inbound": [],
|
||||||
|
"outbound": [],
|
||||||
|
"unmatched": {"inbound": None, "outbound": None, "other": None, "groups": []},
|
||||||
|
"sockets": {"listens": None, "established": None},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def stub(host: str, *, ts: float | None = None) -> dict[str, Any]:
|
||||||
|
"""A configured gateway agent that has **never answered**, in this payload's own shape.
|
||||||
|
|
||||||
|
The `/api/system` and `/api/disks` stub rule (invariants 3 and 4): the source is in the
|
||||||
|
deployment, it has no numbers yet, and `unreachable: true` is what tells those two facts
|
||||||
|
apart. Every count is `None` rather than `0` — an agent that has not spoken has not reported
|
||||||
|
an idle ingress — and `supported` is `None` rather than `False`, because nobody has read its
|
||||||
|
version. The page keeps the row, dims it, and prints `n/a`.
|
||||||
|
"""
|
||||||
|
payload = _skeleton(host)
|
||||||
|
payload["ts"] = ts
|
||||||
|
payload["unreachable"] = True
|
||||||
|
payload["why"] = WHY_NO_ANSWER
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def parse(snapshot: Mapping[str, Any], host: str) -> dict[str, Any]:
|
||||||
|
"""One agent snapshot -> the `/gateway` payload. Pure: no I/O, no clock, no resolver.
|
||||||
|
|
||||||
|
The shape is the page's whole world: `haproxy`/`nginx` are the config as the agent read it
|
||||||
|
(the echo the pipes were built from), `inbound`/`outbound` are the pipes with their live
|
||||||
|
counts, `unmatched` is everything no config claimed (plus the ambiguous port groups),
|
||||||
|
`sockets` is the raw size of what the agent saw, and `supported`/`sections`/`why` are the
|
||||||
|
three degradation arms. Counts are `null` — never `0` — when nothing answered, and no
|
||||||
|
count carries a `severity`.
|
||||||
|
|
||||||
|
`sockets.established` counts **kernel socket rows**, which is what the reconciliation
|
||||||
|
identity is stated against: a loopback connection is two rows, one per end, so a gateway
|
||||||
|
proxying to a service on itself appears once inbound and once outbound — the same traffic
|
||||||
|
seen from both sides, which is exactly what the two pipes are there to show.
|
||||||
|
"""
|
||||||
|
status = section_status(snapshot)
|
||||||
|
supports = supported(snapshot)
|
||||||
|
haproxy = _section(snapshot, "haproxy")
|
||||||
|
nginx = _section(snapshot, "nginx")
|
||||||
|
sockets = _section(snapshot, "sockets")
|
||||||
|
listening, established = socket_rows(sockets)
|
||||||
|
|
||||||
|
payload = _skeleton(host)
|
||||||
|
payload["supported"] = supports
|
||||||
|
payload["ts"] = snapshot.get("ts")
|
||||||
|
payload["sections"] = status
|
||||||
|
payload["why"] = " · ".join(why_lines(snapshot)) or None
|
||||||
|
payload["section_errors"] = section_errors(snapshot)
|
||||||
|
version = _text(snapshot.get("agent_version"))
|
||||||
|
if version:
|
||||||
|
payload["agent_version"] = version
|
||||||
|
if not supports:
|
||||||
|
# The 1.1.0 / --no-proxy arm: every key the page indexes is there and empty, every
|
||||||
|
# count is null, and `why` is the one sentence. Nothing here is red — an agent that
|
||||||
|
# cannot answer this question is unconfigured for it (invariant 3), and the job's own
|
||||||
|
# health record stays `ok` because the snapshot itself arrived fine.
|
||||||
|
return payload
|
||||||
|
|
||||||
|
payload["haproxy"]["frontends"] = haproxy_frontends(haproxy)
|
||||||
|
server_names: dict[int, list[str]] = {}
|
||||||
|
for port, names in nginx_listen_ports(nginx):
|
||||||
|
merged = server_names.setdefault(port, [])
|
||||||
|
for name in names:
|
||||||
|
if name not in merged:
|
||||||
|
merged.append(name)
|
||||||
|
payload["nginx"]["servers"] = [{"port": port, "server_names": names}
|
||||||
|
for port, names in server_names.items()]
|
||||||
|
payload["nginx"]["map_targets"] = nginx_map_targets(nginx)
|
||||||
|
|
||||||
|
# --- inbound pipes, one per configured listen port -------------------------
|
||||||
|
# A port belongs to one socket space whether haproxy or nginx names it (a host cannot
|
||||||
|
# bind it twice), so the port is the key and the domains of every config that names it
|
||||||
|
# are merged onto that one pipe. `source` is the first config to name it, which is the
|
||||||
|
# honest answer to "where did this pipe come from" and nothing more.
|
||||||
|
counting = _available(sockets) is True
|
||||||
|
inbound: dict[int, dict[str, Any]] = {}
|
||||||
|
|
||||||
|
def claim_port(port: int, source: str, domains: Sequence[str]) -> None:
|
||||||
|
pipe = inbound.get(port)
|
||||||
|
if pipe is None:
|
||||||
|
inbound[port] = {"port": port, "source": source, "domains": list(domains),
|
||||||
|
"connections": 0 if counting else None,
|
||||||
|
"clients": [], "metric": f"{SOURCE}.in_conns@{port}"}
|
||||||
|
return
|
||||||
|
merged = pipe["domains"]
|
||||||
|
for domain in domains:
|
||||||
|
if domain not in merged:
|
||||||
|
merged.append(domain)
|
||||||
|
|
||||||
|
for row in payload["haproxy"]["frontends"]:
|
||||||
|
claim_port(int(row["port"]), "haproxy", ())
|
||||||
|
for port, names in server_names.items():
|
||||||
|
claim_port(port, "nginx", names)
|
||||||
|
|
||||||
|
# --- outbound pipes, one per configured target -----------------------------
|
||||||
|
outbound: dict[tuple[str, int | None], dict[str, Any]] = {}
|
||||||
|
|
||||||
|
def claim_target(host_name: str, port: int | None, name: str | None,
|
||||||
|
source: str | None) -> None:
|
||||||
|
"""One pipe per configured `host:port`, first claimer wins.
|
||||||
|
|
||||||
|
haproxy is claimed before nginx before the map, which is the order that keeps
|
||||||
|
information rather than overwriting it: a haproxy `server` line carries a name and an
|
||||||
|
nginx `proxy_pass` does not, so a duplicate never displaces a name the config has.
|
||||||
|
The socket space is one pipe either way — a second pipe for the same endpoint would
|
||||||
|
draw the same connections twice.
|
||||||
|
"""
|
||||||
|
key = _outbound_key(host_name, port)
|
||||||
|
if key not in outbound:
|
||||||
|
outbound[key] = _outbound_pipe(host_name, port, name, source, counting=counting)
|
||||||
|
|
||||||
|
for row in haproxy_targets(haproxy):
|
||||||
|
claim_target(row["host"], row["port"], row["name"], "haproxy")
|
||||||
|
for row in nginx_targets(nginx):
|
||||||
|
claim_target(row["host"], row["port"], None, "nginx")
|
||||||
|
for row in payload["nginx"]["map_targets"]:
|
||||||
|
claim_target(row["host"], row["port"], None, "map")
|
||||||
|
|
||||||
|
if not counting:
|
||||||
|
# The configs are known and the live half is not: every pipe is drawn, every count is
|
||||||
|
# null, and the tallies say nothing rather than zero (rule 2 — the sockets section
|
||||||
|
# answers "no answer", never "no connections").
|
||||||
|
payload["inbound"] = list(inbound.values())
|
||||||
|
payload["outbound"] = list(outbound.values())
|
||||||
|
return payload
|
||||||
|
|
||||||
|
# --- the joins -------------------------------------------------------------
|
||||||
|
live_ports = live_listen_ports(listening)
|
||||||
|
tallies = join_sockets(inbound, outbound, established, live_ports)
|
||||||
|
# A port whose live sockets could not be attributed is unknown for **each** of its candidate
|
||||||
|
# targets — not `0` (there are connections; we cannot say whose) and not the port's total
|
||||||
|
# repeated on every pipe (a fan-out drawing reads that as M times the traffic). A port with
|
||||||
|
# no traffic at all keeps its honest `0` on every candidate, because that is a fact.
|
||||||
|
for ambiguous_port in tallies["ambiguous"]:
|
||||||
|
for (_target_host, target_port), pipe in outbound.items():
|
||||||
|
if target_port == ambiguous_port:
|
||||||
|
pipe["connections"] = None
|
||||||
|
payload["inbound"] = list(inbound.values())
|
||||||
|
payload["outbound"] = list(outbound.values())
|
||||||
|
payload["unmatched"] = {
|
||||||
|
"inbound": tallies["inbound"],
|
||||||
|
"outbound": tallies["outbound"],
|
||||||
|
# the listener-level fact, phase 28's "an unclaimed socket stays visible as one":
|
||||||
|
# a port this host listens on that neither config names.
|
||||||
|
"other": len([port for port in live_ports if port not in inbound]),
|
||||||
|
"groups": _groups(tallies["ambiguous"], outbound),
|
||||||
|
}
|
||||||
|
payload["sockets"] = {"listens": len(listening), "established": len(established)}
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def samples(payload: Mapping[str, Any], ts: float | None = None) -> list[Sample]:
|
||||||
|
"""One gauge per pipe that has a number.
|
||||||
|
|
||||||
|
A `null` count publishes **no** sample — "unknown" is not a zero, and a chart of zeros for
|
||||||
|
a dead socket table is the failure mode rule 2 names. The timestamp is the poll's (when the
|
||||||
|
dashboard read the count), carried on `ts`; nothing here is a rate, so nothing here needs a
|
||||||
|
window.
|
||||||
|
"""
|
||||||
|
when = time.time() if ts is None else ts
|
||||||
|
rows: list[Sample] = []
|
||||||
|
for pipe in payload.get("inbound") or []:
|
||||||
|
if isinstance(pipe, Mapping):
|
||||||
|
rows.extend(_gauge(payload, pipe, when))
|
||||||
|
for pipe in payload.get("outbound") or []:
|
||||||
|
if isinstance(pipe, Mapping):
|
||||||
|
rows.extend(_gauge(payload, pipe, when))
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
def _gauge(payload: Mapping[str, Any], pipe: Mapping[str, Any],
|
||||||
|
when: float) -> list[Sample]:
|
||||||
|
value = pipe.get("connections")
|
||||||
|
metric = pipe.get("metric")
|
||||||
|
if not isinstance(metric, str) or isinstance(value, bool) or not isinstance(value, int | float):
|
||||||
|
return []
|
||||||
|
return [Sample(SOURCE, str(payload.get("host") or ""), metric, float(value), when)]
|
||||||
|
|
||||||
|
|
||||||
|
def parse_gateway_snapshot(name: str, snapshot: Mapping[str, Any],
|
||||||
|
*, now: float | None = None) -> Result:
|
||||||
|
"""One snapshot -> (`/gateway` payload, gauges, no events).
|
||||||
|
|
||||||
|
The `sources/disks.py` shape for the poller: payload first, samples from the same
|
||||||
|
snapshot, and the timestamp the store keys on is the snapshot's when it published one.
|
||||||
|
There are **no events** here on purpose: a connection count has no transition worth an
|
||||||
|
event line (a proxy that stops routing shows up as its host's own poll failing, which the
|
||||||
|
poller already reports), so this source adds nothing to the event feed of its own accord.
|
||||||
|
"""
|
||||||
|
payload = parse(snapshot, name)
|
||||||
|
stamp = payload.get("ts")
|
||||||
|
when = (float(stamp) if isinstance(stamp, int | float) and not isinstance(stamp, bool)
|
||||||
|
else (time.time() if now is None else now))
|
||||||
|
return Result(payload=payload, samples=samples(payload, when))
|
||||||
@@ -4,7 +4,7 @@ Why SQLite and not Prometheus: the fleet is a handful of boxes, the poll resolut
|
|||||||
need is 1-10s, and a single file the container owns beats running a TSDB. Series are
|
need is 1-10s, and a single file the container owns beats running a TSDB. Series are
|
||||||
addressed by `(source, host, metric, res, ts)`:
|
addressed by `(source, host, metric, res, ts)`:
|
||||||
|
|
||||||
source the dashboard family — "host", "llm", "ups"
|
source the dashboard family — "host", "disks", "llm", "ups", "gateway"
|
||||||
host stable target key ("driveripper", "gufo", "apc-ups1")
|
host stable target key ("driveripper", "gufo", "apc-ups1")
|
||||||
metric dotted name, optionally `@stream` for per-device series
|
metric dotted name, optionally `@stream` for per-device series
|
||||||
(cpu.usage_pct, cpu.usage_pct@core7, disk.used_pct@/srv, gpu.busy_pct@0)
|
(cpu.usage_pct, cpu.usage_pct@core7, disk.used_pct@/srv, gpu.busy_pct@0)
|
||||||
|
|||||||
@@ -25,6 +25,7 @@
|
|||||||
<a href="/disks">Disks</a>
|
<a href="/disks">Disks</a>
|
||||||
<a href="/llm">LLM</a>
|
<a href="/llm">LLM</a>
|
||||||
<a href="/ups">UPS</a>
|
<a href="/ups">UPS</a>
|
||||||
|
<a href="/gateway">Gateway</a>
|
||||||
</nav>
|
</nav>
|
||||||
<div class="masthead-right">
|
<div class="masthead-right">
|
||||||
<div id="nav-status" class="gap" aria-live="polite"></div>
|
<div id="nav-status" class="gap" aria-live="polite"></div>
|
||||||
|
|||||||
@@ -18,6 +18,7 @@
|
|||||||
<a href="/disks" aria-current="page">Disks</a>
|
<a href="/disks" aria-current="page">Disks</a>
|
||||||
<a href="/llm">LLM</a>
|
<a href="/llm">LLM</a>
|
||||||
<a href="/ups">UPS</a>
|
<a href="/ups">UPS</a>
|
||||||
|
<a href="/gateway">Gateway</a>
|
||||||
</nav>
|
</nav>
|
||||||
<div class="masthead-right">
|
<div class="masthead-right">
|
||||||
<div id="nav-status" class="gap" aria-live="polite"></div>
|
<div id="nav-status" class="gap" aria-live="polite"></div>
|
||||||
|
|||||||
@@ -0,0 +1,222 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en" data-theme="dark">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||||
|
<title>Homelab · gateway</title>
|
||||||
|
<link rel="stylesheet" href="/static/dash.css">
|
||||||
|
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' rx='22' fill='%23ff8a3d'/></svg>">
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<header class="masthead">
|
||||||
|
<a class="brand" href="/"><span class="brand-mark">◆</span> Homelab</a>
|
||||||
|
<button class="ghost nav-toggle" id="nav-toggle" type="button"
|
||||||
|
aria-expanded="false" aria-controls="nav-tabs" aria-label="Toggle navigation">☰</button>
|
||||||
|
<nav class="tabs" id="nav-tabs" aria-label="Dashboards">
|
||||||
|
<a href="/">Overview</a>
|
||||||
|
<a href="/system">System</a>
|
||||||
|
<a href="/disks">Disks</a>
|
||||||
|
<a href="/llm">LLM</a>
|
||||||
|
<a href="/ups">UPS</a>
|
||||||
|
<a href="/gateway" aria-current="page">Gateway</a>
|
||||||
|
</nav>
|
||||||
|
<div class="masthead-right">
|
||||||
|
<div id="nav-status" class="gap" aria-live="polite"></div>
|
||||||
|
<a class="ghost account" href="/certs" title="Agent certificates">⚷ Certs</a>
|
||||||
|
<a class="ghost account" href="/settings" title="Session and password">⚙ Account</a>
|
||||||
|
<button class="ghost" id="theme-toggle" type="button" aria-pressed="false">☾ Dark</button>
|
||||||
|
<span class="clock" id="clock">--:--:--</span>
|
||||||
|
</div>
|
||||||
|
</header>
|
||||||
|
|
||||||
|
<main>
|
||||||
|
<div class="page-head">
|
||||||
|
<div>
|
||||||
|
<h1>Gateway</h1>
|
||||||
|
<div class="sub" id="sub">reading the ingress…</div>
|
||||||
|
</div>
|
||||||
|
<div class="spacer"></div>
|
||||||
|
<!-- The socket table's own totals beside the totals the pipes carry, so the page prints its
|
||||||
|
reconciliation instead of asserting it: `listeners` and `established` are what the
|
||||||
|
kernel said, and the two rows under the cards add up to them only when every connection
|
||||||
|
belonged to a configured pipe. No cell is coloured — a connection count is not a graded
|
||||||
|
number (`net.*`'s rule, and `severity.py` has no `gateway.*` row). -->
|
||||||
|
<div class="gate-tallies" id="tallies">
|
||||||
|
<div class="metric">
|
||||||
|
<div class="label">listeners</div>
|
||||||
|
<div class="value sm" id="t-listeners">n/a</div>
|
||||||
|
</div>
|
||||||
|
<div class="metric">
|
||||||
|
<div class="label">established</div>
|
||||||
|
<div class="value sm" id="t-established">n/a</div>
|
||||||
|
</div>
|
||||||
|
<div class="metric">
|
||||||
|
<div class="label">pipes in</div>
|
||||||
|
<div class="value sm" id="t-inbound">n/a</div>
|
||||||
|
</div>
|
||||||
|
<div class="metric">
|
||||||
|
<div class="label">pipes out</div>
|
||||||
|
<div class="value sm" id="t-outbound">n/a</div>
|
||||||
|
</div>
|
||||||
|
<div class="metric">
|
||||||
|
<div class="label">unmatched</div>
|
||||||
|
<div class="value sm" id="t-unmatched">n/a</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- The payload's own sentence about why it measured nothing, printed once for the page:
|
||||||
|
`sources/gateway.py` writes it for a pre-1.2.0 agent, for `--no-proxy`, for a host that
|
||||||
|
has never answered and for a config or socket table that did not — each with the agent's
|
||||||
|
own error line when it had one. A card does not repeat it. -->
|
||||||
|
<div class="banner info" id="gate-banner" role="status" hidden></div>
|
||||||
|
|
||||||
|
<!-- The other arm, and it is not an error: `/api/gateway/snapshot` answers
|
||||||
|
`404 {configured: false}` **before** dialling anything, so a deployment with no ingress
|
||||||
|
gets a setup instruction rather than a red panel (invariant 3). It is the same sentence the
|
||||||
|
route returns, which is why it names `GATEWAY_HOST` and the agent version that reads the
|
||||||
|
configs — the two things an operator has to change. -->
|
||||||
|
<div class="empty-state" id="gate-empty" hidden>
|
||||||
|
<h3>No gateway configured</h3>
|
||||||
|
<p>Set <code>GATEWAY_HOST</code> to the <code>HOST_TARGETS</code> name of the machine that
|
||||||
|
routes this fleet, and run <code>agent/host_agent.py</code> <b>1.2.0</b> on it — that is
|
||||||
|
the version that reads the proxy configs and the kernel's socket table.</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<!-- One card per source family the payload names, `data-gate` keyed on a pipe's own `source`
|
||||||
|
word plus `unmatched`: a pipe is filed by what the config said it came from, not by which
|
||||||
|
card the page would rather it sat on. Every card ships hidden and appears on the poll that
|
||||||
|
has pipes for it (or a config that answered and named none), so a gateway with no nginx in
|
||||||
|
it does not spend a panel saying so. -->
|
||||||
|
<section>
|
||||||
|
<div class="page-head"><h2 style="font-size:.95rem;margin:0">Connection map</h2></div>
|
||||||
|
<div class="grid auto items-start gate-grid" id="gate-cards">
|
||||||
|
<article class="card gate-card" data-gate="haproxy" hidden>
|
||||||
|
<header>
|
||||||
|
<span class="dot idle" data-k="dot"></span>
|
||||||
|
<h3>haproxy</h3>
|
||||||
|
<span class="chip" data-k="counts"></span>
|
||||||
|
<span class="spacer"></span>
|
||||||
|
<span class="faint mono updated"></span>
|
||||||
|
</header>
|
||||||
|
<div class="body stack">
|
||||||
|
<div class="pipe-columns">
|
||||||
|
<div class="pipe-col in" data-k="in"></div>
|
||||||
|
<div class="pipe-node" data-k="node" hidden></div>
|
||||||
|
<div class="pipe-col out" data-k="out"></div>
|
||||||
|
</div>
|
||||||
|
<div class="pipe-empty faint" data-k="empty" hidden></div>
|
||||||
|
<details class="charts-box" data-k="charts">
|
||||||
|
<summary>Charts<span class="chip charts-window" data-k="charts-window"></span></summary>
|
||||||
|
<div class="host-charts">
|
||||||
|
<div class="host-chart">
|
||||||
|
<div class="chart sm" data-chart="conns"></div>
|
||||||
|
<div class="legend" data-legend="conns"></div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</details>
|
||||||
|
</div>
|
||||||
|
</article>
|
||||||
|
|
||||||
|
<article class="card gate-card" data-gate="nginx" hidden>
|
||||||
|
<header>
|
||||||
|
<span class="dot idle" data-k="dot"></span>
|
||||||
|
<h3>nginx</h3>
|
||||||
|
<span class="chip" data-k="counts"></span>
|
||||||
|
<span class="spacer"></span>
|
||||||
|
<span class="faint mono updated"></span>
|
||||||
|
</header>
|
||||||
|
<div class="body stack">
|
||||||
|
<div class="pipe-columns">
|
||||||
|
<div class="pipe-col in" data-k="in"></div>
|
||||||
|
<div class="pipe-node" data-k="node" hidden></div>
|
||||||
|
<div class="pipe-col out" data-k="out"></div>
|
||||||
|
</div>
|
||||||
|
<div class="pipe-empty faint" data-k="empty" hidden></div>
|
||||||
|
<details class="charts-box" data-k="charts">
|
||||||
|
<summary>Charts<span class="chip charts-window" data-k="charts-window"></span></summary>
|
||||||
|
<div class="host-charts">
|
||||||
|
<div class="host-chart">
|
||||||
|
<div class="chart sm" data-chart="conns"></div>
|
||||||
|
<div class="legend" data-legend="conns"></div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</details>
|
||||||
|
</div>
|
||||||
|
</article>
|
||||||
|
|
||||||
|
<!-- The stream map's pipes are `map`-sourced while the listener they are reached through is
|
||||||
|
an `nginx` one, so this card is the fan-out alone. Joining the `443` listener into it
|
||||||
|
would be the page inventing a relationship the agent never reported. -->
|
||||||
|
<article class="card gate-card" data-gate="map" hidden>
|
||||||
|
<header>
|
||||||
|
<span class="dot idle" data-k="dot"></span>
|
||||||
|
<h3>stream map</h3>
|
||||||
|
<span class="chip" data-k="counts"></span>
|
||||||
|
<span class="spacer"></span>
|
||||||
|
<span class="faint mono updated"></span>
|
||||||
|
</header>
|
||||||
|
<div class="body stack">
|
||||||
|
<div class="pipe-columns">
|
||||||
|
<div class="pipe-col in" data-k="in"></div>
|
||||||
|
<div class="pipe-node" data-k="node" hidden></div>
|
||||||
|
<div class="pipe-col out" data-k="out"></div>
|
||||||
|
</div>
|
||||||
|
<div class="pipe-empty faint" data-k="empty" hidden></div>
|
||||||
|
<details class="charts-box" data-k="charts">
|
||||||
|
<summary>Charts<span class="chip charts-window" data-k="charts-window"></span></summary>
|
||||||
|
<div class="host-charts">
|
||||||
|
<div class="host-chart">
|
||||||
|
<div class="chart sm" data-chart="conns"></div>
|
||||||
|
<div class="legend" data-legend="conns"></div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</details>
|
||||||
|
</div>
|
||||||
|
</article>
|
||||||
|
|
||||||
|
<!-- Not pipes: there is no config to draw these from, only the socket table's leftovers.
|
||||||
|
`listeners no config names` counts **ports** and the other two count **connections**,
|
||||||
|
which is why they are three cells and never one sum. -->
|
||||||
|
<article class="card gate-card" data-gate="unmatched" hidden>
|
||||||
|
<header>
|
||||||
|
<span class="dot idle" data-k="dot"></span>
|
||||||
|
<h3>unmatched</h3>
|
||||||
|
<span class="spacer"></span>
|
||||||
|
<span class="faint mono updated"></span>
|
||||||
|
</header>
|
||||||
|
<div class="body stack">
|
||||||
|
<div class="metrics">
|
||||||
|
<div class="metric">
|
||||||
|
<div class="label">listeners no config names</div>
|
||||||
|
<div class="value sm" data-k="um-other">n/a</div>
|
||||||
|
</div>
|
||||||
|
<div class="metric">
|
||||||
|
<div class="label">connections inbound</div>
|
||||||
|
<div class="value sm" data-k="um-in">n/a</div>
|
||||||
|
</div>
|
||||||
|
<div class="metric">
|
||||||
|
<div class="label">connections outbound</div>
|
||||||
|
<div class="value sm" data-k="um-out">n/a</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<ul class="gate-groups" data-k="groups" hidden></ul>
|
||||||
|
</div>
|
||||||
|
</article>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
|
||||||
|
<footer class="pagefoot">
|
||||||
|
<span class="faint">a pipe animates while it carries at least one live connection and rests
|
||||||
|
at zero, and is never coloured by a verdict — a busy ingress is the job.</span>
|
||||||
|
<span class="faint">a <code>by port</code> pipe was matched on its socket port
|
||||||
|
alone: the target is a name, the agent resolves nothing, and that port carries more than one
|
||||||
|
configured host — so the count is the port's, not this target's.</span>
|
||||||
|
</footer>
|
||||||
|
</main>
|
||||||
|
|
||||||
|
<script src="/static/charts.js"></script>
|
||||||
|
<script src="/static/app.js"></script>
|
||||||
|
<script src="/static/page-gateway.js"></script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -18,6 +18,7 @@
|
|||||||
<a href="/disks">Disks</a>
|
<a href="/disks">Disks</a>
|
||||||
<a href="/llm">LLM</a>
|
<a href="/llm">LLM</a>
|
||||||
<a href="/ups">UPS</a>
|
<a href="/ups">UPS</a>
|
||||||
|
<a href="/gateway">Gateway</a>
|
||||||
</nav>
|
</nav>
|
||||||
<div class="masthead-right">
|
<div class="masthead-right">
|
||||||
<div id="nav-status" class="gap" aria-live="polite"></div>
|
<div id="nav-status" class="gap" aria-live="polite"></div>
|
||||||
@@ -67,6 +68,10 @@
|
|||||||
<p>Run <code>agent/host_agent.py</code> on each machine and list it in
|
<p>Run <code>agent/host_agent.py</code> on each machine and list it in
|
||||||
<code>HOST_TARGETS</code>.</p>
|
<code>HOST_TARGETS</code>.</p>
|
||||||
</div>
|
</div>
|
||||||
|
<!-- The ingress's one line, filled by page-index.js from `/api/summary`'s `gateway`
|
||||||
|
block and hidden while that source is unconfigured: a gateway nobody installed gets
|
||||||
|
no row here and no red one to learn to ignore (invariant 3). -->
|
||||||
|
<p class="faint mono" id="fleet-gate" style="margin:.5rem 0 0;font-size:.8rem" hidden></p>
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -18,6 +18,7 @@
|
|||||||
<a href="/disks">Disks</a>
|
<a href="/disks">Disks</a>
|
||||||
<a href="/llm" aria-current="page">LLM</a>
|
<a href="/llm" aria-current="page">LLM</a>
|
||||||
<a href="/ups">UPS</a>
|
<a href="/ups">UPS</a>
|
||||||
|
<a href="/gateway">Gateway</a>
|
||||||
</nav>
|
</nav>
|
||||||
<div class="masthead-right">
|
<div class="masthead-right">
|
||||||
<div id="nav-status" class="gap" aria-live="polite"></div>
|
<div id="nav-status" class="gap" aria-live="polite"></div>
|
||||||
|
|||||||
@@ -23,6 +23,7 @@
|
|||||||
<a href="/disks">Disks</a>
|
<a href="/disks">Disks</a>
|
||||||
<a href="/llm">LLM</a>
|
<a href="/llm">LLM</a>
|
||||||
<a href="/ups">UPS</a>
|
<a href="/ups">UPS</a>
|
||||||
|
<a href="/gateway">Gateway</a>
|
||||||
</nav>
|
</nav>
|
||||||
<div class="masthead-right">
|
<div class="masthead-right">
|
||||||
<div id="nav-status" class="gap" aria-live="polite"></div>
|
<div id="nav-status" class="gap" aria-live="polite"></div>
|
||||||
|
|||||||
@@ -18,6 +18,7 @@
|
|||||||
<a href="/disks">Disks</a>
|
<a href="/disks">Disks</a>
|
||||||
<a href="/llm">LLM</a>
|
<a href="/llm">LLM</a>
|
||||||
<a href="/ups">UPS</a>
|
<a href="/ups">UPS</a>
|
||||||
|
<a href="/gateway">Gateway</a>
|
||||||
</nav>
|
</nav>
|
||||||
<div class="masthead-right">
|
<div class="masthead-right">
|
||||||
<div id="nav-status" class="gap" aria-live="polite"></div>
|
<div id="nav-status" class="gap" aria-live="polite"></div>
|
||||||
|
|||||||
@@ -18,6 +18,7 @@
|
|||||||
<a href="/disks">Disks</a>
|
<a href="/disks">Disks</a>
|
||||||
<a href="/llm">LLM</a>
|
<a href="/llm">LLM</a>
|
||||||
<a href="/ups" aria-current="page">UPS</a>
|
<a href="/ups" aria-current="page">UPS</a>
|
||||||
|
<a href="/gateway">Gateway</a>
|
||||||
</nav>
|
</nav>
|
||||||
<div class="masthead-right">
|
<div class="masthead-right">
|
||||||
<div id="nav-status" class="gap" aria-live="polite"></div>
|
<div id="nav-status" class="gap" aria-live="polite"></div>
|
||||||
|
|||||||
@@ -658,6 +658,10 @@ const Dash = (() => {
|
|||||||
{ key: 'disks', label: 'disks', href: '/disks' },
|
{ key: 'disks', label: 'disks', href: '/disks' },
|
||||||
{ key: 'llm', label: 'engines', href: '/llm' },
|
{ key: 'llm', label: 'engines', href: '/llm' },
|
||||||
{ key: 'ups', label: 'UPS', href: '/ups' },
|
{ key: 'ups', label: 'UPS', href: '/ups' },
|
||||||
|
// The fifth dashboard's job is a single job for the one ingress, so this badge is one
|
||||||
|
// dot rather than one per host — and it stays grey when `GATEWAY_HOST` names nothing,
|
||||||
|
// because `refreshStatus` only counts sources the health record calls `configured`.
|
||||||
|
{ key: 'gateway', label: 'gateway', href: '/gateway' },
|
||||||
];
|
];
|
||||||
nav.innerHTML = jobs.map((job) => `<span class="chip" title="${job.label}">
|
nav.innerHTML = jobs.map((job) => `<span class="chip" title="${job.label}">
|
||||||
<span class="dot idle" data-dot="${job.key}"></span>${job.label}</span>`).join('');
|
<span class="dot idle" data-dot="${job.key}"></span>${job.label}</span>`).join('');
|
||||||
|
|||||||
@@ -124,7 +124,15 @@ class LineChart {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
if (!isFinite(lo)) return [0, 1];
|
if (!isFinite(lo)) return [0, 1];
|
||||||
if (lo === hi) { const pad = Math.abs(lo) * 0.1 || 1; return [lo - pad, hi + pad]; }
|
if (lo === hi) {
|
||||||
|
// A flat series still needs a band to draw in, but a declared floor is not negotiable:
|
||||||
|
// padding below `yMin: 0` put "-1 connections" on the axis of a gateway whose every pipe was
|
||||||
|
// idle, which is a scale describing a state that cannot happen (the same reason the page
|
||||||
|
// prints `n/a` instead of a made-up zero).
|
||||||
|
const pad = Math.abs(lo) * 0.1 || 1;
|
||||||
|
const floor = this.opts.yMin !== null ? Math.min(lo, this.opts.yMin) : lo - pad;
|
||||||
|
return [floor, hi + pad];
|
||||||
|
}
|
||||||
const margin = (hi - lo) * 0.12;
|
const margin = (hi - lo) * 0.12;
|
||||||
lo = this.opts.yMin !== null ? this.opts.yMin : lo - margin;
|
lo = this.opts.yMin !== null ? this.opts.yMin : lo - margin;
|
||||||
hi = this.opts.yMax !== null ? this.opts.yMax : hi + margin;
|
hi = this.opts.yMax !== null ? this.opts.yMax : hi + margin;
|
||||||
|
|||||||
@@ -284,20 +284,26 @@ button.ghost:disabled { opacity: .45; cursor: not-allowed; }
|
|||||||
padding-top: calc(.5rem + env(safe-area-inset-top, 0px));
|
padding-top: calc(.5rem + env(safe-area-inset-top, 0px));
|
||||||
}
|
}
|
||||||
|
|
||||||
/* The same treatment one level down, and these three rules are still what a 360px phone
|
/* The same treatment one level down, and these rules are still what a 360px phone actually
|
||||||
* actually requires — phase 18 measured both sides of them: without them the four status
|
* requires — phase 18 measured both sides of them: the status dots fell into a single-file
|
||||||
* dots fell into a single-file column (`#nav-status` carries the shared `.gap` class, which
|
* column and `⚷ Certs` broke over two lines, because every item shrank to a fraction of its
|
||||||
* is `flex-wrap: wrap`) and `⚷ Certs` broke over two lines, because every item shrank to a
|
* content. So the cluster may wrap, its items may neither shrink nor break a label, and the
|
||||||
* fraction of its content. So the cluster may wrap, its items may neither shrink nor break a
|
* dots keep their labels. Row 3 still wraps *within itself*: `⚷ Certs`, `⚙ Account` and the
|
||||||
* label, and the dots stay the one line that makes them a readout instead of a stack. Row 3
|
* theme button do not fit in 320px once over on the narrowest phones, and nothing here hides a
|
||||||
* still wraps *within itself*: `⚷ Certs`, `⚙ Account` and the theme button do not fit in
|
* control to make the arithmetic work — a phone gets the same words every other width does.
|
||||||
* 320px once over on the narrowest phones, and nothing here hides a control to make
|
* The row alignment is the base rule's `flex-end` (phase 25 task 02), restated nowhere: this
|
||||||
* the arithmetic work — a phone gets the same words every other width does. The row
|
* block only tightens the gap, so the two rows cannot disagree about which edge they sit on.
|
||||||
* alignment is the base rule's `flex-end` (phase 25 task 02), restated nowhere: this block
|
*
|
||||||
* only tightens the gap, so the two rows cannot disagree about which edge they sit on. */
|
* The dots wrap too, since phase 29 task 05 gave them a fifth member. Phase 18's `nowrap` was
|
||||||
|
* right for four (measured then at 267px) and wrong for five: `hosts disks engines UPS
|
||||||
|
* gateway` is 352px of chips against this bar's 320px content box, and an item that may not
|
||||||
|
* shrink and may not wrap does not become smaller — it widens the *document* (372px, so a
|
||||||
|
* phone got a sideways scroll, which phase 18 exists to forbid). Wrapping inside a row that is
|
||||||
|
* `flex-basis: 100%` is not the single-file column phase 18 saw either: the fifth chip moves to
|
||||||
|
* a second line of the same right-aligned row and the bar grows by one line box. */
|
||||||
.masthead-right { flex-wrap: wrap; gap: .4rem; }
|
.masthead-right { flex-wrap: wrap; gap: .4rem; }
|
||||||
.masthead-right > * { flex: none; white-space: nowrap; }
|
.masthead-right > * { flex: none; white-space: nowrap; }
|
||||||
#nav-status { flex-wrap: nowrap; }
|
#nav-status { flex-wrap: wrap; row-gap: .25rem; }
|
||||||
|
|
||||||
/* The reveal, through task 01's token: `#nav-toggle` reads its `display` from
|
/* The reveal, through task 01's token: `#nav-toggle` reads its `display` from
|
||||||
* `--nav-toggle-display`, so declaring the token on the element here is the whole flip —
|
* `--nav-toggle-display`, so declaring the token on the element here is the whole flip —
|
||||||
@@ -1253,6 +1259,102 @@ input[type="text"].confirm {
|
|||||||
* and it is the one cell a table cannot nowrap: an address list. */
|
* and it is the one cell a table cannot nowrap: an address list. */
|
||||||
table.data td.san { white-space: normal; }
|
table.data td.san { white-space: normal; }
|
||||||
|
|
||||||
|
/* ---- gateway pipes ---------------------------------------------------------
|
||||||
|
* `/gateway` (phase 29 task 04): the connection map. One card per source family, and inside each
|
||||||
|
* card two columns of pipes joined by one node — the gateway itself. A pipe is one `<path>` with
|
||||||
|
* a travelling dash pattern, so the whole vocabulary here is ink and motion: a connection count
|
||||||
|
* is not a graded number, which is why this section draws no coloured verdict on any of it — the
|
||||||
|
* `net.*` precedent, and the API sends no verdict for a gateway figure at any depth.
|
||||||
|
*
|
||||||
|
* Three layout facts are load-bearing, all of them measured in `data/logs/29-task04-pipes.cjs`:
|
||||||
|
* the node is one rounded rect that does **not** stretch to the taller column (a 17-target
|
||||||
|
* fan-out would otherwise grow a 600 px wall beside a six-pipe side); an empty column collapses
|
||||||
|
* and the node moves to the edge it is joined to (`solo`); and at a phone width the three columns
|
||||||
|
* stack with the node between them, because 320 px of card has no room for a pipe *and* its
|
||||||
|
* label, and an unlabelled pipe is a line nobody can read.
|
||||||
|
*/
|
||||||
|
.gate-tallies { display: flex; flex-wrap: wrap; gap: .9rem; align-items: flex-end; }
|
||||||
|
/* "listeners no config names" is a fact, and `.metric .label` truncates with an ellipsis at a card
|
||||||
|
width that cannot fit it — a clipped label is a clipped fact (phase 27's rule), so these wrap.
|
||||||
|
The five header tallies keep the shared truncating rule: their labels are one word each. */
|
||||||
|
.gate-card .metric .label { white-space: normal; }
|
||||||
|
.gate-grid { min-width: 0; }
|
||||||
|
.pipe-columns {
|
||||||
|
display: grid; gap: .5rem 1rem; min-width: 0; align-items: center;
|
||||||
|
grid-template-columns: minmax(0, 1fr) auto minmax(0, 1fr);
|
||||||
|
}
|
||||||
|
/* A card with pipes on one side only. The stream map's fan-out is the case that ships: its
|
||||||
|
* inbound half is the nginx `443` listener and lives on the nginx card, because the payload never
|
||||||
|
* says which listener a map is reached through and the page does not guess one. Two classes, not
|
||||||
|
* one, because *which* side is empty decides which grid track the node belongs in. */
|
||||||
|
.pipe-columns.solo-out { grid-template-columns: auto minmax(0, 1fr); }
|
||||||
|
.pipe-columns.solo-in { grid-template-columns: minmax(0, 1fr) auto; }
|
||||||
|
.pipe-col { display: grid; gap: .5rem; min-width: 0; align-content: center; }
|
||||||
|
.pipe-node {
|
||||||
|
align-self: center; justify-self: center; text-align: center;
|
||||||
|
min-width: 6rem; max-width: 9rem; padding: .6rem .45rem;
|
||||||
|
border: 1px solid var(--border-strong); border-radius: var(--radius);
|
||||||
|
background: var(--bg-sunken); color: var(--text-muted);
|
||||||
|
font: 600 11.5px var(--mono); letter-spacing: .04em; overflow-wrap: anywhere;
|
||||||
|
}
|
||||||
|
.pipe-row { display: grid; gap: .12rem; min-width: 0; }
|
||||||
|
.pipe-label {
|
||||||
|
display: flex; flex-wrap: wrap; align-items: baseline; gap: .05rem .35rem; min-width: 0;
|
||||||
|
}
|
||||||
|
/* A pipe's label is the page's whole answer — which port, which domain, which target — and this row
|
||||||
|
is what makes the answer readable. Task 06 measured it on live data: as one `nowrap` flex line the
|
||||||
|
name shrank to whatever the `by port` chip, the config's own `server` name and the `flex: none`
|
||||||
|
count left, which drew seven characters of `reeseapps-gitea.reeselink.com:2222`; wrapping the text
|
||||||
|
inside that squeezed box was worse (eleven lines, a word at a time). So the name claims the line:
|
||||||
|
`max-content` as its base size means a name longer than the column is alone on its line and gets
|
||||||
|
the *whole* column (~22 characters), with the chip, the footnote and the count wrapping beneath it,
|
||||||
|
while a short label like `:22` still sits beside its count on one line exactly as before. What is
|
||||||
|
still clipped after that is a target longer than the column, which is a column-width fact
|
||||||
|
(AGENTS.md gap 13) and not a lost one: `renderPipeRow` puts the full string on the element as
|
||||||
|
`title`, `pipeAria` carries the whole endpoint, and
|
||||||
|
`test_a_clipped_pipe_label_keeps_its_full_text_on_the_element` pins both declarations and both
|
||||||
|
strings. The live pass re-counts the clipped labels on every run. */
|
||||||
|
.pipe-name {
|
||||||
|
flex: 1 1 max-content; font-size: 11.5px; min-width: 0;
|
||||||
|
overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
|
||||||
|
}
|
||||||
|
.pipe-detail { font-size: 11px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||||
|
/* the count sits at the node end of the label line, so the eye goes from the number straight
|
||||||
|
into the pipe that carries it */
|
||||||
|
.pipe-count {
|
||||||
|
margin-inline-start: auto; padding-inline-start: .5rem; flex: none;
|
||||||
|
font: 600 12.5px var(--mono);
|
||||||
|
}
|
||||||
|
/* The pipe is a grid row of its own, the full width of its column — which is what makes it touch
|
||||||
|
* the node, since the inbound column is the node's left and the outbound column its right. */
|
||||||
|
.pipe { display: block; width: 100%; min-width: 0; height: 10px; }
|
||||||
|
.pipe-path { fill: none; stroke: var(--border-strong); stroke-width: 2; stroke-dasharray: 7 9; }
|
||||||
|
/* One connection is enough to move. The dash travels the pipe's own direction, which is
|
||||||
|
* *towards* the node in the inbound column and *away from* it in the outbound one — the node is
|
||||||
|
* the far end of one and the near end of the other, so one keyframe is the whole direction
|
||||||
|
* story and a reversed second one would only be a second thing to keep in step. */
|
||||||
|
.pipe.live .pipe-path { stroke: var(--accent); animation: pipe-flow 1.2s linear infinite; }
|
||||||
|
/* Zero is a static, faint line rather than an absent one: "this pipe exists and carries nothing"
|
||||||
|
* is a fact the drawing has to be able to say, and it is a different sentence from `n/a`. */
|
||||||
|
.pipe.still .pipe-path { stroke: var(--text-faint); stroke-dasharray: 3 7; opacity: .55; }
|
||||||
|
@keyframes pipe-flow {
|
||||||
|
/* one full dash+gap cycle (7 + 9), so the loop is seamless */
|
||||||
|
from { stroke-dashoffset: 0; }
|
||||||
|
to { stroke-dashoffset: -16; }
|
||||||
|
}
|
||||||
|
.pipe-clients { font-size: 10.5px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
||||||
|
.pipe-empty { font-size: 12px; }
|
||||||
|
.gate-groups { list-style: none; margin: 0; padding: 0; display: grid; gap: .3rem; }
|
||||||
|
.gate-groups li { display: flex; gap: .45rem; align-items: baseline; min-width: 0; }
|
||||||
|
.gate-groups .pipe-name { flex: none; }
|
||||||
|
|
||||||
|
@media (max-width: 640px) {
|
||||||
|
.pipe-columns,
|
||||||
|
.pipe-columns.solo-out,
|
||||||
|
.pipe-columns.solo-in { grid-template-columns: minmax(0, 1fr); }
|
||||||
|
.pipe-node { max-width: none; width: 100%; }
|
||||||
|
}
|
||||||
|
|
||||||
/* ---- dialog ------------------------------------------------------------- */
|
/* ---- dialog ------------------------------------------------------------- */
|
||||||
dialog {
|
dialog {
|
||||||
border: 1px solid var(--border-strong); border-radius: var(--radius); background: var(--bg-raised);
|
border: 1px solid var(--border-strong); border-radius: var(--radius); background: var(--bg-raised);
|
||||||
@@ -1313,4 +1415,11 @@ footer.pagefoot {
|
|||||||
|
|
||||||
nav.tabs,
|
nav.tabs,
|
||||||
nav.tabs.is-open { transition: none; }
|
nav.tabs.is-open { transition: none; }
|
||||||
|
|
||||||
|
/* `/gateway`'s pipes, **both states**. The animated rule out-specifies the base one exactly
|
||||||
|
* the way `.is-open` out-specifies `nav.tabs`, so stilling the static state alone would leave
|
||||||
|
* a live pipe flowing for someone who asked for no motion — the same phase-46 lesson the menu
|
||||||
|
* pair above carries, applied the day a second animated component landed in this sheet. */
|
||||||
|
.pipe.live .pipe-path,
|
||||||
|
.pipe.still .pipe-path { animation: none; }
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,646 @@
|
|||||||
|
/* Gateway page: what this ingress is connected to, and how much is on right now.
|
||||||
|
|
||||||
|
`/system` answers what the gateway box is doing. This answers the topology question it cannot:
|
||||||
|
which sockets face inbound, where each one's traffic goes, and how many connections are live on
|
||||||
|
each. The source is `/api/gateway/snapshot` — `sources/gateway.py`'s join of the agent's
|
||||||
|
`haproxy` and `nginx` config parses with its `/proc/net/tcp{,6}` tallies.
|
||||||
|
|
||||||
|
The page draws **pipes**, and a pipe's only claim is a count. It animates while that count is at
|
||||||
|
least one connection and rests at zero; it is never coloured by a verdict (`net.*`'s precedent:
|
||||||
|
a busy ingress is the job, and `severity.py` rates no `gateway.*` metric). `n/a` means the join
|
||||||
|
did not decide — a socket table that did not answer, or a port whose configured targets cannot be
|
||||||
|
told apart — and `n/a` never becomes `0` here (rule 2).
|
||||||
|
|
||||||
|
Two rules keep the drawing honest:
|
||||||
|
|
||||||
|
* **The label is the config's own text.** A bind keeps its port, a `server_name` keeps its
|
||||||
|
domain, a `proxy_pass` keeps the address it was written with, and a target the join matched
|
||||||
|
on its port alone says `by port` rather than pretending the socket named the host.
|
||||||
|
* **The unreconciled remainder is printed, not hidden.** `unmatched` is the connections no
|
||||||
|
configured pipe claims, and it is the operator's fact that the gateway talks to something
|
||||||
|
the configs do not name.
|
||||||
|
|
||||||
|
Charts follow the `/system` discipline: one collapsed `<details>` per card, painted only while
|
||||||
|
open, fed by **one** `/api/gateway/history` request per card per refresh naming every gauge that
|
||||||
|
card could draw.
|
||||||
|
*/
|
||||||
|
'use strict';
|
||||||
|
|
||||||
|
const { $, $$, fetchJSON, setTxt, setDot, num, duration, poll, markStale } = Dash;
|
||||||
|
|
||||||
|
const SVG_NS = 'http://www.w3.org/2000/svg';
|
||||||
|
|
||||||
|
const state = {
|
||||||
|
/* No window chips: the page's history is the count trend, and one window for every card is
|
||||||
|
* the honest default for a source that re-reads every 30 s. */
|
||||||
|
range: 86400,
|
||||||
|
rangeKey: '24h',
|
||||||
|
every: 30, // GATEWAY_INTERVAL_SECONDS, from /api/config — never guessed here
|
||||||
|
cards: new Map(), // card key -> its element (the skeletons the page ships)
|
||||||
|
rows: new Map(), // card key -> Map(pipe key -> row element): patched, not rebuilt
|
||||||
|
charts: new Map(), // card key -> LineChart, built on first expand only
|
||||||
|
history: new Map(), // card key -> { range, response } from the last snapshot pass
|
||||||
|
snapshot: null, // last payload, so a toggle paints without fetching
|
||||||
|
poller: null,
|
||||||
|
/* Half the job's interval: an age stamp cannot sit at "updated 29s" for a whole cycle, and
|
||||||
|
* nothing on the page claims a fresher reading than the agent took. */
|
||||||
|
pollMs: 15000,
|
||||||
|
};
|
||||||
|
|
||||||
|
// ---- the cards --------------------------------------------------------------
|
||||||
|
//
|
||||||
|
// One card per source family, keyed on the payload's own `source` word plus `unmatched`. That
|
||||||
|
// keying is not a layout preference: `source` is the only fact in the document that says which
|
||||||
|
// config a pipe came from, and the stream fan-out's pipes are `map`-sourced while the listener
|
||||||
|
// they are reached through is an `nginx` one — so the map card is outbound by construction, and
|
||||||
|
// joining the two would be the page inventing a relationship the agent never reported.
|
||||||
|
const CARDS = [
|
||||||
|
{ key: 'haproxy', title: 'haproxy', section: 'haproxy',
|
||||||
|
empty: 'haproxy names no listening port on this host' },
|
||||||
|
{ key: 'nginx', title: 'nginx', section: 'nginx',
|
||||||
|
empty: 'nginx names no listening port on this host' },
|
||||||
|
{ key: 'map', title: 'the stream map', section: 'nginx',
|
||||||
|
empty: 'the stream map routes to nothing' },
|
||||||
|
];
|
||||||
|
|
||||||
|
const CHART_HEIGHT = 120;
|
||||||
|
const COUNT_FORMAT = (v) => num(v, 0);
|
||||||
|
const CLIENTS_SHOWN = 2;
|
||||||
|
|
||||||
|
/** The measured count as text; the unknown is `null` so `setTxt` prints `n/a` and not `0`. */
|
||||||
|
function countText(value) {
|
||||||
|
return typeof value === 'number' && isFinite(value) ? num(value, 0) : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** One pipe family's pipes, split on the payload's own `source` word. */
|
||||||
|
function familyPipes(snapshot, key) {
|
||||||
|
return {
|
||||||
|
inbound: (snapshot.inbound || []).filter((pipe) => pipe.source === key),
|
||||||
|
outbound: (snapshot.outbound || []).filter((pipe) => pipe.source === key),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** `host:port` -> the stream-map keys that route to it, out of `nginx.map_targets`. */
|
||||||
|
function mapKeys(snapshot) {
|
||||||
|
const keys = new Map();
|
||||||
|
((snapshot.nginx || {}).map_targets || []).forEach((entry) => {
|
||||||
|
if (!entry || !entry.key) return;
|
||||||
|
const name = `${entry.host}:${entry.port}`;
|
||||||
|
const list = keys.get(name) || [];
|
||||||
|
list.push(entry.key);
|
||||||
|
keys.set(name, list);
|
||||||
|
});
|
||||||
|
return keys;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The label a pipe carries.
|
||||||
|
*
|
||||||
|
* Inbound is a port, because that is what the config binds. Outbound is the address the config
|
||||||
|
* was written with, and for a stream-map target the SNI name the map matched on rides in front of
|
||||||
|
* it — both of those strings are the document's own, and nothing here resolves or reverses one.
|
||||||
|
*/
|
||||||
|
function pipeName(pipe, keys) {
|
||||||
|
if (pipe.port != null) return `:${pipe.port}`;
|
||||||
|
const target = pipe.target || {};
|
||||||
|
const host = target.host || '?';
|
||||||
|
const endpoint = `${host}${target.port == null ? '' : `:${target.port}`}`;
|
||||||
|
const through = keys ? keys.get(`${target.host}:${target.port}`) : null;
|
||||||
|
return through && through.length === 1 ? `${through[0]} → ${endpoint}` : endpoint;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A pipe's stable identity: its own gauge name, unique per listen port and per target. */
|
||||||
|
function pipeKey(pipe) {
|
||||||
|
return pipe.metric || `${pipe.port}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The row's footnote: the haproxy `server` name, or the count of map keys on one endpoint. */
|
||||||
|
function pipeDetail(pipe, keys) {
|
||||||
|
const target = pipe.target || {};
|
||||||
|
if (target.name) return `server ${target.name}`;
|
||||||
|
const through = keys ? keys.get(`${target.host}:${target.port}`) : null;
|
||||||
|
if (through && through.length > 1) return `${through.length} map names`;
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The accessible sentence for one pipe — every figure in it came from the payload. */
|
||||||
|
function pipeAria(direction, pipe, keys) {
|
||||||
|
const bits = [direction === 'in' ? 'inbound pipe' : 'outbound pipe'];
|
||||||
|
bits.push(direction === 'in' ? `port ${pipe.port}` : `to ${pipeName(pipe, keys)}`);
|
||||||
|
const domains = pipe.domains || [];
|
||||||
|
if (domains.length) bits.push(`names ${domains.join(', ')}`);
|
||||||
|
const clients = pipe.clients || [];
|
||||||
|
if (clients.length) bits.push(`clients ${clients.join(', ')}`);
|
||||||
|
bits.push(pipe.connections == null ? 'connections unknown'
|
||||||
|
: `${pipe.connections} connections`);
|
||||||
|
return bits.join(', ');
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The pipe itself: hand-rolled SVG — one path, `role="img"`, an `aria-label` of its own. */
|
||||||
|
function buildPipe() {
|
||||||
|
const svg = document.createElementNS(SVG_NS, 'svg');
|
||||||
|
svg.setAttribute('class', 'pipe still');
|
||||||
|
svg.setAttribute('viewBox', '0 0 100 10');
|
||||||
|
svg.setAttribute('preserveAspectRatio', 'none');
|
||||||
|
svg.setAttribute('focusable', 'false');
|
||||||
|
const path = document.createElementNS(SVG_NS, 'path');
|
||||||
|
path.setAttribute('class', 'pipe-path');
|
||||||
|
// One geometry for both columns: the dash runs from x=0 to x=100, which is *towards* the
|
||||||
|
// gateway node in the inbound column and *away from* it in the outbound one. The node is the
|
||||||
|
// far end of one and the near end of the other, so a single keyframe is the whole direction
|
||||||
|
// story and a reversed second one would be a second thing to keep in step.
|
||||||
|
path.setAttribute('d', 'M0 5 H100');
|
||||||
|
path.setAttribute('vector-effect', 'non-scaling-stroke');
|
||||||
|
svg.appendChild(path);
|
||||||
|
return svg;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A pipe row: the label line (name, `by port`, the config's own name for the target, count),
|
||||||
|
* the SVG pipe under it, and the client addresses under that. Built once, patched afterwards.
|
||||||
|
*
|
||||||
|
* The pipe is the full width of its column, which is what makes it touch the node: the inbound
|
||||||
|
* column is the node's left and the outbound column its right, so one geometry pointing right
|
||||||
|
* runs *into* the gateway in one column and *away from* it in the other.
|
||||||
|
*/
|
||||||
|
function pipeRow(direction) {
|
||||||
|
const el = document.createElement('div');
|
||||||
|
el.className = `pipe-row ${direction}`;
|
||||||
|
el.innerHTML = `
|
||||||
|
<span class="pipe-label">
|
||||||
|
<span class="pipe-name mono"></span>
|
||||||
|
<span class="chip pipe-flag" hidden></span>
|
||||||
|
<span class="pipe-detail faint" hidden></span>
|
||||||
|
<span class="pipe-count"></span>
|
||||||
|
</span>
|
||||||
|
<span class="pipe-clients faint" hidden></span>`;
|
||||||
|
el.insertBefore(buildPipe(), el.querySelector('.pipe-clients'));
|
||||||
|
return el;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Patch one row from its pipe.
|
||||||
|
*
|
||||||
|
* Every string here is upstream-provided — a `server_name`, a `proxy_pass`, a client address —
|
||||||
|
* so it goes out through `textContent` and `title`, never through markup.
|
||||||
|
*/
|
||||||
|
function renderPipeRow(el, direction, pipe, keys) {
|
||||||
|
const name = el.querySelector('.pipe-name');
|
||||||
|
const text = pipeName(pipe, keys);
|
||||||
|
if (name.textContent !== text) name.textContent = text;
|
||||||
|
const domains = pipe.domains || [];
|
||||||
|
name.title = domains.length ? domains.join(', ') : text;
|
||||||
|
|
||||||
|
const flag = el.querySelector('.pipe-flag');
|
||||||
|
const byPort = !!(pipe.target && pipe.target.by_port);
|
||||||
|
flag.hidden = !byPort;
|
||||||
|
if (byPort) {
|
||||||
|
setTxt(flag, 'by port');
|
||||||
|
flag.title = 'matched on the port alone — this target is a name and the port carries more '
|
||||||
|
+ 'than one configured host, so the count is the port\'s, not this target\'s';
|
||||||
|
} else {
|
||||||
|
flag.removeAttribute('title');
|
||||||
|
}
|
||||||
|
|
||||||
|
// the config's own label for this target — the haproxy `server` name, or how many map names
|
||||||
|
// arrive at one endpoint
|
||||||
|
const detail = el.querySelector('.pipe-detail');
|
||||||
|
const note = pipeDetail(pipe, keys);
|
||||||
|
detail.hidden = note == null;
|
||||||
|
if (note != null) setTxt(detail, note);
|
||||||
|
|
||||||
|
const shown = countText(pipe.connections);
|
||||||
|
setTxt(el.querySelector('.pipe-count'), shown);
|
||||||
|
|
||||||
|
const svg = el.querySelector('.pipe');
|
||||||
|
const live = typeof pipe.connections === 'number' && isFinite(pipe.connections)
|
||||||
|
&& pipe.connections > 0;
|
||||||
|
svg.classList.toggle('live', live);
|
||||||
|
svg.classList.toggle('still', !live);
|
||||||
|
svg.setAttribute('role', 'img');
|
||||||
|
const aria = pipeAria(direction, pipe, keys);
|
||||||
|
if (svg.getAttribute('aria-label') !== aria) svg.setAttribute('aria-label', aria);
|
||||||
|
|
||||||
|
// The clients are the row's third line and its tooltip: the addresses are the operator's
|
||||||
|
// shortest path from "this port is busy" to "that machine is busy".
|
||||||
|
const clients = (pipe.clients || []).slice(0, CLIENTS_SHOWN);
|
||||||
|
const rest = (pipe.clients || []).length - clients.length;
|
||||||
|
const track = el.querySelector('.pipe-clients');
|
||||||
|
const shown2 = clients.length ? `${clients.join(' ')}${rest > 0 ? ` +${rest}` : ''}` : '';
|
||||||
|
track.hidden = shown2.length === 0;
|
||||||
|
setTxt(track, shown2);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Make a column's rows match `pipes`, patching the ones that stay and dropping the rest. */
|
||||||
|
function renderPipes(card, direction, key, pipes, keys) {
|
||||||
|
const box = card.querySelector(`[data-k="${direction}"]`);
|
||||||
|
const seen = state.rows.get(key) || new Map();
|
||||||
|
state.rows.set(key, seen);
|
||||||
|
const wanted = new Set();
|
||||||
|
pipes.forEach((pipe) => {
|
||||||
|
const id = `${direction}:${pipeKey(pipe)}`;
|
||||||
|
wanted.add(id);
|
||||||
|
let row = seen.get(id);
|
||||||
|
if (!row) {
|
||||||
|
row = pipeRow(direction);
|
||||||
|
seen.set(id, row);
|
||||||
|
box.appendChild(row);
|
||||||
|
}
|
||||||
|
renderPipeRow(row, direction, pipe, keys);
|
||||||
|
});
|
||||||
|
Array.from(seen.keys()).forEach((id) => {
|
||||||
|
if (wanted.has(id) || !id.startsWith(`${direction}:`)) return;
|
||||||
|
seen.get(id).remove();
|
||||||
|
seen.delete(id);
|
||||||
|
});
|
||||||
|
// An empty column collapses rather than holding half the card for nothing — which is what
|
||||||
|
// moves the node to the edge it is being joined to (the `solo` class below).
|
||||||
|
box.hidden = pipes.length === 0;
|
||||||
|
return pipes.length;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The gateway node: one rounded rect mid-card, labelled with the host the reading came from. */
|
||||||
|
function renderNode(card, snapshot, columns) {
|
||||||
|
const node = card.querySelector('[data-k="node"]');
|
||||||
|
const visible = columns.inCount + columns.outCount > 0;
|
||||||
|
node.hidden = !visible;
|
||||||
|
// One column of pipes only: the node moves to the edge it is joined to instead of floating in
|
||||||
|
// the middle of an empty half-card. Two classes, because *which* half is empty decides which
|
||||||
|
// grid track the node belongs in, and one shared class could only ever be right about one.
|
||||||
|
const frame = card.querySelector('.pipe-columns');
|
||||||
|
frame.classList.toggle('solo-out', visible && columns.inCount === 0);
|
||||||
|
frame.classList.toggle('solo-in', visible && columns.outCount === 0);
|
||||||
|
const label = snapshot.host || 'gateway';
|
||||||
|
if (node.dataset.text !== label) {
|
||||||
|
node.dataset.text = label;
|
||||||
|
node.textContent = label;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- the unmatched card ------------------------------------------------------
|
||||||
|
//
|
||||||
|
// Not pipes: there is no config to draw these from, only the socket table's own leftovers. The
|
||||||
|
// two tallies are two different facts and never one sum — `other` counts *ports* no config
|
||||||
|
// claims, the other counts *connections* the join could not attribute to a target.
|
||||||
|
|
||||||
|
/** Does this payload leave anything unaccounted for? An all-zero family gets no card. */
|
||||||
|
/**
|
||||||
|
* Is there a remainder to report?
|
||||||
|
*
|
||||||
|
* A positive count or a group, yes. Three `null`s is not "unknown worth printing" — it is the join
|
||||||
|
* having measured nothing at all (a pre-1.2.0 agent, a silent socket table), and a card of three
|
||||||
|
* `n/a`s says less than the page's own `why` line above it already says (the dead-cell rule).
|
||||||
|
*/
|
||||||
|
function unmatchedVisible(snapshot) {
|
||||||
|
const un = snapshot.unmatched || {};
|
||||||
|
return [un.other, un.inbound, un.outbound].some(
|
||||||
|
(value) => typeof value === 'number' && value > 0) || (un.groups || []).length > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** One `<li>` of the "could belong to any of N" list — built, not templated: the names are data. */
|
||||||
|
function groupRow(group) {
|
||||||
|
const li = document.createElement('li');
|
||||||
|
const port = document.createElement('span');
|
||||||
|
port.className = 'pipe-name mono';
|
||||||
|
port.textContent = `:${group.port}`;
|
||||||
|
const why = document.createElement('span');
|
||||||
|
why.className = 'faint';
|
||||||
|
const hosts = (group.hosts || []).slice(0, CLIENTS_SHOWN);
|
||||||
|
const rest = (group.hosts || []).length - hosts.length;
|
||||||
|
why.textContent = `${countText(group.connections) || 'n/a'} connections · could be any of `
|
||||||
|
+ `${group.host_count} targets${hosts.length ? ` (${hosts.join(', ')}${rest > 0
|
||||||
|
? ` +${rest}` : ''})` : ''}`;
|
||||||
|
li.appendChild(port);
|
||||||
|
li.appendChild(why);
|
||||||
|
return li;
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderUnmatched(card, snapshot, row) {
|
||||||
|
const un = snapshot.unmatched || {};
|
||||||
|
card.hidden = false;
|
||||||
|
setDot(card.querySelector('[data-k="dot"]'), dotFor(row, snapshot));
|
||||||
|
setTxt(card.querySelector('.updated'), stalenessText(row));
|
||||||
|
markStale(card, dimmed(snapshot));
|
||||||
|
setTxt(card.querySelector('[data-k="um-other"]'), countText(un.other));
|
||||||
|
setTxt(card.querySelector('[data-k="um-in"]'), countText(un.inbound));
|
||||||
|
setTxt(card.querySelector('[data-k="um-out"]'), countText(un.outbound));
|
||||||
|
|
||||||
|
const groups = un.groups || [];
|
||||||
|
const list = card.querySelector('[data-k="groups"]');
|
||||||
|
const wanted = groups.map((group) => `${group.port}|${group.connections}|${group.host_count}`);
|
||||||
|
const same = list.dataset.wanted === wanted.join(',') && list.children.length === groups.length;
|
||||||
|
list.hidden = groups.length === 0;
|
||||||
|
if (same) return;
|
||||||
|
list.dataset.wanted = wanted.join(',');
|
||||||
|
list.textContent = '';
|
||||||
|
groups.forEach((group) => list.appendChild(groupRow(group)));
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- card chrome ---------------------------------------------------------------
|
||||||
|
|
||||||
|
/** The reachability dot: the job's own health, never a number. */
|
||||||
|
function dotFor(row, snapshot) {
|
||||||
|
if (!row || !row.configured) return 'idle';
|
||||||
|
if (row.ok === false || snapshot.unreachable) return 'fail';
|
||||||
|
return Dash.ageSeconds(row.last_ok) > 3 * state.every ? 'warn' : 'ok';
|
||||||
|
}
|
||||||
|
|
||||||
|
function stalenessText(row) {
|
||||||
|
if (!row || !row.configured) return 'not configured';
|
||||||
|
if (row.last_ok == null) return row.error || 'never reached';
|
||||||
|
const seconds = Dash.ageSeconds(row.last_ok);
|
||||||
|
return seconds > state.every ? `stale ${duration(seconds)}`
|
||||||
|
: `updated ${seconds < 2 ? 'now' : duration(seconds)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Did this payload fail to measure at all? Every arm of it dims; none of it reddens. */
|
||||||
|
function dimmed(snapshot) {
|
||||||
|
return !!snapshot.unreachable || snapshot.supported !== true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Every card skeleton, wired once to the markup the page ships. */
|
||||||
|
function cardElements() {
|
||||||
|
const cards = new Map();
|
||||||
|
$$('#gate-cards [data-gate]').forEach((el) => {
|
||||||
|
cards.set(el.dataset.gate, el);
|
||||||
|
const box = el.querySelector('[data-k="charts"]');
|
||||||
|
if (box) box.addEventListener('toggle', () => chartsToggled(el.dataset.gate));
|
||||||
|
const chip = el.querySelector('[data-k="charts-window"]');
|
||||||
|
if (chip) chip.textContent = state.rangeKey;
|
||||||
|
});
|
||||||
|
return cards;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Paint one card from the snapshot.
|
||||||
|
*
|
||||||
|
* A card appears when it has a fact: pipes of its own. A family whose config section did not
|
||||||
|
* answer has none, and the page's own `why` line is the one sentence that says which section —
|
||||||
|
* so the card stays off the grid instead of repeating it in a smaller font.
|
||||||
|
*/
|
||||||
|
function renderCard(card, snapshot, row) {
|
||||||
|
const key = card.dataset.gate;
|
||||||
|
if (key === 'unmatched') {
|
||||||
|
if (unmatchedVisible(snapshot)) renderUnmatched(card, snapshot, row);
|
||||||
|
else card.hidden = true;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const definition = CARDS.find((entry) => entry.key === key);
|
||||||
|
const { inbound, outbound } = familyPipes(snapshot, key);
|
||||||
|
if (inbound.length + outbound.length === 0) {
|
||||||
|
// Two different nothings. The section answered and named no pipe: that is a fact about this
|
||||||
|
// gateway's config, and the card says it. The section did not answer at all: the page's own
|
||||||
|
// `why` line is the one sentence that says which section, so no card repeats it.
|
||||||
|
const answered = (snapshot.sections || {})[definition.section] === true;
|
||||||
|
card.hidden = !answered;
|
||||||
|
if (answered) setTxt(card.querySelector('[data-k="empty"]'), definition.empty);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
card.hidden = false;
|
||||||
|
card.querySelector('[data-k="empty"]').hidden = true;
|
||||||
|
const keys = mapKeys(snapshot);
|
||||||
|
setDot(card.querySelector('[data-k="dot"]'), dotFor(row, snapshot));
|
||||||
|
setTxt(card.querySelector('.updated'), stalenessText(row));
|
||||||
|
markStale(card, dimmed(snapshot));
|
||||||
|
setTxt(card.querySelector('[data-k="counts"]'),
|
||||||
|
`${inbound.length} in · ${outbound.length} out`);
|
||||||
|
|
||||||
|
const columns = {
|
||||||
|
inCount: renderPipes(card, 'in', key, inbound, keys),
|
||||||
|
outCount: renderPipes(card, 'out', key, outbound, keys),
|
||||||
|
};
|
||||||
|
renderNode(card, snapshot, columns);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- per-card charts ------------------------------------------------------------
|
||||||
|
//
|
||||||
|
// One plot per card, one request per card, every refresh: the query names every gauge that card
|
||||||
|
// could draw (`/api/gateway/history` answers an unknown name as an empty series on purpose) and
|
||||||
|
// the legend names **pipes**, not hosts — the card heading already says which config family this
|
||||||
|
// is, and the page is about one machine.
|
||||||
|
|
||||||
|
/** Every gauge this card's pipes publish. */
|
||||||
|
function cardStreams(snapshot, key) {
|
||||||
|
const keys = mapKeys(snapshot);
|
||||||
|
const { inbound, outbound } = familyPipes(snapshot, key);
|
||||||
|
return inbound.concat(outbound)
|
||||||
|
.filter((pipe) => !!pipe.metric)
|
||||||
|
.map((pipe) => ({ metric: pipe.metric, label: pipeName(pipe, keys) }));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The chart inside one card, built the first time that card's stack is open. */
|
||||||
|
function cardChart(key) {
|
||||||
|
const built = state.charts.get(key);
|
||||||
|
if (built) return built;
|
||||||
|
const card = state.cards.get(key);
|
||||||
|
if (!card) return null;
|
||||||
|
const definition = CARDS.find((entry) => entry.key === key);
|
||||||
|
const chart = new DashCharts.LineChart(card.querySelector('[data-chart="conns"]'), {
|
||||||
|
height: CHART_HEIGHT, xSpan: state.range, yMin: 0, yFormat: COUNT_FORMAT,
|
||||||
|
ariaLabel: `${(state.snapshot || {}).host || 'gateway'} — live connections on each `
|
||||||
|
+ `${definition.title} pipe over the last ${state.rangeKey}`,
|
||||||
|
});
|
||||||
|
state.charts.set(key, chart);
|
||||||
|
return chart;
|
||||||
|
}
|
||||||
|
|
||||||
|
function seriesFor(response, definitions) {
|
||||||
|
return definitions.map((definition) => {
|
||||||
|
const found = ((response || {}).series || []).find((s) => s.metric === definition.metric);
|
||||||
|
return { name: definition.label,
|
||||||
|
points: ((found || {}).points || []).map((p) => ({ ts: p.ts, value: p.value })) };
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function chartsOpen(key) {
|
||||||
|
const card = state.cards.get(key);
|
||||||
|
const box = card && card.querySelector('[data-k="charts"]');
|
||||||
|
return !!(box && box.open);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Opening paints from the response this card already has, so the first expand needs no poll. */
|
||||||
|
function chartsToggled(key) {
|
||||||
|
if (!chartsOpen(key) || !state.snapshot) return;
|
||||||
|
const cached = state.history.get(key);
|
||||||
|
if (cached) drawCardCharts(key, cached);
|
||||||
|
else renderCardCharts(key, state.snapshot);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The paint pass, and the only place a `LineChart` here is built or drawn. */
|
||||||
|
function drawCardCharts(key, response) {
|
||||||
|
if (!chartsOpen(key) || !state.snapshot) return;
|
||||||
|
const card = state.cards.get(key);
|
||||||
|
if (!card) return;
|
||||||
|
const definitions = cardStreams(state.snapshot, key);
|
||||||
|
if (!definitions.length) return;
|
||||||
|
const chart = cardChart(key);
|
||||||
|
if (!chart) return;
|
||||||
|
const series = seriesFor(response, definitions);
|
||||||
|
chart.opts.xSpan = state.range;
|
||||||
|
chart.opts.area = false;
|
||||||
|
chart.setData(series);
|
||||||
|
// The legend names the lines the plot actually drew. A pipe whose join published no sample has
|
||||||
|
// no line, and a permanent `n/a` row for it is the dead-cell anti-pattern the vLLM slot grid
|
||||||
|
// avoids; `LineChart`'s own "no samples yet" says the same thing once when none of them did.
|
||||||
|
// `setData` stamps every series with the colour its path uses, so a swatch still matches its
|
||||||
|
// line after the filter instead of sliding to the next one in the palette.
|
||||||
|
const plotted = chart.series.filter((s) => s.points && s.points.length);
|
||||||
|
DashCharts.renderLegend(card.querySelector('[data-legend="conns"]'), plotted, COUNT_FORMAT,
|
||||||
|
false);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* One card's history: one request, named by that card's own pipes.
|
||||||
|
*
|
||||||
|
* The request is per card per refresh whether or not the disclosure is open — the `/system`
|
||||||
|
* shape, whose cached response is what lets the first expand paint from data in hand instead of
|
||||||
|
* waiting a poll — while the *plot* is built only while it is open (`drawCardCharts`).
|
||||||
|
*/
|
||||||
|
async function renderCardCharts(key, snapshot) {
|
||||||
|
const definitions = cardStreams(snapshot, key);
|
||||||
|
if (!definitions.length) return;
|
||||||
|
const range = state.range;
|
||||||
|
const query = `/api/gateway/history?hosts=${encodeURIComponent(snapshot.host || '')}`
|
||||||
|
+ `&metrics=${encodeURIComponent(definitions.map((d) => d.metric).join(','))}`
|
||||||
|
+ `&range=${range}`;
|
||||||
|
const response = await fetchJSON(query).catch(() => null);
|
||||||
|
if (!response || !state.cards.has(key)) return; // the card went away mid-flight
|
||||||
|
state.history.set(key, response);
|
||||||
|
drawCardCharts(key, response);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- the page header -------------------------------------------------------------
|
||||||
|
|
||||||
|
/**
|
||||||
|
* What the pipes on one side carry: the sum of the ones that answered, plus how many did not.
|
||||||
|
*
|
||||||
|
* This is the page's only arithmetic and it is a sum of published gauges — adding two counts is
|
||||||
|
* neither a rate nor a verdict. A pipe whose join could not decide contributes nothing and is
|
||||||
|
* counted in `unknown`, which is `/api/summary`'s `partial` flag drawn rather than a second
|
||||||
|
* opinion: the sockets behind it are already inside the `unmatched` column beside it, so
|
||||||
|
* `attributed + unmatched` stays the kernel's number and no figure on the row is invented. The
|
||||||
|
* marker is the whole point — a total that quietly left unknowns out reads as the whole number.
|
||||||
|
*/
|
||||||
|
function sumConnections(snapshot, side) {
|
||||||
|
const pipes = snapshot[side] || [];
|
||||||
|
if (!pipes.length) return null;
|
||||||
|
let attributed = 0;
|
||||||
|
let unknown = 0;
|
||||||
|
for (const pipe of pipes) {
|
||||||
|
if (typeof pipe.connections === 'number' && isFinite(pipe.connections)) {
|
||||||
|
attributed += pipe.connections;
|
||||||
|
} else {
|
||||||
|
unknown += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Nothing answered is not "nothing carrying": with every pipe unknown the side has no number at
|
||||||
|
// all, and `0` here would be a claim about traffic the join never measured (rule 2).
|
||||||
|
return unknown === pipes.length ? null : { attributed, unknown, total: pipes.length };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* One reconciliation cell: its number, the shared `.stale` dim while it is a partial sum, and the
|
||||||
|
* tooltip naming what is missing — `/llm`'s fleet-block idiom for a total that left something out.
|
||||||
|
*/
|
||||||
|
function setTally(id, sum) {
|
||||||
|
const el = $(id);
|
||||||
|
setTxt(el, sum == null ? null : sum.attributed);
|
||||||
|
const metric = el.closest('.metric');
|
||||||
|
if (metric) metric.classList.toggle('stale', !!sum && sum.unknown > 0);
|
||||||
|
el.title = sum == null ? 'the join did not measure this side'
|
||||||
|
: `${sum.attributed} of ${sum.total} pipes answered${sum.unknown
|
||||||
|
? `; ${sum.unknown} could not be told apart, and their connections are under unmatched`
|
||||||
|
: ''}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderHeader(snapshot) {
|
||||||
|
const sockets = snapshot.sockets || {};
|
||||||
|
const un = snapshot.unmatched || {};
|
||||||
|
const unmatched = un.inbound == null && un.outbound == null
|
||||||
|
? null : (un.inbound || 0) + (un.outbound || 0);
|
||||||
|
setTxt($('#t-listeners'), countText(sockets.listens));
|
||||||
|
setTxt($('#t-established'), countText(sockets.established));
|
||||||
|
setTally('#t-inbound', sumConnections(snapshot, 'inbound'));
|
||||||
|
setTally('#t-outbound', sumConnections(snapshot, 'outbound'));
|
||||||
|
setTxt($('#t-unmatched'), countText(unmatched));
|
||||||
|
markStale($('#tallies'), dimmed(snapshot));
|
||||||
|
$('#sub').textContent = `${snapshot.host || 'gateway'} · agent `
|
||||||
|
+ `${snapshot.agent_version || 'n/a'} · `
|
||||||
|
+ `${(snapshot.inbound || []).length} inbound pipes · `
|
||||||
|
+ `${(snapshot.outbound || []).length} outbound`;
|
||||||
|
const banner = $('#gate-banner');
|
||||||
|
const why = snapshot.why || null;
|
||||||
|
banner.hidden = why == null;
|
||||||
|
if (why != null) setTxt(banner, why);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- snapshot ----------------------------------------------------------------------
|
||||||
|
|
||||||
|
async function refresh() {
|
||||||
|
let snapshot;
|
||||||
|
try {
|
||||||
|
snapshot = await fetchJSON('/api/gateway/snapshot');
|
||||||
|
} catch (err) {
|
||||||
|
// Two different refusals. `404 {configured: false}` is the unconfigured arm, answered before
|
||||||
|
// anything was dialled, and the route's own sentence names the variable to set — so the setup
|
||||||
|
// block appears. Anything else is a failed read, and invariant 4's rule holds: the last good
|
||||||
|
// drawing stays on screen with the reason up top, rather than blanking the page.
|
||||||
|
$('#sub').textContent = err.message;
|
||||||
|
$('#gate-empty').hidden = err.status !== 404;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
$('#gate-empty').hidden = true;
|
||||||
|
state.snapshot = snapshot;
|
||||||
|
|
||||||
|
const states = {};
|
||||||
|
(snapshot.states || []).forEach((row) => { states[row.dashboard || row.key] = row; });
|
||||||
|
const row = states.gateway;
|
||||||
|
renderHeader(snapshot);
|
||||||
|
CARDS.forEach((definition) => renderCard(state.cards.get(definition.key), snapshot, row));
|
||||||
|
renderCard(state.cards.get('unmatched'), snapshot, row);
|
||||||
|
|
||||||
|
const visible = CARDS.filter((definition) => !state.cards.get(definition.key).hidden)
|
||||||
|
.map((definition) => definition.key);
|
||||||
|
pruneCards(visible);
|
||||||
|
await Promise.all(visible.map((key) => renderCardCharts(key, snapshot)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Clear the row and chart state of every family that left the payload. */
|
||||||
|
function pruneCards(visible) {
|
||||||
|
CARDS.forEach((definition) => {
|
||||||
|
if (visible.includes(definition.key)) return;
|
||||||
|
const card = state.cards.get(definition.key);
|
||||||
|
if (card) {
|
||||||
|
card.hidden = true;
|
||||||
|
const rows = state.rows.get(definition.key);
|
||||||
|
if (rows) rows.clear();
|
||||||
|
}
|
||||||
|
state.rows.delete(definition.key);
|
||||||
|
state.history.delete(definition.key);
|
||||||
|
// The `LineChart` instance stays: a card is hidden rather than removed, so its plot is
|
||||||
|
// built once and repainted whenever the family comes back.
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
document.addEventListener('DOMContentLoaded', async () => {
|
||||||
|
state.cards = cardElements();
|
||||||
|
try {
|
||||||
|
const config = await fetchJSON('/api/config');
|
||||||
|
const gateway = config.gateway || {};
|
||||||
|
if (Number.isFinite(gateway.interval_seconds) && gateway.interval_seconds > 0) {
|
||||||
|
state.every = gateway.interval_seconds;
|
||||||
|
}
|
||||||
|
if (gateway.enabled === false) {
|
||||||
|
// The unconfigured arm before the first request: the setup block, no cards, and no poll
|
||||||
|
// loop dialling a route that answers 404 forever (invariant 3).
|
||||||
|
$('#sub').textContent = gateway.host
|
||||||
|
? `no host named ${gateway.host} in HOST_TARGETS — set GATEWAY_HOST`
|
||||||
|
: 'no gateway configured — set GATEWAY_HOST to a HOST_TARGETS name';
|
||||||
|
$('#gate-empty').hidden = false;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
} catch (err) { /* the defaults are the shipped ones */ }
|
||||||
|
|
||||||
|
state.poller = poll(async () => { await refresh(); }, state.pollMs);
|
||||||
|
Dash.bindShortcuts();
|
||||||
|
});
|
||||||
@@ -96,6 +96,49 @@ function renderAlerts(alerts) {
|
|||||||
}).join('');
|
}).join('');
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/* The two figures the overview's ingress line prints, in the order it prints them. Each `key` is
|
||||||
|
* a key of `/api/summary`'s `gateway` block, and `tests/test_api.py` resolves this table against
|
||||||
|
* the payload the route actually serves (the `SLOT_FIELDS` device): a rename on either side fails
|
||||||
|
* there rather than printing `n/a` beside a busy ingress. */
|
||||||
|
const GATEWAY_FIELDS = [
|
||||||
|
{ key: 'inbound', word: 'inbound' },
|
||||||
|
{ key: 'outbound', word: 'outbound' },
|
||||||
|
];
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The ingress's one line under the fleet table: the pipe totals `/api/summary` summed, linked to
|
||||||
|
* the page that draws them.
|
||||||
|
*
|
||||||
|
* Nothing here is counted in the browser and nothing here is a route. The link is the block's own
|
||||||
|
* `href` (`alerts.py`'s table is where a dashboard's page is written, the same rule the alert rows
|
||||||
|
* keep), the counts are that block's sums of the pipes, and `n/a` is what a `null` prints: an
|
||||||
|
* agent that has not answered, one that predates the proxy sections, and a socket table that did
|
||||||
|
* not reply all measured **nothing**, which is not the sentence "0 connections" (rule 2).
|
||||||
|
* `configured: false` renders no line at all — the `disks` block's invariant-3 rule: an ingress
|
||||||
|
* nobody installed gets no row here and no red one to learn to ignore.
|
||||||
|
*/
|
||||||
|
function renderGateway(gateway) {
|
||||||
|
const row = $('#fleet-gate');
|
||||||
|
const block = gateway || {};
|
||||||
|
if (block.configured !== true) {
|
||||||
|
row.hidden = true;
|
||||||
|
row.textContent = '';
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
row.hidden = false;
|
||||||
|
// the shared dim for "these are the last good numbers" (invariant 4)
|
||||||
|
row.classList.toggle('stale', block.unreachable === true);
|
||||||
|
const why = block.unreachable === true ? `${block.host} is not answering`
|
||||||
|
: block.partial ? 'a pipe could not be told from another on its port, so the total leaves it out'
|
||||||
|
: '';
|
||||||
|
const counts = GATEWAY_FIELDS
|
||||||
|
.map(({ key, word }) => `${num(block[key], 0)} ${word}`).join(' · ');
|
||||||
|
row.innerHTML = `<a href="${DashCharts.escapeHtml(String(block.href || ''))}">`
|
||||||
|
+ `${DashCharts.escapeHtml(String(block.host || ''))}</a> `
|
||||||
|
+ `${DashCharts.escapeHtml(counts)}`;
|
||||||
|
if (why) row.title = why;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The fleet table: one row per host, columns unchanged.
|
* The fleet table: one row per host, columns unchanged.
|
||||||
*
|
*
|
||||||
@@ -161,6 +204,7 @@ async function refresh() {
|
|||||||
`${live}/${total} hosts · ${engines} · ${ups} · ${problems}`;
|
`${live}/${total} hosts · ${engines} · ${ups} · ${problems}`;
|
||||||
renderAlerts(alerts);
|
renderAlerts(alerts);
|
||||||
renderFleet(hosts, system.states);
|
renderFleet(hosts, system.states);
|
||||||
|
renderGateway(summary.gateway);
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
$('#summary-line').textContent = `summary failed: ${err.message}`;
|
$('#summary-line').textContent = `summary failed: ${err.message}`;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -243,6 +243,76 @@ def disk_snapshot(agent_snapshots: dict[str, dict[str, Any]], smart_shape: Any)
|
|||||||
return build
|
return build
|
||||||
|
|
||||||
|
|
||||||
|
# ---- phase 29: the gateway page (the /gateway source, API and page) -------- #
|
||||||
|
|
||||||
|
#: The captured gateway tree, as the paths the agent's own `--root` resolution expects.
|
||||||
|
#: These are the verbatim phase-28 captures (`scripts/capture-gateway-config.sh`), so the
|
||||||
|
#: config half of a `/gateway` test is the file the real ingress runs, not a memory of it.
|
||||||
|
GATEWAY_TREE = {
|
||||||
|
"etc/haproxy/haproxy.cfg": "gateway_haproxy.cfg",
|
||||||
|
"etc/nginx/nginx.conf": "gateway_nginx_main.conf",
|
||||||
|
"etc/nginx/conf.d/http.conf": "gateway_nginx_conf_d_http.conf",
|
||||||
|
"etc/nginx/stream.d/stream.conf": "gateway_nginx_stream_d_stream.conf",
|
||||||
|
"proc/net/tcp": "gateway_proc_net_tcp.txt",
|
||||||
|
"proc/net/tcp6": "gateway_proc_net_tcp6.txt",
|
||||||
|
}
|
||||||
|
|
||||||
|
GATEWAY_MODES = ("full", "predates", "no_haproxy", "no_nginx", "sockets_silent")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def gateway_snapshot(host_agent_module: Any, tmp_path: Path) -> Any:
|
||||||
|
"""`build(mode="full")` -> a 1.2.0-style agent snapshot carrying the three proxy sections.
|
||||||
|
|
||||||
|
Not a hand-written dict: the captured configs and `/proc` tables are laid into a fake
|
||||||
|
`--root` tree and read by the **real agent `Collector`**, so the sections handed to
|
||||||
|
`sources/gateway.py` are the bytes agent 1.2.0 serves. A field rename on either side of the
|
||||||
|
wire therefore fails a source test instead of drawing an empty pipe on the page (rule 7).
|
||||||
|
|
||||||
|
Modes are the degradation arms the page has to render:
|
||||||
|
|
||||||
|
* `full` — all three sections `available: true`, live sockets from the capture
|
||||||
|
* `predates` — none of the three keys at all (an agent older than 1.2.0, or `--no-proxy`)
|
||||||
|
* `no_haproxy` / `no_nginx` — that one config file is absent: `available: false` plus the
|
||||||
|
agent's own **not found** line, the other half of the page untouched
|
||||||
|
* `sockets_silent` — neither `/proc` table readable: `available: null`, so every live count
|
||||||
|
must come back `None` and never `0`
|
||||||
|
"""
|
||||||
|
def build(mode: str = "full") -> dict[str, Any]:
|
||||||
|
if mode not in GATEWAY_MODES:
|
||||||
|
raise AssertionError(f"unknown gateway snapshot mode {mode!r}")
|
||||||
|
root = tmp_path / f"root-{mode}"
|
||||||
|
for relative, fixture in GATEWAY_TREE.items():
|
||||||
|
if not fixture:
|
||||||
|
continue
|
||||||
|
target = root / relative
|
||||||
|
target.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
target.write_text((FIXTURES / fixture).read_text())
|
||||||
|
if mode == "no_haproxy":
|
||||||
|
(root / "etc/haproxy/haproxy.cfg").unlink()
|
||||||
|
if mode == "no_nginx":
|
||||||
|
for path in (root / "etc/nginx/nginx.conf", root / "etc/nginx/conf.d/http.conf",
|
||||||
|
root / "etc/nginx/stream.d/stream.conf"):
|
||||||
|
path.unlink()
|
||||||
|
if mode == "sockets_silent":
|
||||||
|
(root / "proc/net/tcp").unlink()
|
||||||
|
(root / "proc/net/tcp6").unlink()
|
||||||
|
|
||||||
|
errors: list[str] = []
|
||||||
|
collector = host_agent_module.Collector(root)
|
||||||
|
snapshot: dict[str, Any] = {
|
||||||
|
"ok": True, "schema": 1, "ts": 1_760_000_000.0, "host": "gateway",
|
||||||
|
"agent_version": host_agent_module.AGENT_VERSION, "errors": errors,
|
||||||
|
}
|
||||||
|
if mode != "predates":
|
||||||
|
snapshot["haproxy"] = collector.haproxy(errors)
|
||||||
|
snapshot["nginx"] = collector.nginx(errors)
|
||||||
|
snapshot["sockets"] = collector.sockets(errors)
|
||||||
|
return snapshot
|
||||||
|
|
||||||
|
return build
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
def metrics_text() -> Any:
|
def metrics_text() -> Any:
|
||||||
"""metrics_text('gufo') / metrics_text('vllm') -> the captured exposition verbatim."""
|
"""metrics_text('gufo') / metrics_text('vllm') -> the captured exposition verbatim."""
|
||||||
|
|||||||
+79
-1
@@ -14,9 +14,10 @@ are the ones that make the landing page trustworthy rather than merely busy:
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import inspect
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from dashboards import alerts
|
from dashboards import alerts, severity
|
||||||
from dashboards.config import Settings
|
from dashboards.config import Settings
|
||||||
from dashboards.sources import nut
|
from dashboards.sources import nut
|
||||||
from dashboards.sources.nut_sim import NutSimulator
|
from dashboards.sources.nut_sim import NutSimulator
|
||||||
@@ -481,6 +482,83 @@ def test_a_dead_upsd_is_one_alert_and_not_its_stale_cards_too() -> None:
|
|||||||
assert [item["source"] for item in out] == ["NUT server"]
|
assert [item["source"] for item in out] == ["NUT server"]
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# the gateway page: reachability is the only voice, and a dead agent is still one line
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
def gateway_state(**kwargs: Any) -> dict[str, Any]:
|
||||||
|
"""The health record of the one `gateway` job (the `nut` shape: key `gateway`, label = host)."""
|
||||||
|
return state("gateway", "gateway", "gateway", **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_stale_gateway_job_is_a_warn_measured_in_its_own_thirty_second_poll() -> None:
|
||||||
|
"""The pipes are drawn from a 30 s poll, so 60 s of silence is two healthy polls.
|
||||||
|
|
||||||
|
Judging this source by the 10 s system interval would amber the landing page every minute for
|
||||||
|
a job behaving exactly as configured — the same mistake the `disks` rule exists to avoid, one
|
||||||
|
cadence later.
|
||||||
|
"""
|
||||||
|
settings = Settings(gateway_interval=30.0)
|
||||||
|
assert alerts.build(settings, [gateway_state(age=60.0)]) == []
|
||||||
|
out = alerts.build(settings, [gateway_state(age=100.0)])
|
||||||
|
assert out[0]["severity"] == "warn"
|
||||||
|
assert out[0]["href"] == "/gateway", "the overview's row must link to the page it describes"
|
||||||
|
assert "every 30s" in out[0]["message"], "the message prints the gateway's own interval"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_gateway_job_that_fails_while_its_host_answers_is_the_only_voice_it_has() -> None:
|
||||||
|
"""A wedged proxy/socket section is a fault nothing else can report (the `disks` rule, once more)."""
|
||||||
|
out = alerts.build(SETTINGS, [
|
||||||
|
gateway_state(ok=False, error="UpstreamError: gateway: timed out"),
|
||||||
|
state("host:gateway", "system", "gateway"),
|
||||||
|
], now=2_000.0)
|
||||||
|
assert len(out) == 1
|
||||||
|
assert out[0]["source"] == "gateway"
|
||||||
|
assert out[0]["href"] == "/gateway"
|
||||||
|
assert "timed out" in out[0]["message"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_gateway_job_that_failed_because_its_agent_is_down_is_one_line_not_two() -> None:
|
||||||
|
"""Three jobs dial one agent; the operator has one problem, so the overview says it once."""
|
||||||
|
down = {"ok": False,
|
||||||
|
"error": "UpstreamError: gateway: http://h:9120/snapshot unreachable (ConnectError)"}
|
||||||
|
out = alerts.build(SETTINGS, [
|
||||||
|
gateway_state(**down),
|
||||||
|
state("disks:gateway", "disks", "gateway", **down),
|
||||||
|
state("host:gateway", "system", "gateway", **down),
|
||||||
|
], now=2_000.0)
|
||||||
|
assert len(out) == 1, "the echo of one outage must not read as three problems"
|
||||||
|
assert out[0]["id"].startswith("gateway:host:"), out[0]["id"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_unconfigured_gateway_source_says_nothing_even_when_its_job_says_down() -> None:
|
||||||
|
"""No `HOST_TARGETS` entry named `GATEWAY_HOST` means no job — and a reported one stays quiet."""
|
||||||
|
out = alerts.build(Settings(gateway_interval=30.0),
|
||||||
|
[gateway_state(ok=False, configured=False, error="not configured")],
|
||||||
|
now=2_000.0)
|
||||||
|
assert out == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_gateway_payload_with_a_fabricated_verdict_alerts_nothing() -> None:
|
||||||
|
"""The phase's no-verdict rule, asserted as a rule and not only as an absence.
|
||||||
|
|
||||||
|
Three ways, because deleting a rule satisfies an absence-only assertion:
|
||||||
|
|
||||||
|
* `alerts.build` takes no gateway payload at all — its signature is the whole surface, so a
|
||||||
|
gateway count can reach an alert only through a threshold added to this module;
|
||||||
|
* `severity.THRESHOLDS` has no `gateway.*` row, so no such threshold could be written without
|
||||||
|
editing the one file that owns thresholds (and a busy proxy is the job, the `net.*` case);
|
||||||
|
* the module's `HREF` does carry `/gateway`, so reachability still reaches the overview.
|
||||||
|
"""
|
||||||
|
assert not {"gateway", "gateway_payload"} & set(inspect.signature(alerts.build).parameters)
|
||||||
|
assert not [name for name in severity.THRESHOLDS if name.startswith("gateway.")], (
|
||||||
|
"a connection count is not a graded number")
|
||||||
|
assert alerts.HREF["gateway"] == "/gateway"
|
||||||
|
assert alerts.AGENT_ECHO_DASHBOARDS == ("disks", "gateway")
|
||||||
|
# and an answering gateway with nothing else going on is the all clear
|
||||||
|
assert alerts.build(Settings(gateway_interval=30.0), [gateway_state()], [], [], None, [],
|
||||||
|
now=2_000.0) == []
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# configuration
|
# configuration
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
|
|||||||
+1406
-49
File diff suppressed because it is too large
Load Diff
+13
-4
@@ -56,7 +56,12 @@ CURL_HEADERS = {"accept": "*/*", "user-agent": "curl/8.5.0"}
|
|||||||
PROTECTED_API = (
|
PROTECTED_API = (
|
||||||
"/api/config", "/api/events", "/api/metrics", "/api/history", "/api/summary",
|
"/api/config", "/api/events", "/api/metrics", "/api/history", "/api/summary",
|
||||||
"/api/system/snapshot", "/api/system/history", "/api/disks/snapshot",
|
"/api/system/snapshot", "/api/system/history", "/api/disks/snapshot",
|
||||||
"/api/disks/history", "/api/llm/snapshot", "/api/llm/history", "/api/ups/snapshot",
|
"/api/disks/history",
|
||||||
|
# Phase 29: the ingress's pipes. `…/snapshot` carries the gateway's whole routing map — every
|
||||||
|
# domain it serves and every address it dials — so it is listed here for the same reason the
|
||||||
|
# bundle download is, whatever the payload happens to hold today.
|
||||||
|
"/api/gateway/snapshot", "/api/gateway/history",
|
||||||
|
"/api/llm/snapshot", "/api/llm/history", "/api/ups/snapshot",
|
||||||
"/api/ups/history", "/api/ups/commands", "/api/auth/status",
|
"/api/ups/history", "/api/ups/commands", "/api/auth/status",
|
||||||
# Phase 15: the certificate authority's own surface. `/api/certs/<name>/bundle` is every
|
# Phase 15: the certificate authority's own surface. `/api/certs/<name>/bundle` is every
|
||||||
# agent's private key, so "it needs a session" is the whole point of listing it here.
|
# agent's private key, so "it needs a session" is the whole point of listing it here.
|
||||||
@@ -68,7 +73,11 @@ PROTECTED_API = (
|
|||||||
#: thing that must never regress is a new route answering anonymously. `/certs` is the sharper
|
#: thing that must never regress is a new route answering anonymously. `/certs` is the sharper
|
||||||
#: case of the two — an anonymous caller reaching it would see the CA's serials and every host
|
#: case of the two — an anonymous caller reaching it would see the CA's serials and every host
|
||||||
#: the dashboard dials.
|
#: the dashboard dials.
|
||||||
PROTECTED_PAGES = ("/", "/system", "/disks", "/llm", "/ups", "/settings", "/certs")
|
#: `/gateway` (phase 29 task 04) is the sixth dashboard: it prints this fleet's ingress map —
|
||||||
|
#: every listening port, every `server_name`, every internal target — so it is gated by being a
|
||||||
|
#: page and nothing else.
|
||||||
|
PROTECTED_PAGES = ("/", "/system", "/disks", "/llm", "/ups", "/gateway", "/settings",
|
||||||
|
"/certs")
|
||||||
|
|
||||||
|
|
||||||
#: The one write this app has, with the fields the route needs (the UPS is unconfigured, so a
|
#: The one write this app has, with the fields the route needs (the UPS is unconfigured, so a
|
||||||
@@ -1598,8 +1607,8 @@ def test_the_served_route_table_is_the_surface_the_gate_describes(anon: TestClie
|
|||||||
pages = sorted(str(route.path) for route in served_routes(anon)
|
pages = sorted(str(route.path) for route in served_routes(anon)
|
||||||
if isinstance(getattr(route, "path", None), str)
|
if isinstance(getattr(route, "path", None), str)
|
||||||
and not str(route.path).startswith(("/api", "/static")))
|
and not str(route.path).startswith(("/api", "/static")))
|
||||||
assert pages == ["/", "/certs", "/disks", "/llm", "/login", "/settings", "/system",
|
assert pages == ["/", "/certs", "/disks", "/gateway", "/llm", "/login", "/settings",
|
||||||
"/ups"]
|
"/system", "/ups"]
|
||||||
for path in pages:
|
for path in pages:
|
||||||
response = anon.get(path, headers=BROWSER_HEADERS)
|
response = anon.get(path, headers=BROWSER_HEADERS)
|
||||||
if path == "/login":
|
if path == "/login":
|
||||||
|
|||||||
@@ -28,11 +28,13 @@ from dashboards.collectors import (
|
|||||||
disks_jobs,
|
disks_jobs,
|
||||||
fetch_disk_snapshot,
|
fetch_disk_snapshot,
|
||||||
fetch_engine,
|
fetch_engine,
|
||||||
|
fetch_gateway_snapshot,
|
||||||
fetch_host_snapshot,
|
fetch_host_snapshot,
|
||||||
fetch_snapshot,
|
fetch_snapshot,
|
||||||
|
gateway_job,
|
||||||
host_jobs,
|
host_jobs,
|
||||||
)
|
)
|
||||||
from dashboards.config import Target
|
from dashboards.config import Settings, Target
|
||||||
from dashboards.poller import Result
|
from dashboards.poller import Result
|
||||||
from dashboards.rates import RateEstimator
|
from dashboards.rates import RateEstimator
|
||||||
from dashboards.sources import nut
|
from dashboards.sources import nut
|
||||||
@@ -409,6 +411,80 @@ async def test_smart_addon_errors_reach_the_disk_payload_verbatim(
|
|||||||
assert "degraded" not in system.payload, "and /system stays unbothered by it"
|
assert "degraded" not in system.payload, "and /system stays unbothered by it"
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# gateway topology (the /gateway source's fetch half, phase 29)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
async def test_the_gateway_fetch_reads_the_same_agent_snapshot_as_the_other_two(
|
||||||
|
gateway_snapshot: Any) -> None:
|
||||||
|
"""One endpoint, three readers: the pipes come out of `/snapshot` and nothing else.
|
||||||
|
|
||||||
|
The gateway job has no URL of its own (`GATEWAY_HOST` is a *name*), so this is the pin that
|
||||||
|
keeps it that way — and that the projection's own honesty rules survive the trip through the
|
||||||
|
collector: gauges under the `gateway` family, **no** events (a count has no transition worth a
|
||||||
|
line), and no `window_seconds` for a number that was never diffed (rule 2).
|
||||||
|
"""
|
||||||
|
body = gateway_snapshot("full")
|
||||||
|
seen: list[str] = []
|
||||||
|
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
assert request.url.path == "/snapshot", "the gateway page has no endpoint of its own"
|
||||||
|
seen.append(request.url.path)
|
||||||
|
return httpx.Response(200, json=body)
|
||||||
|
|
||||||
|
target = Target("gateway", "http://agent.test")
|
||||||
|
async with client_for(handler) as client:
|
||||||
|
result = await fetch_gateway_snapshot(client, target, timeout=10)
|
||||||
|
assert seen == ["/snapshot"]
|
||||||
|
assert result.payload["host"] == "gateway"
|
||||||
|
assert result.payload["supported"] is True
|
||||||
|
assert result.payload["inbound"], "the configured pipes arrive with the snapshot"
|
||||||
|
assert result.samples and all(sample.source == "gateway" for sample in result.samples), (
|
||||||
|
"the gauges are their own sample family, not `host`'s")
|
||||||
|
assert result.events == [], "a connection count has no transition to report"
|
||||||
|
assert "window_seconds" not in result.payload, "a count is not a rate"
|
||||||
|
|
||||||
|
|
||||||
|
async def test_the_gateway_poll_carries_its_own_timeout_and_never_disables_one() -> None:
|
||||||
|
"""The third budget on one endpoint: the client default until the job asks for its own.
|
||||||
|
|
||||||
|
Same rule the `disks` poll was pinned with — a literal `None` means *no timeout at all* to
|
||||||
|
httpx, which would hang the only poller this source has (invariant 4) — and the reason
|
||||||
|
`GATEWAY_TIMEOUT_SECONDS` is a setting rather than a constant inherited from `system`.
|
||||||
|
"""
|
||||||
|
def handler(request: httpx.Request) -> httpx.Response:
|
||||||
|
return httpx.Response(200, json={"ok": True, "ts": 1.0, "errors": []})
|
||||||
|
|
||||||
|
client = TimeoutRecordingClient(transport=httpx.MockTransport(handler),
|
||||||
|
timeout=4.0, base_url="http://agent.test")
|
||||||
|
target = Target("gateway", "http://agent.test")
|
||||||
|
async with client:
|
||||||
|
await fetch_gateway_snapshot(client, target)
|
||||||
|
assert client.timeouts[-1] is httpx.USE_CLIENT_DEFAULT
|
||||||
|
await fetch_gateway_snapshot(client, target, timeout=10)
|
||||||
|
assert client.timeouts[-1] == 10, "the gateway poll waits on its own budget"
|
||||||
|
assert all(timeout is not None for timeout in client.timeouts)
|
||||||
|
|
||||||
|
|
||||||
|
async def test_the_gateway_job_is_one_job_named_after_the_host_it_reads() -> None:
|
||||||
|
"""`key`/`dashboard`/`label`/`interval` at the factory, and `None` for a name nobody dials.
|
||||||
|
|
||||||
|
The label is what an alert and an event print and what `alerts.py` uses to find the `host:` job
|
||||||
|
reading the same agent, so it is the host's name and not the source's — while the key stays the
|
||||||
|
single word `gateway` because there is exactly one such job per deployment.
|
||||||
|
"""
|
||||||
|
settings = Settings(host_targets=(Target("gateway", "https://gw.test"),),
|
||||||
|
gateway_interval=45.0, gateway_timeout=7.0)
|
||||||
|
entry = AgentTls(verify="/ca.pem", cert=("/c.pem", "/k.pem"), status=lambda: certs.ACTIVE)
|
||||||
|
async with httpx.AsyncClient() as client:
|
||||||
|
job = gateway_job(settings, client, tls={"gateway": entry})
|
||||||
|
assert job is not None
|
||||||
|
assert (job.key, job.dashboard, job.label) == ("gateway", "gateway", "gateway")
|
||||||
|
assert job.interval == 45.0
|
||||||
|
assert job.enabled is True, "a job that exists is a job that runs"
|
||||||
|
missing = Settings(host_targets=(Target("deskwork", "http://desk.test"),))
|
||||||
|
assert gateway_job(missing, httpx.AsyncClient()) is None, "unconfigured is no task at all"
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
# TLS: the dashboard dials a protected exporter with its own client certificate
|
# TLS: the dashboard dials a protected exporter with its own client certificate
|
||||||
# --------------------------------------------------------------------------- #
|
# --------------------------------------------------------------------------- #
|
||||||
|
|||||||
@@ -87,6 +87,54 @@ def test_settings_default_to_no_sources_configured() -> None:
|
|||||||
assert settings.any_source_enabled is False
|
assert settings.any_source_enabled is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_gateway_source_names_the_ingress_host_and_its_own_cadence() -> None:
|
||||||
|
"""Phase 29: one `HOST_TARGETS` name is the gateway, and it is polled on its own clock.
|
||||||
|
|
||||||
|
`gateway_host` defaults to the fleet's real name so installing agent 1.2.0 on a host called
|
||||||
|
`gateway` is the whole configuration — but the **default is not a target**: with no
|
||||||
|
`HOST_TARGETS` entry of that name the source is unconfigured, `/api/gateway/*` 404s without
|
||||||
|
dialling, and the page says so (invariant 3). The interval sits between the system poll and
|
||||||
|
SMART because a connection count is the shape of half a minute, not of ten seconds.
|
||||||
|
"""
|
||||||
|
settings = Settings()
|
||||||
|
assert settings.gateway_host == "gateway"
|
||||||
|
assert settings.gateway_interval == 30.0
|
||||||
|
assert settings.gateway_timeout == 10.0
|
||||||
|
assert settings.gateway_enabled is False
|
||||||
|
assert settings.any_source_enabled is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_gateway_enabled_is_true_only_for_a_name_that_is_a_target() -> None:
|
||||||
|
"""A name with no agent behind it is unconfigured, not broken — the `nut_enabled` shape."""
|
||||||
|
def with_targets(raw: str, name: str = "gateway") -> bool:
|
||||||
|
targets, _problems = parse_targets(raw)
|
||||||
|
return Settings(host_targets=tuple(targets), gateway_host=name).gateway_enabled
|
||||||
|
|
||||||
|
assert with_targets("gateway=https://gw.example.net:9120") is True
|
||||||
|
assert with_targets(
|
||||||
|
"deskwork=https://d.example.net:9120 gateway=https://gw.example.net:9120") is True
|
||||||
|
assert with_targets("deskwork=https://d.example.net:9120") is False
|
||||||
|
assert with_targets("router=https://gw.example.net:9120", name="router") is True
|
||||||
|
assert with_targets("Gateway=https://gw.example.net:9120") is False, ("names are lowercase")
|
||||||
|
assert with_targets("") is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_gateway_settings_come_from_the_environment(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""`GATEWAY_HOST=router` is how a deployment whose ingress is called something else opts in."""
|
||||||
|
monkeypatch.setenv("GATEWAY_HOST", "router")
|
||||||
|
monkeypatch.setenv("GATEWAY_INTERVAL_SECONDS", "15")
|
||||||
|
monkeypatch.setenv("GATEWAY_TIMEOUT_SECONDS", "20")
|
||||||
|
reloaded = importlib.reload(config)
|
||||||
|
try:
|
||||||
|
assert reloaded.SETTINGS.gateway_host == "router"
|
||||||
|
assert reloaded.SETTINGS.gateway_interval == 15.0
|
||||||
|
assert reloaded.SETTINGS.gateway_timeout == 20.0
|
||||||
|
finally:
|
||||||
|
for name in ("GATEWAY_HOST", "GATEWAY_INTERVAL_SECONDS", "GATEWAY_TIMEOUT_SECONDS"):
|
||||||
|
monkeypatch.delenv(name)
|
||||||
|
importlib.reload(config)
|
||||||
|
|
||||||
|
|
||||||
def test_the_disk_source_defaults_to_a_five_minute_cadence() -> None:
|
def test_the_disk_source_defaults_to_a_five_minute_cadence() -> None:
|
||||||
"""SMART is not a per-second signal (phase 09), and the fetch waits for the agent's
|
"""SMART is not a per-second signal (phase 09), and the fetch waits for the agent's
|
||||||
own 20 s smartctl budget. Both are env-only, like every other interval."""
|
own 20 s smartctl budget. Both are env-only, like every other interval."""
|
||||||
|
|||||||
@@ -0,0 +1,718 @@
|
|||||||
|
"""`sources/gateway.py`: the config + socket join that becomes the `/gateway` pipes.
|
||||||
|
|
||||||
|
The shape half of these tests runs the **captured** gateway configs and `/proc` tables through
|
||||||
|
the real agent `Collector` (the `gateway_snapshot` fixture), so a rename of a section key on
|
||||||
|
either side of the wire fails here rather than drawing an empty pipe on a live page. The
|
||||||
|
arithmetic half uses synthetic sections, because the point of those tests is which socket lands
|
||||||
|
in which pipe and what the source refuses to decide — facts a capture states once and a
|
||||||
|
re-capture may lawfully change (the same rule `tests/test_agent.py` holds for the socket tables).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import socket as socket_module
|
||||||
|
from collections.abc import Mapping, Sequence
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from dashboards.sources import gateway
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# synthetic builders: a section exactly as the agent serves it, minus the file
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def bind(port: int | None, addr: str = "::") -> dict[str, Any]:
|
||||||
|
return {"addr": addr, "port": port}
|
||||||
|
|
||||||
|
|
||||||
|
def haproxy_section(frontends: Sequence[dict[str, Any]],
|
||||||
|
backends: Sequence[dict[str, Any]] = (), *,
|
||||||
|
available: bool | None = True) -> dict[str, Any]:
|
||||||
|
return {"available": available, "config": "/etc/haproxy/haproxy.cfg",
|
||||||
|
"frontends": list(frontends), "backends": list(backends)}
|
||||||
|
|
||||||
|
|
||||||
|
def frontend(name: str, ports: Sequence[int | None], *, default: str | None = None,
|
||||||
|
use: Sequence[str] = ()) -> dict[str, Any]:
|
||||||
|
return {"name": name, "binds": [bind(port) for port in ports],
|
||||||
|
"default_backend": default, "backends": list(use)}
|
||||||
|
|
||||||
|
|
||||||
|
def backend(name: str, servers: Sequence[dict[str, Any]]) -> dict[str, Any]:
|
||||||
|
return {"name": name, "servers": list(servers)}
|
||||||
|
|
||||||
|
|
||||||
|
def server(name: str, target: str | None, port: int | None = None) -> dict[str, Any]:
|
||||||
|
return {"name": name, "target": target, "port": port}
|
||||||
|
|
||||||
|
|
||||||
|
def nginx_section(servers: Sequence[dict[str, Any]] = (), maps: Sequence[dict[str, Any]] = (), *,
|
||||||
|
available: bool | None = True) -> dict[str, Any]:
|
||||||
|
return {"available": available, "config": "/etc/nginx/nginx.conf", "files": [],
|
||||||
|
"servers": list(servers), "upstreams": [], "maps": list(maps)}
|
||||||
|
|
||||||
|
|
||||||
|
def nginx_server(ports: Sequence[int | None], names: Sequence[str] = (),
|
||||||
|
passes: Sequence[dict[str, Any]] = ()) -> dict[str, Any]:
|
||||||
|
return {"listen": [bind(port, addr="") for port in ports], "server_names": list(names),
|
||||||
|
"proxy_passes": list(passes)}
|
||||||
|
|
||||||
|
|
||||||
|
def proxy_pass(target: tuple[str, int] | None, *, variable: str | None = None,
|
||||||
|
location: str | None = "/") -> dict[str, Any]:
|
||||||
|
host, port = target if target else (None, None)
|
||||||
|
return {"location": location, "scheme": "http" if target else None,
|
||||||
|
"target": None if host is None else {"host": host, "port": port},
|
||||||
|
"variable": variable}
|
||||||
|
|
||||||
|
|
||||||
|
def mapping_block(variable: str,
|
||||||
|
entries: Sequence[tuple[str, tuple[str, int | None] | None]]) -> dict[str, Any]:
|
||||||
|
return {"variable": variable,
|
||||||
|
"entries": [{"key": key,
|
||||||
|
"target": None if target is None else {"host": target[0],
|
||||||
|
"port": target[1]}}
|
||||||
|
for key, target in entries]}
|
||||||
|
|
||||||
|
|
||||||
|
def sockets_section(listens: Sequence[dict[str, Any]] = (),
|
||||||
|
established: Sequence[dict[str, Any]] = (), *,
|
||||||
|
available: bool | None = True) -> dict[str, Any]:
|
||||||
|
return {"available": available, "listens": list(listens), "established": list(established)}
|
||||||
|
|
||||||
|
|
||||||
|
def listen(port: int, addr: str = "0.0.0.0", family: str = "v4") -> dict[str, Any]:
|
||||||
|
return {"family": family, "addr": addr, "port": port}
|
||||||
|
|
||||||
|
|
||||||
|
def conn(local: tuple[str, int], remote: tuple[str, int], *,
|
||||||
|
family: str = "v4") -> dict[str, Any]:
|
||||||
|
return {"family": family, "local": {"addr": local[0], "port": local[1]},
|
||||||
|
"remote": {"addr": remote[0], "port": remote[1]}}
|
||||||
|
|
||||||
|
|
||||||
|
def snapshot(*, haproxy: dict[str, Any] | None = None, nginx: dict[str, Any] | None = None,
|
||||||
|
sockets: dict[str, Any] | None = None, errors: Sequence[str] = (),
|
||||||
|
agent_version: str | None = "1.2.0") -> dict[str, Any]:
|
||||||
|
"""The document one agent `/snapshot` would answer, with only what a test names."""
|
||||||
|
doc: dict[str, Any] = {"ok": True, "schema": 1, "ts": 1_760_000_000.0, "host": "gateway",
|
||||||
|
"errors": list(errors)}
|
||||||
|
if agent_version is not None:
|
||||||
|
doc["agent_version"] = agent_version
|
||||||
|
if haproxy is not None:
|
||||||
|
doc["haproxy"] = haproxy
|
||||||
|
if nginx is not None:
|
||||||
|
doc["nginx"] = nginx
|
||||||
|
if sockets is not None:
|
||||||
|
doc["sockets"] = sockets
|
||||||
|
return doc
|
||||||
|
|
||||||
|
|
||||||
|
def pipe_by_port(payload: Mapping[str, Any], port: int) -> dict[str, Any]:
|
||||||
|
for pipe in payload["inbound"]:
|
||||||
|
if pipe["port"] == port:
|
||||||
|
return pipe
|
||||||
|
raise AssertionError(f"no inbound pipe on port {port}: {[p['port'] for p in payload['inbound']]}")
|
||||||
|
|
||||||
|
|
||||||
|
def pipe_to(payload: Mapping[str, Any], host: str, port: int | None) -> dict[str, Any]:
|
||||||
|
for pipe in payload["outbound"]:
|
||||||
|
if pipe["target"]["host"] == host and pipe["target"]["port"] == port:
|
||||||
|
return pipe
|
||||||
|
raise AssertionError(f"no outbound pipe to {host}:{port}")
|
||||||
|
|
||||||
|
|
||||||
|
def walk_keys(node: Any) -> set[str]:
|
||||||
|
"""Every key name anywhere in a payload — the device the "no severity / no window" pins use."""
|
||||||
|
if isinstance(node, Mapping):
|
||||||
|
found = set(node.keys())
|
||||||
|
for value in node.values():
|
||||||
|
found |= walk_keys(value)
|
||||||
|
return found
|
||||||
|
if isinstance(node, Sequence) and not isinstance(node, str | bytes):
|
||||||
|
return set().union(*(walk_keys(item) for item in node)) if node else set()
|
||||||
|
return set()
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# the captured gateway: every configured pipe, from the real agent code path
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def test_the_captured_configs_yield_every_configured_pipe(gateway_snapshot: Any) -> None:
|
||||||
|
"""Six haproxy frontends, four nginx listen ports, the stream map's fan-out — all present."""
|
||||||
|
payload = gateway.parse(gateway_snapshot("full"), "gateway")
|
||||||
|
assert payload["configured"] is True
|
||||||
|
assert payload["supported"] is True
|
||||||
|
assert payload["why"] is None, "a gateway whose every section answered gets no why-line"
|
||||||
|
assert payload["sections"] == {"haproxy": True, "nginx": True, "sockets": True}
|
||||||
|
|
||||||
|
haproxy_ports = [row["port"] for row in payload["haproxy"]["frontends"]]
|
||||||
|
assert haproxy_ports == [22, 2000, 5201, 8080, 25565, 25566], "config order, one row per bind"
|
||||||
|
assert {row["backend"] for row in payload["haproxy"]["frontends"]} == {
|
||||||
|
"gitea-2222", "sec-gitea-2000", "iperf3-5201", "unifi-external-8080",
|
||||||
|
"testcraft-25565", "nimcraft-25566"}
|
||||||
|
|
||||||
|
assert [row["port"] for row in payload["nginx"]["servers"]] == [80, 443, 4443, 8443]
|
||||||
|
names = {row["port"]: row["server_names"] for row in payload["nginx"]["servers"]}
|
||||||
|
assert names[80] == ["_"], "the catch-all is the config's own text, kept verbatim"
|
||||||
|
assert names[443] == [], "the stream server routes by SNI, so it names no domain of its own"
|
||||||
|
assert "nextcloud.reeseapps.com" in names[4443] and "gitea.reeseapps.com" in names[4443], (
|
||||||
|
"five vhosts share 4443 and their domains belong to the one pipe for that port")
|
||||||
|
|
||||||
|
assert payload["nginx"]["map_targets"], "the stream map is the fan-out the page draws"
|
||||||
|
assert len(payload["nginx"]["map_targets"]) == 17
|
||||||
|
homepage = next(row for row in payload["nginx"]["map_targets"]
|
||||||
|
if row["key"] == "homepage.reeseapps.com")
|
||||||
|
assert homepage == {"key": "homepage.reeseapps.com",
|
||||||
|
"host": "reeseapps-homepage.reeselink.com", "port": 443}
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_source_reads_the_keys_the_agent_actually_publishes(gateway_snapshot: Any,
|
||||||
|
host_agent_module: Any) -> None:
|
||||||
|
"""The coupling is pinned: `binds`/`port`, `servers`/`target`, `listen`/`server_names`.
|
||||||
|
|
||||||
|
The fixture runs the real parser, so this compares the source's output against the agent's
|
||||||
|
own document rather than against a copy of its field names — which is what makes a rename
|
||||||
|
fail here instead of printing `n/a` beside a healthy gateway (AGENTS.md rule 7).
|
||||||
|
"""
|
||||||
|
snap = gateway_snapshot("full")
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
agent_bind_ports = {bind["port"] for frontend in snap["haproxy"]["frontends"]
|
||||||
|
for bind in frontend["binds"]}
|
||||||
|
assert {row["port"] for row in payload["haproxy"]["frontends"]} == agent_bind_ports
|
||||||
|
agent_servers = {(row["name"], row["target"], row["port"])
|
||||||
|
for block in snap["haproxy"]["backends"] for row in block["servers"]}
|
||||||
|
assert {(pipe["target"]["name"], pipe["target"]["host"], pipe["target"]["port"])
|
||||||
|
for pipe in payload["outbound"] if pipe["source"] == "haproxy"} == agent_servers
|
||||||
|
agent_listen_ports = {listen["port"] for server in snap["nginx"]["servers"]
|
||||||
|
for listen in server["listen"]}
|
||||||
|
assert {row["port"] for row in payload["nginx"]["servers"]} == agent_listen_ports
|
||||||
|
# and the three section names themselves, as the snapshot publishes them
|
||||||
|
assert {"haproxy", "nginx", "sockets"} <= set(snap)
|
||||||
|
assert host_agent_module.AGENT_VERSION == "1.2.0", "the sections these keys belong to"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_live_counts_reconcile_with_the_socket_table(gateway_snapshot: Any) -> None:
|
||||||
|
"""Every ESTABLISHED row is counted exactly once — in a pipe, or in `unmatched`.
|
||||||
|
|
||||||
|
This is the property phase 29 task 06 checks against a direct `/proc/net/tcp` read; pinning
|
||||||
|
it on the capture means the arithmetic is already right before anything is dialled.
|
||||||
|
"""
|
||||||
|
snap = gateway_snapshot("full")
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
established = len(snap["sockets"]["established"])
|
||||||
|
in_pipes = sum(pipe["connections"] for pipe in payload["inbound"])
|
||||||
|
out_pipes = sum(pipe["connections"] or 0 for pipe in payload["outbound"])
|
||||||
|
unmatched = payload["unmatched"]
|
||||||
|
assert payload["sockets"] == {"listens": len(snap["sockets"]["listens"]),
|
||||||
|
"established": established}
|
||||||
|
assert in_pipes + out_pipes + unmatched["inbound"] + unmatched["outbound"] == established
|
||||||
|
# the ambiguous fan-out is inside `unmatched.outbound`, not on top of it
|
||||||
|
assert sum(group["connections"] for group in unmatched["groups"]) <= unmatched["outbound"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_listening_port_no_config_claims_stays_visible(gateway_snapshot: Any) -> None:
|
||||||
|
"""Phase 28's "an unclaimed socket stays visible as one", counted, not silently dropped."""
|
||||||
|
snap = gateway_snapshot("full")
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
live = {row["port"] for row in snap["sockets"]["listens"]}
|
||||||
|
configured = {pipe["port"] for pipe in payload["inbound"]}
|
||||||
|
assert payload["unmatched"]["other"] == len(live - configured)
|
||||||
|
# the agent's own port is dialled by nobody in either config, so it is always the example
|
||||||
|
assert 9120 in live - configured
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_map_fanout_becomes_pipes_labelled_by_domain(gateway_snapshot: Any) -> None:
|
||||||
|
"""Every map entry is a pipe the page can draw, and `$var` never becomes a host."""
|
||||||
|
payload = gateway.parse(gateway_snapshot("full"), "gateway")
|
||||||
|
map_pipes = [pipe for pipe in payload["outbound"] if pipe["source"] == "map"]
|
||||||
|
assert len(map_pipes) == len(payload["nginx"]["map_targets"])
|
||||||
|
assert not [pipe for pipe in payload["outbound"]
|
||||||
|
if pipe["target"]["host"] is None or "$" in str(pipe["target"]["host"])], (
|
||||||
|
"a proxy_pass $var carries no address, so it must not become a pipe of its own")
|
||||||
|
brain = pipe_to(payload, "reeseapps-brain-of-reese.reeselink.com", 443)
|
||||||
|
assert brain["target"]["by_port"] is True, "a name can only ever be matched on its port"
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# the inbound join
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def test_a_socket_lands_in_the_frontend_that_owns_its_local_port() -> None:
|
||||||
|
snap = snapshot(
|
||||||
|
haproxy=haproxy_section([frontend("ssh", [22], default="ssh-be")]),
|
||||||
|
nginx=nginx_section([nginx_server([443], ["a.example.test"])]),
|
||||||
|
sockets=sockets_section(
|
||||||
|
listens=[listen(22), listen(443)],
|
||||||
|
established=[conn(("10.1.1.27", 443), ("203.0.113.7", 51000)),
|
||||||
|
conn(("10.1.1.27", 443), ("203.0.113.9", 51001)),
|
||||||
|
conn(("10.1.1.27", 22), ("198.51.100.4", 33000))]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
https = pipe_by_port(payload, 443)
|
||||||
|
assert https["source"] == "nginx"
|
||||||
|
assert https["connections"] == 2
|
||||||
|
assert https["clients"] == ["203.0.113.7", "203.0.113.9"], "the client is the remote end"
|
||||||
|
assert https["domains"] == ["a.example.test"]
|
||||||
|
assert https["metric"] == "gateway.in_conns@443"
|
||||||
|
assert pipe_by_port(payload, 22)["source"] == "haproxy"
|
||||||
|
assert payload["unmatched"] == {"inbound": 0, "outbound": 0, "other": 0, "groups": []}
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_frontend_that_is_idle_is_still_a_pipe_with_zero() -> None:
|
||||||
|
"""A configured port with no connections is `0`, not `n/a` — the socket table did answer."""
|
||||||
|
snap = snapshot(haproxy=haproxy_section([frontend("ssh", [22])]),
|
||||||
|
sockets=sockets_section(listens=[listen(22)]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
assert pipe_by_port(payload, 22)["connections"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_connection_on_a_port_no_config_names_is_reported_not_dropped() -> None:
|
||||||
|
"""`unmatched.inbound`: this host is serving something neither file mentions."""
|
||||||
|
snap = snapshot(
|
||||||
|
nginx=nginx_section([nginx_server([443], ["a.example.test"])]),
|
||||||
|
sockets=sockets_section(
|
||||||
|
listens=[listen(443), listen(8081)],
|
||||||
|
established=[conn(("10.1.1.27", 8081), ("10.1.0.9", 41000)),
|
||||||
|
conn(("10.1.1.27", 8081), ("10.1.0.9", 41001))]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
assert pipe_by_port(payload, 443)["connections"] == 0
|
||||||
|
assert payload["unmatched"]["inbound"] == 2, "the page's 'N other connections'"
|
||||||
|
assert payload["unmatched"]["other"] == 1, "8081 is a listener no frontend claims"
|
||||||
|
|
||||||
|
|
||||||
|
def test_one_port_named_by_both_configs_is_one_pipe_carrying_both_domains() -> None:
|
||||||
|
"""A host cannot bind a port twice, so the socket space is one pipe and the facts merge."""
|
||||||
|
snap = snapshot(
|
||||||
|
haproxy=haproxy_section([frontend("http", [80])]),
|
||||||
|
nginx=nginx_section([nginx_server([80], ["a.example.test"])]),
|
||||||
|
sockets=sockets_section(listens=[listen(80)],
|
||||||
|
established=[conn(("10.1.1.27", 80), ("203.0.113.1", 1234))]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
ports = [pipe["port"] for pipe in payload["inbound"]]
|
||||||
|
assert ports == [80], "no second pipe, and no socket counted down both"
|
||||||
|
assert pipe_by_port(payload, 80)["connections"] == 1
|
||||||
|
assert pipe_by_port(payload, 80)["domains"] == ["a.example.test"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_client_list_is_unique_sorted_and_bounded() -> None:
|
||||||
|
ports = [80]
|
||||||
|
many = [conn(("10.1.1.27", 80), (f"203.0.113.{i % 256}", 40000 + i))
|
||||||
|
for i in range(gateway.MAX_CLIENTS + 12)]
|
||||||
|
many.append(conn(("10.1.1.27", 80), ("203.0.113.1", 45678)))
|
||||||
|
snap = snapshot(nginx=nginx_section([nginx_server(ports, ["a.example.test"])]),
|
||||||
|
sockets=sockets_section(listens=[listen(80)], established=many))
|
||||||
|
pipe = pipe_by_port(gateway.parse(snap, "gateway"), 80)
|
||||||
|
assert len(pipe["clients"]) == gateway.MAX_CLIENTS
|
||||||
|
assert pipe["clients"] == sorted(pipe["clients"])
|
||||||
|
assert pipe["connections"] == len(many), "the count stays the truth when the list is capped"
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# the outbound join
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def test_a_target_written_as_an_address_matches_exactly_and_says_by_port_false() -> None:
|
||||||
|
snap = snapshot(
|
||||||
|
nginx=nginx_section([nginx_server([8443], ["app.example.test"],
|
||||||
|
[proxy_pass(("127.0.0.1", 8081))])]),
|
||||||
|
sockets=sockets_section(
|
||||||
|
listens=[listen(8443)],
|
||||||
|
established=[conn(("10.1.1.27", 39000), ("127.0.0.1", 8081)),
|
||||||
|
conn(("10.1.1.27", 39002), ("10.9.9.9", 8081))]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
exact = pipe_to(payload, "127.0.0.1", 8081)
|
||||||
|
assert exact["target"]["by_port"] is False
|
||||||
|
assert exact["connections"] == 1, "the one socket dialled to that address"
|
||||||
|
assert exact["metric"] == "gateway.out_conns@127.0.0.1:8081"
|
||||||
|
assert payload["unmatched"]["outbound"] == 1, (
|
||||||
|
"the same port on a different address is not this target's connection")
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_domain_target_is_matched_on_port_alone_and_labelled_by_port() -> None:
|
||||||
|
"""The address half is the config's, the socket half is the kernel's, and `by_port` says so."""
|
||||||
|
snap = snapshot(
|
||||||
|
nginx=nginx_section([nginx_server([4443], ["cloud.example.test"],
|
||||||
|
[proxy_pass(("nextcloud.example.test", 11000))])]),
|
||||||
|
sockets=sockets_section(
|
||||||
|
listens=[listen(4443)],
|
||||||
|
established=[conn(("10.1.1.27", 4443), ("10.1.0.4", 55555)),
|
||||||
|
conn(("10.1.1.27", 54340), ("10.1.0.254", 11000))]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
pipe = pipe_to(payload, "nextcloud.example.test", 11000)
|
||||||
|
assert pipe["target"]["by_port"] is True
|
||||||
|
assert pipe["connections"] == 1
|
||||||
|
assert pipe["source"] == "nginx"
|
||||||
|
assert payload["unmatched"]["groups"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_port_shared_by_several_names_resolves_none_of_them() -> None:
|
||||||
|
"""The real `stream.conf` shape: sixteen names on 443, and no basis for picking one.
|
||||||
|
|
||||||
|
The sockets go to `unmatched.outbound` with a group naming the candidates; each candidate's
|
||||||
|
own count stays `null` — a repeated port total drawn on sixteen pipes reads as sixteen times
|
||||||
|
the traffic, and a split of it is arithmetic nobody measured (rule 2).
|
||||||
|
"""
|
||||||
|
entries = [(f"site{i}.example.test", (f"backend{i}.example.test", 443)) for i in range(4)]
|
||||||
|
snap = snapshot(
|
||||||
|
nginx=nginx_section([nginx_server([443], [], [proxy_pass(None, variable="targetBackend")])],
|
||||||
|
maps=[mapping_block("$targetBackend", entries)]),
|
||||||
|
sockets=sockets_section(
|
||||||
|
listens=[listen(443)],
|
||||||
|
established=[conn(("10.1.1.27", 443), ("203.0.113.5", 4000)),
|
||||||
|
conn(("10.1.1.27", 50000), ("10.2.2.2", 443)),
|
||||||
|
conn(("10.1.1.27", 50002), ("10.2.2.3", 443))]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
assert len(payload["nginx"]["map_targets"]) == 4
|
||||||
|
for _key, (host, port) in entries:
|
||||||
|
assert pipe_to(payload, host, port)["connections"] is None, "unknown, never 0 and never 2"
|
||||||
|
group = payload["unmatched"]["groups"][0]
|
||||||
|
assert group == {"port": 443, "connections": 2,
|
||||||
|
"hosts": ["backend0.example.test", "backend1.example.test",
|
||||||
|
"backend2.example.test", "backend3.example.test"],
|
||||||
|
"host_count": 4}
|
||||||
|
assert payload["unmatched"]["outbound"] == 2
|
||||||
|
assert payload["unmatched"]["inbound"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_group_host_list_is_bounded_and_reports_the_real_total() -> None:
|
||||||
|
entries = [(f"site{i}.example.test", (f"backend{i}.example.test", 443))
|
||||||
|
for i in range(gateway.MAX_GROUP_HOSTS + 6)]
|
||||||
|
snap = snapshot(
|
||||||
|
nginx=nginx_section([nginx_server([443], [], [proxy_pass(None, variable="be")])],
|
||||||
|
maps=[mapping_block("$be", entries)]),
|
||||||
|
sockets=sockets_section(established=[conn(("10.1.1.27", 50000), ("10.2.2.2", 443))]))
|
||||||
|
group = gateway.parse(snap, "gateway")["unmatched"]["groups"][0]
|
||||||
|
assert len(group["hosts"]) == gateway.MAX_GROUP_HOSTS
|
||||||
|
assert group["host_count"] == len(entries)
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_join_ever_asks_a_resolver() -> None:
|
||||||
|
"""The whole point of `by_port`: this module must not turn a name into an address.
|
||||||
|
|
||||||
|
Spied on the live `socket` module the stdlib resolvers use, and run over the captured
|
||||||
|
gateway as well as a synthetic one, so a "helpful" future `socket.getaddrinfo` call inside
|
||||||
|
the join fails here rather than making 33 DNS queries every 30 seconds from the poll loop.
|
||||||
|
"""
|
||||||
|
calls: list[tuple[Any, ...]] = []
|
||||||
|
|
||||||
|
def spy(*args: Any, **kwargs: Any) -> Any: # pragma: no cover - must never run
|
||||||
|
calls.append((args, kwargs))
|
||||||
|
raise AssertionError("the gateway join must never resolve a name")
|
||||||
|
|
||||||
|
original = socket_module.getaddrinfo
|
||||||
|
socket_module.getaddrinfo = spy
|
||||||
|
try:
|
||||||
|
gateway.parse(snapshot(
|
||||||
|
haproxy=haproxy_section([frontend("mc", [25565])],
|
||||||
|
[backend("mc-be", [server("mc1", "minecraft.example.test",
|
||||||
|
25565)])]),
|
||||||
|
nginx=nginx_section([nginx_server([443], ["a.example.test"],
|
||||||
|
[proxy_pass(("app.example.test", 8080))])]),
|
||||||
|
sockets=sockets_section(listens=[listen(25565), listen(443)],
|
||||||
|
established=[conn(("10.1.1.27", 443), ("203.0.113.5", 900)),
|
||||||
|
conn(("10.1.1.27", 1), ("10.2.2.2", 8080))])),
|
||||||
|
"gateway")
|
||||||
|
finally:
|
||||||
|
socket_module.getaddrinfo = original
|
||||||
|
assert calls == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_two_services_on_one_host_are_two_pipes_not_one_sum() -> None:
|
||||||
|
"""The reason the outbound series name carries its port (`minecraft` 25565 and 25566)."""
|
||||||
|
snap = snapshot(
|
||||||
|
haproxy=haproxy_section([frontend("mc1", [25565]), frontend("mc2", [25566])],
|
||||||
|
[backend("testcraft", [server("mc1", "mc.example.test", 25565)]),
|
||||||
|
backend("nimcraft", [server("mc2", "mc.example.test", 25566)])]),
|
||||||
|
sockets=sockets_section(
|
||||||
|
established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 25565)),
|
||||||
|
conn(("10.1.1.27", 40002), ("10.5.5.5", 25566)),
|
||||||
|
conn(("10.1.1.27", 40004), ("10.5.5.5", 25566))]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
assert pipe_to(payload, "mc.example.test", 25565)["connections"] == 1
|
||||||
|
assert pipe_to(payload, "mc.example.test", 25566)["connections"] == 2
|
||||||
|
assert [pipe["metric"] for pipe in payload["outbound"]] == [
|
||||||
|
"gateway.out_conns@mc.example.test:25565", "gateway.out_conns@mc.example.test:25566"]
|
||||||
|
names = {pipe["target"]["name"] for pipe in payload["outbound"]}
|
||||||
|
assert names == {"mc1", "mc2"}, "the config's own server names survive the dedup"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_target_with_no_port_is_a_pipe_with_an_unknown_count() -> None:
|
||||||
|
"""`server x host` names no port, so there is nothing to join on and nothing to invent."""
|
||||||
|
snap = snapshot(
|
||||||
|
haproxy=haproxy_section([], [backend("be", [server("solo", "solo.example.test")])]),
|
||||||
|
sockets=sockets_section(established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 8080))]))
|
||||||
|
pipe = pipe_to(gateway.parse(snap, "gateway"), "solo.example.test", None)
|
||||||
|
assert pipe["target"]["by_port"] is False
|
||||||
|
assert pipe["connections"] is None
|
||||||
|
assert pipe["metric"] == "gateway.out_conns@solo.example.test"
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_same_target_from_two_configs_is_one_pipe() -> None:
|
||||||
|
snap = snapshot(
|
||||||
|
nginx=nginx_section([nginx_server([4443], ["a.example.test"],
|
||||||
|
[proxy_pass(("keycloak.example.test", 8080))]),
|
||||||
|
nginx_server([8443], ["admin.a.example.test"],
|
||||||
|
[proxy_pass(("keycloak.example.test", 8080))])]),
|
||||||
|
sockets=sockets_section(established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 8080))]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
targets = [pipe for pipe in payload["outbound"]
|
||||||
|
if pipe["target"]["host"] == "keycloak.example.test"]
|
||||||
|
assert len(targets) == 1
|
||||||
|
assert targets[0]["connections"] == 1, "one pipe, so one count, so no double draw"
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# the map: which maps route, and which are just header words
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def test_a_map_no_proxy_pass_references_contributes_nothing() -> None:
|
||||||
|
"""`$connection_upgrade`'s values are `upgrade`/`close` — header words, never a host."""
|
||||||
|
snap = snapshot(
|
||||||
|
nginx=nginx_section([nginx_server([443], [], [proxy_pass(None, variable="targetBackend")])],
|
||||||
|
maps=[mapping_block("connection_upgrade",
|
||||||
|
[("default", ("upgrade", None)),
|
||||||
|
("", ("close", None))]),
|
||||||
|
mapping_block("targetBackend",
|
||||||
|
[("a.example.test", ("be.example.test", 443))])]),
|
||||||
|
sockets=sockets_section(listens=[listen(443)]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
assert [row["key"] for row in payload["nginx"]["map_targets"]] == ["a.example.test"]
|
||||||
|
assert [pipe["target"]["host"] for pipe in payload["outbound"]] == ["be.example.test"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_map_entry_with_no_address_is_skipped() -> None:
|
||||||
|
snap = snapshot(
|
||||||
|
nginx=nginx_section([nginx_server([443], [], [proxy_pass(None, variable="be")])],
|
||||||
|
maps=[mapping_block("be", [("a.example.test", None),
|
||||||
|
("b.example.test", ("be2.example.test", 443))])]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
assert [row["key"] for row in payload["nginx"]["map_targets"]] == ["b.example.test"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_duplicate_endpoint_keeps_the_name_the_config_has() -> None:
|
||||||
|
"""haproxy is claimed first, so nginx writing the same endpoint loses nothing."""
|
||||||
|
snap = snapshot(
|
||||||
|
haproxy=haproxy_section([frontend("app", [80])],
|
||||||
|
[backend("app-be", [server("app1", "app.example.test", 9000)])]),
|
||||||
|
nginx=nginx_section([nginx_server([8443], ["app.example.test"],
|
||||||
|
[proxy_pass(("app.example.test", 9000))])]),
|
||||||
|
sockets=sockets_section(established=[conn(("10.1.1.27", 40000), ("10.5.5.5", 9000))]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
app_pipes = [pipe for pipe in payload["outbound"]
|
||||||
|
if pipe["target"]["host"] == "app.example.test"]
|
||||||
|
assert len(app_pipes) == 1
|
||||||
|
assert app_pipes[0]["target"]["name"] == "app1"
|
||||||
|
assert app_pipes[0]["source"] == "haproxy"
|
||||||
|
assert app_pipes[0]["connections"] == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_proxy_pass_naming_no_host_is_no_pipe() -> None:
|
||||||
|
"""`target: {host: ""}` is drift, and drift yields no pipe rather than a pipe to nowhere."""
|
||||||
|
snap = snapshot(nginx=nginx_section([nginx_server(
|
||||||
|
[443], [], [{"location": "/", "scheme": "http", "target": {"host": " ", "port": 80},
|
||||||
|
"variable": None}])]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
assert payload["outbound"] == []
|
||||||
|
assert payload["nginx"]["servers"] == [{"port": 443, "server_names": []}]
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_readers_refuse_the_shapes_that_are_not_values() -> None:
|
||||||
|
"""The three guards every section read goes through, on the inputs that are not data.
|
||||||
|
|
||||||
|
Pinned directly (the way `tests/test_config.py` pins `_flag`/`_num`) because each one is the
|
||||||
|
difference between a pipe and no pipe: an empty string is not an address, a bare string is
|
||||||
|
not a list of names, and whitespace is not a host.
|
||||||
|
"""
|
||||||
|
assert gateway._is_ip_literal("") is False
|
||||||
|
assert gateway._is_ip_literal(None) is False
|
||||||
|
assert gateway._is_ip_literal("203.0.113.7") is True
|
||||||
|
assert gateway._is_ip_literal("[::1]") is False, "/proc spells addresses without brackets"
|
||||||
|
assert gateway._is_ip_literal("app.example.test") is False
|
||||||
|
assert gateway._names("not a list") == []
|
||||||
|
assert gateway._names(None) == []
|
||||||
|
assert gateway._names([None, "a-be", "a-be", " ", "b-be"]) == ["a-be", "b-be"]
|
||||||
|
assert gateway._target_of({"host": " ", "port": 80}) is None
|
||||||
|
assert gateway._target_of(["host", 80]) is None
|
||||||
|
assert gateway._target_of({"host": "be.example.test", "port": "nope"}) == (
|
||||||
|
"be.example.test", None)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# the degradation arms: three ways to have nothing, none of them red
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def test_an_agent_without_the_sections_predates_gateway_support(gateway_snapshot: Any) -> None:
|
||||||
|
"""The 1.1.0 / `--no-proxy` arm: same keys, empty and null, one sentence, no verdict."""
|
||||||
|
payload = gateway.parse(gateway_snapshot("predates"), "gateway")
|
||||||
|
assert payload["supported"] is False
|
||||||
|
assert payload["why"] == gateway.WHY_PREDATES
|
||||||
|
assert "predates gateway support" in payload["why"]
|
||||||
|
assert payload["haproxy"] == {"frontends": []}
|
||||||
|
assert payload["nginx"] == {"servers": [], "map_targets": []}
|
||||||
|
assert payload["inbound"] == [] and payload["outbound"] == []
|
||||||
|
assert payload["unmatched"] == {"inbound": None, "outbound": None, "other": None,
|
||||||
|
"groups": []}
|
||||||
|
assert payload["sockets"] == {"listens": None, "established": None}
|
||||||
|
assert payload["sections"] == {"haproxy": None, "nginx": None, "sockets": None}
|
||||||
|
assert gateway.samples(payload, 1.0) == [], "an unknown count publishes no gauge"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_host_running_no_haproxy_says_so_and_keeps_the_nginx_pipes(gateway_snapshot: Any) -> None:
|
||||||
|
payload = gateway.parse(gateway_snapshot("no_haproxy"), "gateway")
|
||||||
|
assert payload["supported"] is True, "the agent knows the feature; the host has no haproxy"
|
||||||
|
assert payload["sections"]["haproxy"] is False
|
||||||
|
assert "no haproxy" in (payload["why"] or "")
|
||||||
|
assert "not found" not in (payload["why"] or ""), "the page's sentence, not the agent's log"
|
||||||
|
assert payload["section_errors"], "the agent's own line still travels, verbatim"
|
||||||
|
assert any("not found" in line for line in payload["section_errors"])
|
||||||
|
assert [row["port"] for row in payload["nginx"]["servers"]] == [80, 443, 4443, 8443]
|
||||||
|
assert [pipe["port"] for pipe in payload["inbound"] if pipe["source"] == "nginx"] == \
|
||||||
|
[80, 443, 4443, 8443]
|
||||||
|
assert payload["haproxy"]["frontends"] == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_unreadable_config_is_a_different_sentence_from_no_config() -> None:
|
||||||
|
snap = snapshot(haproxy=haproxy_section([], available=None),
|
||||||
|
nginx=nginx_section([nginx_server([80], ["a.example.test"])]),
|
||||||
|
sockets=sockets_section(listens=[listen(80)]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
assert payload["sections"]["haproxy"] is None
|
||||||
|
assert "could not be read or parsed" in (payload["why"] or "")
|
||||||
|
assert [pipe["port"] for pipe in payload["inbound"]] == [80]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_socket_table_that_did_not_answer_leaves_every_count_null(
|
||||||
|
gateway_snapshot: Any) -> None:
|
||||||
|
"""`available: null` is "no answer", and the only honest counts are `null` (rule 2)."""
|
||||||
|
payload = gateway.parse(gateway_snapshot("sockets_silent"), "gateway")
|
||||||
|
assert payload["sections"]["sockets"] is None
|
||||||
|
assert "sockets:" in (payload["why"] or "")
|
||||||
|
assert payload["sockets"] == {"listens": None, "established": None}
|
||||||
|
assert payload["inbound"], "the configured pipes are still drawn — the config did answer"
|
||||||
|
assert all(pipe["connections"] is None for pipe in payload["inbound"])
|
||||||
|
assert all(pipe["connections"] is None for pipe in payload["outbound"])
|
||||||
|
assert payload["unmatched"] == {"inbound": None, "outbound": None, "other": None,
|
||||||
|
"groups": []}
|
||||||
|
assert gateway.samples(payload, 1.0) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_partial_snapshot_names_the_missing_section_only() -> None:
|
||||||
|
"""A hypothetical agent publishing `sockets` but no configs: supported, with a gap named."""
|
||||||
|
snap = snapshot(sockets=sockets_section(listens=[listen(9999)]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
assert payload["supported"] is True
|
||||||
|
assert payload["sections"] == {"haproxy": None, "nginx": None, "sockets": True}
|
||||||
|
assert payload["unmatched"]["other"] == 1, "9999 is a listener nothing claims"
|
||||||
|
assert "haproxy" in (payload["why"] or "") and "nginx" in (payload["why"] or "")
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# what the payload must never contain, and the gauges it must
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def test_no_count_carries_a_verdict_or_a_window(gateway_snapshot: Any) -> None:
|
||||||
|
"""`net.*` precedent for colour, rule 2 for the `Δ` label: a count is a count.
|
||||||
|
|
||||||
|
Both arms are walked, so a `severity` bolted onto a pipe or a `window_seconds` copied from
|
||||||
|
`sources/host.py` fails here — the first would be a judgement `severity.py` never made, the
|
||||||
|
second would claim a measurement this source does not take.
|
||||||
|
"""
|
||||||
|
for mode in ("full", "predates", "sockets_silent"):
|
||||||
|
keys = walk_keys(gateway.parse(gateway_snapshot(mode), "gateway"))
|
||||||
|
assert "severity" not in keys, mode
|
||||||
|
assert not {key for key in keys if "window" in key or key.startswith("Δ")}, mode
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_payload_is_json_serialisable_end_to_end(gateway_snapshot: Any) -> None:
|
||||||
|
"""It goes out over `/api/gateway/snapshot` verbatim, so nothing in it may be exotic."""
|
||||||
|
json.dumps(gateway.parse(gateway_snapshot("full"), "gateway"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_samples_are_one_gauge_per_pipe_under_the_gateway_family(gateway_snapshot: Any) -> None:
|
||||||
|
snap = gateway_snapshot("full")
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
rows = gateway.samples(payload, 1_760_000_000.0)
|
||||||
|
assert rows, "a live gateway publishes gauges"
|
||||||
|
assert {row.source for row in rows} == {gateway.SOURCE}
|
||||||
|
assert {row.host for row in rows} == {"gateway"}
|
||||||
|
named = {row.metric: row.value for row in rows}
|
||||||
|
assert named["gateway.in_conns@443"] == float(pipe_by_port(payload, 443)["connections"])
|
||||||
|
counted = [pipe for pipe in payload["inbound"] + payload["outbound"]
|
||||||
|
if pipe["connections"] is not None]
|
||||||
|
assert len(named) == len(counted), "one gauge per pipe with a number, and none without"
|
||||||
|
for pipe in counted:
|
||||||
|
assert named[pipe["metric"]] == float(pipe["connections"]), (
|
||||||
|
"the stored gauge is the number the page prints, never a second reading of it")
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_gateway_snapshot_stamps_its_samples_from_the_snapshot() -> None:
|
||||||
|
"""The x-axis is when the agent took the snapshot, the way `sources/disks.py` does it."""
|
||||||
|
result = gateway.parse_gateway_snapshot(
|
||||||
|
"gateway", snapshot(haproxy=haproxy_section([frontend("ssh", [22])]),
|
||||||
|
sockets=sockets_section(listens=[listen(22)])), now=999.0)
|
||||||
|
assert result.events == [], "a connection count has no transition worth an event line"
|
||||||
|
assert [sample.ts for sample in result.samples] == [1_760_000_000.0]
|
||||||
|
assert [sample.metric for sample in result.samples] == ["gateway.in_conns@22"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_snapshot_with_no_timestamp_falls_back_to_the_callers_clock() -> None:
|
||||||
|
doc = snapshot(haproxy=haproxy_section([frontend("ssh", [22])]),
|
||||||
|
sockets=sockets_section(listens=[listen(22)]))
|
||||||
|
doc.pop("ts")
|
||||||
|
result = gateway.parse_gateway_snapshot("gateway", doc, now=42.0)
|
||||||
|
assert [sample.ts for sample in result.samples] == [42.0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_gauges_skip_a_pipe_with_no_number() -> None:
|
||||||
|
"""No sample for an unknown count — a chart of zeros for a dead socket table is the
|
||||||
|
failure mode rule 2 names, and `sources/disks.py` already refuses it for unreadable drives."""
|
||||||
|
payload = gateway.parse(snapshot(haproxy=haproxy_section([frontend("ssh", [22])])), "gateway")
|
||||||
|
assert payload["inbound"][0]["connections"] is None
|
||||||
|
assert gateway.samples(payload, 1.0) == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_snapshot_whose_sections_are_empty_is_an_honest_empty_page() -> None:
|
||||||
|
"""A 1.2.0 agent on a box with two empty configs: supported, and nothing drawn."""
|
||||||
|
snap = snapshot(haproxy=haproxy_section([]), nginx=nginx_section(),
|
||||||
|
sockets=sockets_section())
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
assert payload["supported"] is True and payload["why"] is None
|
||||||
|
assert payload["inbound"] == [] and payload["outbound"] == []
|
||||||
|
assert payload["sockets"] == {"listens": 0, "established": 0}
|
||||||
|
assert payload["unmatched"] == {"inbound": 0, "outbound": 0, "other": 0, "groups": []}
|
||||||
|
|
||||||
|
|
||||||
|
def test_junk_in_the_sections_yields_no_pipes_and_no_exception() -> None:
|
||||||
|
"""The source is a projection of a document it does not control — drift is `n/a`, not a crash."""
|
||||||
|
snap: dict[str, Any] = {"ts": 1.0, "host": "gateway",
|
||||||
|
"haproxy": {"available": True, "frontends": [3, None],
|
||||||
|
"backends": {"nope": 1}},
|
||||||
|
"nginx": {"available": True, "servers": "not a list", "maps": None},
|
||||||
|
"sockets": {"available": True, "listens": [None],
|
||||||
|
"established": [{"local": 5, "remote": None}]}}
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
assert payload["inbound"] == [] and payload["outbound"] == []
|
||||||
|
assert payload["unmatched"]["inbound"] == 0
|
||||||
|
assert payload["sockets"] == {"listens": 0, "established": 1}
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_port_of_zero_or_a_name_of_nothing_is_no_match() -> None:
|
||||||
|
"""`0` is not a port and an empty target is not a host, so neither becomes a pipe (rule 2)."""
|
||||||
|
snap = snapshot(
|
||||||
|
haproxy=haproxy_section([frontend("u", [None, 0]), frontend("ok", [22])],
|
||||||
|
[backend("b", [server("?", "", 22), server("z", "z.example.test",
|
||||||
|
0)])]),
|
||||||
|
sockets=sockets_section(listens=[listen(0), listen(22)]))
|
||||||
|
payload = gateway.parse(snap, "gateway")
|
||||||
|
assert [row["port"] for row in payload["haproxy"]["frontends"]] == [22]
|
||||||
|
assert [pipe["target"]["host"] for pipe in payload["outbound"]] == ["z.example.test"]
|
||||||
|
assert payload["outbound"][0]["target"]["port"] is None
|
||||||
|
assert payload["unmatched"]["other"] == 0, "a 0 is not a port, claimed or otherwise"
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_use_backend_map_is_reported_as_routes_not_one_chosen_backend() -> None:
|
||||||
|
"""haproxy picks per request here; the payload names every candidate and chooses none."""
|
||||||
|
snap = snapshot(haproxy=haproxy_section(
|
||||||
|
[frontend("fe", [80], default="base-be", use=("blue-be", "green-be"))]))
|
||||||
|
row = gateway.parse(snap, "gateway")["haproxy"]["frontends"][0]
|
||||||
|
assert row["backend"] == "base-be"
|
||||||
|
assert row["routes"] == ["base-be", "blue-be", "green-be"]
|
||||||
@@ -33,7 +33,7 @@ from dashboards import certs, collectors
|
|||||||
from dashboards.config import Settings, load
|
from dashboards.config import Settings, load
|
||||||
from dashboards.rates import RateEstimator
|
from dashboards.rates import RateEstimator
|
||||||
from dashboards.severity import classify
|
from dashboards.severity import classify
|
||||||
from dashboards.sources import nut
|
from dashboards.sources import gateway, nut
|
||||||
from dashboards.sources.disks import SMART_SERIES, DisksCollector
|
from dashboards.sources.disks import SMART_SERIES, DisksCollector
|
||||||
from dashboards.sources.host import HostCollector
|
from dashboards.sources.host import HostCollector
|
||||||
from slots_contract import assert_slot_shape
|
from slots_contract import assert_slot_shape
|
||||||
@@ -255,6 +255,46 @@ async def test_disks_source_charts_the_physical_drives_the_agent_reports(
|
|||||||
pytest.skip("no host agent answered")
|
pytest.skip("no host agent answered")
|
||||||
|
|
||||||
|
|
||||||
|
async def test_the_gateway_source_joins_the_live_ingress(settings: Settings,
|
||||||
|
agent_tls: dict[str, collectors.AgentTls]) -> None:
|
||||||
|
"""Phase 29: `sources/gateway.py` over the **real** gateway agent, read-only.
|
||||||
|
|
||||||
|
The production path — the same `fetch_snapshot` the poller will run, through the stored
|
||||||
|
mTLS authority — so what fails here is a drift between the agent's sections and the join,
|
||||||
|
not a typo in a fixture. The assertions are the ones that hold whatever the traffic is:
|
||||||
|
the agent answers the three sections, every configured frontend is a pipe, and the
|
||||||
|
inbound/outbound/unmatched tallies **partition** the kernel's ESTABLISHED table, which is
|
||||||
|
what lets the page's numbers reconcile against a direct `/proc/net/tcp` read.
|
||||||
|
"""
|
||||||
|
target = next((item for item in settings.host_targets if item.name == settings.gateway_host),
|
||||||
|
None)
|
||||||
|
if target is None:
|
||||||
|
pytest.skip(f"GATEWAY_HOST={settings.gateway_host!r} is not in HOST_TARGETS")
|
||||||
|
async with httpx.AsyncClient(timeout=settings.gateway_timeout) as client:
|
||||||
|
try:
|
||||||
|
snapshot = await collectors.fetch_snapshot(client, target,
|
||||||
|
timeout=settings.gateway_timeout,
|
||||||
|
tls=agent_tls.get(target.name))
|
||||||
|
except collectors.UpstreamError as exc:
|
||||||
|
pytest.skip(f"{target.name} did not answer: {exc}")
|
||||||
|
payload = gateway.parse(snapshot, target.name)
|
||||||
|
if not payload["supported"]:
|
||||||
|
pytest.fail(f"{target.name} runs agent {payload.get('agent_version')} — the gateway page "
|
||||||
|
"needs 1.2.0's haproxy/nginx/sockets sections (AGENTS.md gap 11's rollout)")
|
||||||
|
assert payload["host"] == target.name
|
||||||
|
assert payload["sockets"]["established"] == len(snapshot["sockets"]["established"])
|
||||||
|
assert payload["inbound"], "no frontend on either config would leave the page empty"
|
||||||
|
for pipe in payload["inbound"]:
|
||||||
|
assert isinstance(pipe["connections"], int), "the socket table answered, so a count is known"
|
||||||
|
for group in payload["unmatched"]["groups"]:
|
||||||
|
assert group["connections"] > 0 and group["host_count"] > 1, (
|
||||||
|
"a group is only ever the unresolved fan-out of several names on one port")
|
||||||
|
attributed = sum(pipe["connections"] or 0 for pipe in payload["inbound"] + payload["outbound"])
|
||||||
|
assert attributed + payload["unmatched"]["inbound"] + payload["unmatched"]["outbound"] == \
|
||||||
|
payload["sockets"]["established"], "every socket counted exactly once"
|
||||||
|
assert all(sample.source == gateway.SOURCE for sample in gateway.samples(payload, 0.0))
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("engine", ["gufo", "vllm"])
|
@pytest.mark.parametrize("engine", ["gufo", "vllm"])
|
||||||
async def test_engine_still_publishes_the_series_we_parse(engine: str,
|
async def test_engine_still_publishes_the_series_we_parse(engine: str,
|
||||||
settings: Settings) -> None:
|
settings: Settings) -> None:
|
||||||
|
|||||||
Reference in New Issue
Block a user