services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 30_gateway_connection_map

Browse Source 
**Phase 30 final verification pass — all 6 tasks already in `complete/`, so this was the phase-completion pass.**

- **Verified (re-run, not trusted):** `/gateway` serves exactly two panels (nginx→haproxy, `GATEWAY_CARD_KEYS`); haproxy rows `.solo` straight across, nginx `:443` branched with one tick per leaf (18 live, fixture 17); every pipe computes the theme `--ok` in dark+light, animates only at count ≥1, fully static under `prefers-reduced-motion`; switch = `role="switch"`+`aria-checked`+`aria-disabled`, no listener, footer+title name the missing write path; zero non-GET requests anywhere.
- **Live read-only re-run** (`data/logs/30-final-live-verify.out`): 17 PASS — `warnings: []`, 6 haproxy frontends→own `server`, `:443` 18 map leaves incl. `default→127.0.0.1:4443`, `:4443` 6 / `:8443` 3, `:80` bare listener, no `unmatched`/`outbound`/`by_port`/`partial`, no `9120`; inbound reconciliation **page 16 = kernel 16**; `gateway` block = the 12 declared keys; no `gateway:config` alert on a clean config. `git diff --stat agent/ deploy/` empty; agents 1.2.0 (gateway) / 1.1.0 ×2, all `tls`, `bundle_rev 2`.
- **Tests/lint (exact):** `uv run pytest` → **1791 passed, 16 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0**, `--cov` **98.94%** (floor 94), `ruff check src tests agent` clean, `pyright` 0 errors, `node --check` all page scripts, hygiene clean · `uv run pytest -m live --live` → **12 passed** · `uv pip list --outdated` empty · `./scripts/dev-server.sh status` → **10 sources ok**, `gateway` included · browser pass `30-connection-map.out` 63 PASS/0 FAIL (5 screenshots at 1440/360) · `30-final-mobile-audit.out` 371 PASS/0 FAIL/0 FINDING, exit 0, nine pages.
- **Docs fixed:** README `alerts.py` bullet omitted phase 30's keyword-only `gateway_payload` and its one `warn` config-integrity row (+suppression) — added; overview ingress line labelled a dated reading. AGENTS gap 13 corrected 391 px → **361 px** (task 05's pill took 30 px), "task 06 measured" → "phase 29 measured", coverage figure un-pinned to 98.9 %, live-suite count noted (12 today), "all five pages" severity sentence clarified for `/gateway`'s existence-only palette. Everything else reviewed in full and **verified consistent**.
- **Defect fixed:** duplicated footer-assertion block at the end of `tests/test_api.py::test_the_switch_is_built_and_explicitly_unwired` (one copy removed).
- **Notable:** `NOTES.md` is empty in the working tree — both items in it (non-mTLS agents, disappearing admin links) landed in phases 24 and 20; left cleared. Fixture/live `map`-count drift (17 vs 18) is recorded in AGENTS.md, not silently re-captured.
- **Next pending phase:** none — `30_gateway_connection_map` was the only entry in `.agents/phases/todo/`.
This commit is contained in:
ducoterra
2026-10-23 22:56:21 -04:00
parent fa4f38c694
commit e366f599f3
51 changed files with 5419 additions and 1519 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+101 -47
View File
@@ -10,7 +10,7 @@ file:
| `/disks` | Are any of my **physical** disks failing, and which one — health, temperature, power-on hours, wear, sector alarms | the same agents, running `smartctl -i -A -H` per whole drive (never a partition or a mount); a host whose agent has no `smartctl` is not a host on this page |
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
| `/gateway` | Which domains are coming in on which port, and where the traffic goes — the ingress's listeners, `server_name`s and SNI map joined to its live connections | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |
| `/gateway` | Which domains are coming in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`): a trunk per bound port with its configured endpoints branching off it, each line labelled with the config's own word for the route (`server_name`, `server`, `map` key) and its live connection count | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
runtime; every asset is served by this app.
@@ -207,8 +207,10 @@ not know its domain or its backend, so the **gateway page** is what joins these
in `sources/gateway.py`, never in the agent: `/snapshot` keeps the three sections as the agent read
them, `/api/gateway/snapshot` serves the joined topology, and only the counts that join produces are
stored — as the `gateway.in_conns@<port>` / `gateway.out_conns@<host>:<port>` gauges the page
charts. Nothing in `/api/system/snapshot` grows a proxy or socket field, and no `sample` row is
written for a config.
charts (since phase 30 task 02 the outbound gauge exists **only** for an endpoint the config wrote as
an address: the name-keyed ones were a chart of the port-only guess the join no longer makes, so they
would be permanently empty). Nothing in `/api/system/snapshot` grows a proxy or socket field, and no
`sample` row is written for a config.
All three parsers are pinned to captures rather than to descriptions, and
`./scripts/capture-gateway-config.sh` re-records them over the `GATEWAY_SSH` alias (default
@@ -806,9 +808,9 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
**no row on purpose**: a non-zero counter is an alarm chip on presence, zero is the absence
of a chip and never a green "healthy drive", and an odometer has no bad value.
* `alerts.py` — "what is wrong right now", as data. `alerts.build(settings, job_states,
host_payloads, engine_payloads, ups_payload, recent_events)` is a pure projection of the
state the poller already keeps plus the verdicts `severity.py` and the UPS health checks
already reached — no I/O, no thresholds of its own — and `/api/summary` returns it as
host_payloads, engine_payloads, ups_payload, recent_events, *, gateway_payload=None)` is a pure
projection of the state the poller already keeps plus the verdicts `severity.py` and the UPS health
checks already reached — no I/O, no thresholds of its own — and `/api/summary` returns it as
`alerts`, worst first: `{id, severity, source, message, href, at}`. A configured source
that stopped answering is `fail`; one that is stale but still serving its last good value
is `warn`; an **unconfigured** source never alerts; a dead source is one line rather than
@@ -819,6 +821,13 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
answers* keeps its own voice and links to the page that explains it (`/disks` for a wedged SMART
pass, `/gateway` for a proxy or socket section that went quiet). Staleness is measured in each
source's own interval, so the gateway page goes amber after 90 quiet seconds and not after 30.
The keyword-only `gateway_payload` (phase 30 task 03) is the one non-reachability voice this
module has for that source: a non-empty `warnings` list — the two config files not connecting to
themselves — is **one** `warn` row (`N proxy config problem(s) to address — <first message>`,
`href` `/gateway`), because a dangling backend is a structural fact and not a graded number, and
it is emitted only while the job is configured, `ok` and not stale, so a router that has stopped
answering keeps its single reachability line instead of growing a second one about a reading the
operator cannot refresh. No count of connections can produce a row here at all.
* `api.py` — the JSON every page reads: `/api/config`, `/api/health`, `/api/events`,
`/api/history`, `/api/metrics`, `/api/summary`, one `snapshot` + `history` pair per
dashboard (`/api/system/*`, `/api/disks/*`, `/api/gateway/*`, `/api/llm/*`, `/api/ups/*` —
@@ -860,25 +869,50 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
was, and the key paths of both are pinned with and without targets configured. Since phase 22
that block lists the **same hosts** `/api/disks/snapshot` does, and `fleet.hosts_total` is
that row count, so `/` and `/disks` can never describe two different fleets. `/api/summary` grew a
`gateway` block the same way in phase 29 (`configured`, `host`, `supported`, `unreachable`,
`inbound`, `outbound`, `map_targets`, `unmatched`, `unmatched_listeners`, `partial`, `states`):
the ingress's own tallies, `null` and never `0` in every arm that did not measure, `partial: true`
when a total had to leave an unattributable pipe out, and no `severity` anywhere — a connection
count is not a graded number. `/api/config` carries the matching row, including
`gateway` block the same way in phase 29, reshaped by phase 30 task 02 (`configured`, `host`,
`href`, `supported`, `unreachable`, `listeners`, `connections`, `live_in`, `live_out`,
`map_targets`, `states`) and given `warnings` by task 03: what the configs declare (`listeners`,
`connections`, `map_targets`, `warnings`) beside what the sockets are doing (`live_in`, `live_out`
— the latter summed over **distinct endpoints**, since several routes reaching one address is one
count and not two). `null` and never `0` in every arm that did not measure — **except**
`warnings`, which is a config fact and so reads a real `0` when the two files connect to
themselves. No `severity` anywhere: a connection count is not a graded number, and a structural
fault needs no threshold because the hardware of a config file either names a thing that exists or
it does not. The `unmatched` / `unmatched_listeners` / `partial` fields went out with the
guessing rule they existed to explain. `/api/config` carries the matching row, including
`gateway.interval_seconds`/`timeout_seconds` and the named target's redacted `url`.
`/api/gateway/*` (phase 29 task 03) is that page's pair. `/snapshot` is one host's whole
topology — the haproxy frontends and the nginx `listen`/`map` echo the pipes were built from,
the inbound pipes (port, its domains, its live connection count, up to 32 client addresses), the
outbound pipes (target `host:port`, `by_port` naming which half of the match is a socket fact,
the count, the config that named it) each carrying the `metric` it is stored under, the
`unmatched` tallies, and `supported`/`sections`/`why` for the degradation arms — and the counts
reconcile: inbound + outbound + unmatched == `sockets.established`, so the page's numbers are
provable against a direct `/proc/net/tcp` read. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
the `listeners` (port, its domains, its live connection count, up to 32 client addresses), the
`connections` graph (phase 30: one row per route the two config files declare, each naming its
inbound port, its outbound endpoint, and the config's own selector for it — a haproxy backend
name, an nginx `server_name`, a stream `map` key) with its **live count** — the socket table's
ESTABLISHED rows, inbound by the socket's local port and outbound only where a remote
`(address, port)` equals a target the config wrote as an address. Each row carries the `metric` it
is stored under, and `shared: true` when more than one route reaches its endpoint (one count, shown
twice, never split). `supported`/`sections`/`why` cover the degradation arms. `warnings` (phase 30
task 03) is the list of the two config files **not connecting to themselves** — seven kinds, each
one row of `{id, kind, subject, message}` where `id` is `"<kind>:<subject>"` and the message is the
sentence the page prints: `haproxy_route_dangling`, `haproxy_backend_orphan`,
`haproxy_backend_empty`, `nginx_server_unlistened`, `nginx_pool_undeclared`,
`nginx_variable_unmapped`, `nginx_map_route_missing`. Nothing is inferred to fill a gap, so three
shapes are deliberately **silent**: a vhost that listens and passes nothing (a `return 301` looks
identical here), a frontend that routes by a fetch expression (agent 1.2.0 reports "no name", which
is the same value "no routing directive" has), and an entry of a `map` no `proxy_pass` names (the
`$connection_upgrade` map's `upgrade`/`close` are header words, not missing routes). An unreadable
or absent section warns about nothing — a host that cannot be read has not been found guilty of
anything — and a `warnings: []` is therefore the truth, not a shrug. Non-zero, `/` gets **one**
amber row (`N proxy config problems to address — <first message>`) and this page gets the list. **Nothing reconciles
to `sockets.established` and that is the design**: a connection to a target the config wrote as a
name (most of this fleet's fan-out) or on a port no config names is ignored — not counted, not
totalled, not warned about — so a pipe count means "this route has N live connections" and nothing
more. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
a `404 {"configured": false}` answered before any client is built (both arms spy-pinned), a host
that has never answered is the `unreachable` stub whose every count is `null`, and a 1.1.0 agent
is `supported: false` plus its one `why` sentence — three arms, no red among them (invariant 3).
`/history` serves the `gateway.in_conns@<port>` and `gateway.out_conns@<host>:<port>` gauges —
the port in the outbound name because two services on one host are two facts — with `hosts=`
the port in the outbound name because two services on one host are two facts, and only ever for an
address endpoint — with `hosts=`
defaulting to the ingress alone, unknown names answering as empty series, and no dial either way
because history reads only SQLite.
* `web/` — six static dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`, `/gateway`) plus
@@ -927,9 +961,13 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
list reads as an explicit "all clear", and a live problem moves the panel above the
table. A host that is not answering keeps its row — dimmed, every number `n/a` — so the
fleet never quietly gets smaller and last good values are never mistaken for live ones.
Under that table is the ingress's **one line** (`gateway · 14 inbound · 4 outbound`, linked to
`/gateway`): the counts are `/api/summary`'s own sums, a count the source did not measure prints
`n/a` rather than `0`, and a deployment with no `GATEWAY_HOST` renders no line at all.
Under that table is the ingress's **one line** (`gateway · 10 listeners · 33 connections · 17 live
in · 2 live out` as the live router answered on 2026-10-23 — a reading, the same way a listener count
is, and only the field names behind it are a contract), linked to `/gateway`: every figure is `/api/summary`'s own, a count the source
did not measure prints `n/a` rather than `0`, and a deployment with no `GATEWAY_HOST` renders no
line at all. A non-zero `warnings` appends ` · N to address` and nothing else — `0` and `null` both
print nothing, so a clean ingress never grows a permanent sentence about the checks that did not
fire (phase 30 task 03).
A big colour-coded number is `Dash.hero(el, {...})` on the
`.hero` / `.hero-bar` styles, coloured by the `severity` in the payload via
`Dash.severityFrom()` — the only colour vocabulary is `ok` / `warn` / `fail`, and a value
@@ -1036,26 +1074,38 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
`No test initiated` or no variable at all — says the same "run a battery test" sentence as
`unknown`. Only a word the map has never seen stays a warning, printed verbatim so it can
be added.
`/gateway` answers the topology question `/system` cannot: which sockets on the ingress face
inbound, where each one's traffic goes, and how many connections are live on each. It is four
cards keyed on what the **config** said a pipe came from — `haproxy`, `nginx`, the `stream map`,
and `unmatched` — each hidden until a poll actually has pipes for it, so a router with no nginx
in it spends no panel saying so. An inbound row is a listen port with its `server_name`s and the
client addresses on it; an outbound row is the target **exactly as the config wrote it** (the SNI
name the stream map matched on in front of it when one key routes there), and a target the join
could only match on its socket **port** carries a `by port` chip, because that count belongs to
the port and not to that service. A pipe animates while it carries at least one live connection
and rests at zero, and **nothing on the page is coloured by a verdict** — a busy ingress is the
job, so `severity.py` rates no `gateway.*` metric and no number gets a green/amber/red tint. A
count the join did not decide prints `n/a` and never `0`, which is why the fan-out on a port many
domains share is `n/a` rather than one number repeated as if it were sixteen. The header's five
tallies print the reconciliation instead of asserting it: what the kernel's socket table reported
(`listeners`, `established`) beside what the pipes carry (`pipes in`, `pipes out`) and what no
config claims (`unmatched` — ports and connections kept as separate cells). Each card's trend is
`/gateway` answers the topology question `/system` cannot: which ports on the ingress are open,
what routes out of each, and how many connections are live on each. It is **two panels** — one per
config file, `nginx` then `haproxy` — each hidden until a poll actually has rows for it, so a router
with no nginx in it spends no panel saying so. Inside a panel the rows are **groups**: a *trunk* is
one port that file binds, and hanging off it are the *leaves* — the endpoints it names, the endpoint
**exactly as the config wrote it** with the config's own word for the route above it (a `server_name`,
a haproxy `server` name, a stream `map` key). That is the shape of the file, which is why the stream
`map` is not a third panel: its entries are the nginx panel's `:443` leaves, so every SNI domain
reads beside the port it reaches. A trunk that names no target — this fleet's `:80` `return 301` —
prints "no target named in the config" instead of drawing nothing. Each line is joined to its
counterparts: nginx's ports **branch** — a dashed spine down the fan-out with a tick into every
endpoint — and haproxy's one-`server` frontends run **straight across**, because a port with one
route behind it is not a branch. A line is **green because the config declares the route**, an
existence colour of the page's own, and it **animates only while it carries at least one live
connection** — motion is the socket, colour is the config, and a resting route keeps the same green
as a busy one. **Nothing on the page is coloured by a verdict** — a busy ingress is the job, so
`severity.py` rates no `gateway.*` metric and no number gets a green/amber/red tint. A line to an endpoint the config wrote as a **name** prints `—`, never `0`:
counting it would need a DNS lookup and the join never asks one, and the row's tooltip says which of
the two facts it is looking at. The header's four tallies are `ports`, `connections`, `live` and
`warnings`, every one of them the payload's own figure. A `warnings` count above zero raises a panel
above the map with the config-integrity rows (`a backend no frontend routes to`, a `proxy_pass` with
no `upstream` behind it, an SNI entry that reaches no address) in the overview's alert vocabulary;
the live ingress here is clean, so that panel is off. Each panel's trend is
behind a collapsed **Charts** toggle like `/system`'s, one `/api/gateway/history` request per
card per refresh. With no `GATEWAY_HOST` the page is the setup sentence rather than a red panel,
panel per refresh. With no `GATEWAY_HOST` the page is the setup sentence rather than a red panel,
and an ingress running an agent older than **1.2.0** says so in the agent's own words with every
count `n/a` and nothing coloured.
count `n/a` and nothing coloured. Each line's head carries the sketched **switch** — a real
`role="switch"` button, green and to the position because the route is in the config, named in full
for a screen reader (`endpoint, reached by <selector>, in the nginx config, on`) — and it is
**unwired on purpose**: `aria-disabled` with a tooltip that says so, and this app writes to no
upstream, so a route is disabled by editing `haproxy.cfg` or an nginx `server` block and by nothing
in a browser.
The four event feeds (`/system`, `/disks`, `/llm`, `/ups`) are a **paged window** over the `event`
table rather than a printout of it: the list is a bounded box that scrolls, 50 events to a
page, with `1–50 of 200` between `‹ prev` and `next ›` (disabled at the ends) underneath.
@@ -1237,14 +1287,18 @@ the page never reads it.
from both ends. The input is the **captured** gateway — `gateway_haproxy.cfg`, the nginx main config
plus its included documents, and the two `/proc/net/tcp{,6}` captures — run through the real
`host_agent` parsers, so the test compares the join against the agent's own document rather than
against a copy of its field names. The assertions are the join's: the pipes carry exactly the ports,
`server_name`s and targets those documents name; `inbound + outbound + unmatched` equals the socket
table's own ESTABLISHED count, so no connection is counted twice or dropped; a listening port no
config claims stays visible as `unmatched.other`; several names sharing one port produce `null`
counts and a `unmatched.groups` line rather than one number repeated on sixteen pipes; and **no join
ever asks a resolver** — `socket.getaddrinfo` is spied on and fails the test, because deciding
"already an address, or a name?" is the only question that module may ask, and a DNS lookup in the
poll loop would be rule 2 with a round trip attached.
against a copy of its field names. The assertions are the join's two rules and the limits around
them: the graph carries exactly the ports, `server_name`s and targets those documents declare (all
32 routes on this capture, as a **set**, so a missing or invented route fails by name); a socket is
inbound by its local port and outbound only when its remote `(address, port)` equals a target the
config wrote as an address; **every other socket is ignored** — `established`, `live_in` and
`live_out` are all printed and deliberately never reconciled, because a connection to a name, or on
a port nobody configured, is not a fact this page may claim; several routes reaching one endpoint
carry one count repeated with `shared: true` rather than a split nobody measured (so the totals, in
the source and on `/2`, sum **distinct endpoints**); a name target is never credited, and `0` never
stands for "not counted"; and **no join ever asks a resolver** — `socket.getaddrinfo` is spied on and
fails the test, because deciding "already an address, or a name?" is the only question that module
may ask, and a DNS lookup in the poll loop would be rule 2 with a round trip attached.
`tests/rate_window_contract.py` is the same device for one number rather than a field table: the Δ
window each source measures its rates over — `RATE_WINDOW_SECONDS` for the host agents,