services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 32_gateway_resolved_target_liveness

Browse Source 
Fixture and live stream.conf are byte-identical (modulo header comment). Let me test the parser directly:
This commit is contained in:
ducoterra
2026-10-26 14:03:42 -04:00
parent 25fd74b7c5
commit 51772ffef2
58 changed files with 5272 additions and 478 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+128 -35
View File
@@ -10,7 +10,7 @@ file:
| `/disks` | Are any of my **physical** disks failing, and which one — health, temperature, power-on hours, wear, sector alarms | the same agents, running `smartctl -i -A -H` per whole drive (never a partition or a mount); a host whose agent has no `smartctl` is not a host on this page |
| `/llm` | Tokens/sec, active sessions, queue depth, KV-cache pressure, latency percentiles | llama.cpp-family (gufo) `/metrics` + `/slots`, vLLM `/metrics` — read-only |
| `/ups` | Battery health, runtime left, power events, and battery-test buttons | NUT `upsd`, on whatever port it really listens on (its own wire-protocol client) |
| `/gateway` | Which domains are coming in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`): a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) on line 1 over the counterpart it routes to on line 2 (`→ host:port`), plus its live connection count — and one filter box above the map that hides the routes whose own labels do not contain what you typed, asking nothing of the server | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |
| `/gateway` | Which domains are coming in on which port, and where each one goes — **two connection tables, one per proxy config file** (`nginx`, then `haproxy`): a trunk per bound port with its configured routes branching off it, each line reading the config's own **inbound** word (`server_name`, `map` key, `backend` name) on line 1 over the counterpart it routes to on line 2 (`→ host:port`), plus its live connection count — and one filter box above the map that hides the routes whose own labels do not contain what you typed, asking nothing of the server | the agent on **one** host, `GATEWAY_HOST` (default `gateway`), reading its `haproxy.cfg` + nginx tree and `/proc/net/tcp{,6}` — so it needs **agent 1.2.0**, plus **1.3.0**'s `resolver` section to put a number on a route whose target the config writes as a *name* (without it those lines are drawn and dashed), and the whole page is *unconfigured* (never red) unless `GATEWAY_HOST` names a `HOST_TARGETS` entry |
No Prometheus, no Grafana, no CDN, no build step. Nothing is fetched from the internet at
runtime; every asset is served by this app.
@@ -212,23 +212,88 @@ an address: the name-keyed ones were a chart of the port-only guess the join no
would be permanently empty). Nothing in `/api/system/snapshot` grows a proxy or socket field, and no
`sample` row is written for a config.
All three parsers are pinned to captures rather than to descriptions, and
The `resolver` section is the fourth and the only one that leaves the host. The two configs above
name their targets two ways — an address (`proxy_pass http://127.0.0.1:8081`) and a name
(`server mc1 minecraft.reeselink.com:25565`) — and the kernel's socket table speaks addresses, so a
name is the half a page could never count. The agent therefore takes every `(name, port)` the two
**parsed** sections write as a target (haproxy `server` lines, then nginx `upstream` members,
`proxy_pass` targets and `map` entry targets, deduplicated in config order) and asks libc what each
one is: `getaddrinfo` over `SOCK_STREAM`, published as `via: "libc"` because that is the resolution
haproxy's own libc path performs **on this box** — the box that opens the connections. The nginx
config's own `resolver` directive travels beside it verbatim as `config_directive` (`10.1.0.1`
here, read out of `stream.d/stream.conf`), so a divergence between the two vantages is something an
operator can see rather than something this file guesses; the agent does not query that resolver
directly, and the dashboard never resolves anything at all (`sources/gateway.py` has a test that
fails if it ever calls a resolver). An address literal is reported `status: "literal"` with its
address and **no** lookup, and a name that does not answer right now is `nxdomain` or `error` with
`addresses: []` — the route keeps its row and carries no count, never a `0`.
That pass is bounded and cached exactly the way the SMART one is, for the SMART reason: at most
**64** lookups inside **3 s** of wall clock, cached **300 s**, the first pass inline so a freshly
started agent answers with real addresses and every later expiry refreshed on a single background
thread while the request is served the previous result. `getaddrinfo` takes no timeout argument, so
stopping between lookups is the only defence a black-holing resolver leaves — and the cost of not
stopping is the *system* page's poll timeout, spent on an add-on that page never draws. Each entry
carries the time **it** was answered, so a cache-served entry cannot pretend to be current; the
budget and the name bound each report themselves in one `errors` line and leave `available` at
`true`. `available` is the family's three-way answer (`false` means there was nothing to resolve
because no proxy config was read, and its one line says **not found** or **unreadable** according to
which of the two it was), and
`--no-resolve` removes the section in silence the way `--no-smart` removes SMART — while `--no-proxy`
removes it too, because a box that collects no proxy has no declared names to look up.
All four parsers are pinned to captures rather than to descriptions, and
`./scripts/capture-gateway-config.sh` re-records them over the `GATEWAY_SSH` alias (default
`gateway-ops`): `/etc/haproxy/haproxy.cfg`, `/etc/nginx/nginx.conf` plus every document its own
glob `include` lines name (`gateway_nginx_conf_d_*.conf`, `gateway_nginx_stream_d_*.conf`), and
`/proc/net/tcp{,6}`. No sudo — those paths are world-readable, which is exactly what the agent
reads as itself — and a read that fails, or answers empty, leaves the existing fixture alone and
exits 1, so a laptop with no route to the gateway cannot turn six captures into six empty files.
glob `include` lines name (`gateway_nginx_conf_d_*.conf`, `gateway_nginx_stream_d_*.conf`),
`/proc/net/tcp{,6}`, and `tests/fixtures/gateway_resolver.json` — the `resolver` section itself, so
the name→address join is testable with no network. No sudo and no DNS client: those paths are
world-readable, which is exactly what the agent reads as itself, and the resolver is recorded with
`getent ahostsv4` / `getent ahostsv6`, which is libc on the box that dials (the same vantage
`via: "libc"` claims). A read that fails, or answers empty, leaves the existing fixture alone and
exits 1, so a laptop with no route to the gateway cannot turn seven captures into seven empty files.
The resolver document is **derived, never listed**: the script feeds its own freshly captured
configs through `agent.host_agent`'s parsers and `target_names()`, so it asks about exactly the
targets the agent will ask about, an address literal costs no remote read, and a route added to the
gateway moves the fixture with it. `getent` answering `rc 2` is a fact — that name has no record and
becomes `status: "nxdomain"`, kept beside the names that did answer — while a dead transport, a
missing `getent`, or **not one name answering** is a broken vantage: the run refuses, because a
document of nothing but `nxdomain` would make every route on `/gateway` a permanent dash and every
join test a test of nothing. Re-capturing a gateway nobody changed is a no-op (fixed `ts`,
deduplicated, sorted with the agent's own `sort_resolver_addresses`), so a diff in that file means
the ingress changed.
The configs are not rewritten (their domains are the ingress map, and key material appears only
as the paths nginx is told to read); the socket tables lose only their identifiers, in line with
the rule that scrubs a drive's serial: a globally routable address becomes a documentation one
(RFC 5737 / RFC 3849), because a global IPv6 on Linux is EUI-64 and an inbound visitor's address
is not ours to publish. Every port, state, column and width in those tables is verbatim, and
those files are volatile by nature — a proxy in service has a different connection set every
second — so the tests assert shape and the ports the configs claim, never a particular row.
`AGENT_VERSION` is **1.2.0**, the version that ships all three sections; a host still running
1.1.0 answers `GET /schema` with a block that names none of them, and its snapshot simply has no
such keys.
as the paths nginx is told to read); the socket tables and the resolver's answers lose only their
identifiers, in line with the rule that scrubs a drive's serial: a globally routable address becomes
a documentation one (RFC 5737 / RFC 3849), because a global IPv6 on Linux is EUI-64 and an inbound
visitor's address is not ours to publish. Every port, state, column and width in those tables is
verbatim, and those files are volatile by nature — a proxy in service has a different connection set
every second — so the tests assert shape and the ports the configs claim, never a particular row.
`AGENT_VERSION` is **1.3.0**, the version that adds the `resolver` section on top of the three
phase 28 shipped; `GET /schema` names all four, each with its `available` flag, and `--no-proxy`
and `--no-resolve` are listed there too. A host still running 1.2.0 answers with a snapshot that
simply has no `resolver` key — which is why the version spread costs nothing: a section an agent
did not collect is absence, and the gateway page counts the address-written routes it has and draws
the name-written ones with a dash.
**The dashboard reads that section and never produces one.** `sources/gateway.py` indexes the
entries by `(name, port)`, stamps each route's `out` with `resolved` / `via` / `resolved_ts` — the
address the gateway dialled, the agent's own string for how it got it, and the moment that **entry**
was answered — and lets the socket join reach a name-written route through it. Which is the whole
of phase 32: the outbound endpoint of a `server` line, a `proxy_pass` or a `map` entry is now an
address the box that opens the connection published, so the `gateway.out_conns@<domain>:<port>`
gauges phase 30 task 02 stopped writing are written again, under the config's own spelling, with a
figure that belongs to the domain instead of to whatever else was listening on its port. Four rules
keep the join honest, each pinned in `tests/test_gateway_source.py`: a name nobody resolved keeps
`resolved: null`, `connections: null`, `metric: null` and **no stored sample** (never `0`, never an
empty list); matching is by normalised address, so `::ffff:10.1.2.64` in `/proc` and `10.1.2.64`
from `getaddrinfo` are one endpoint while a zero-padded `010.1.2.64` — which is no one's address
notation — matches nothing; two names resolving to one address, or two route lines reaching it, are
**one** count held by every line that reaches it; and the module still never calls a resolver, a spy
pin covering the parse, the gauge name and the capture alike. An `address: true` route claims none
of the three provenance fields, because the config wrote that address and no lookup happened.
`deploy/install-agent.sh --fetch-bundle <url> --cert-name <name> --session <token>` is the same
TLS install with the download done inside the registration run: it asks
@@ -431,9 +496,12 @@ No `--cert-bundle`, no `--fetch-bundle`, no `--force`: the bundle already on dis
render reads — the flag only replaces the file — so the render equals the installed unit, the re-run
guard prints nothing, and `/etc/dashboards-agent.certs` and the token file keep their bytes. Then
the same two checks the registration uses: the CA-only `--status` probe (with `--bind`) answers
`{"ok": true, …, "version": "1.2.0"}` on `/health`, and the dashboard's own `https://` dial answers
`{"ok": true, …, "version": "1.3.0"}` on `/health`, and the dashboard's own `https://` dial answers
whatever the new version added — `agent_version` plus the `haproxy` / `nginx` / `sockets` sections
for a 1.2.0 agent — beside the sections it always had.
(1.2.0) and the `resolver` section (1.3.0) — beside the sections it always had. That is exactly how
`gateway` moved 1.1.0 → 1.2.0 (phase 28) and 1.2.0 → **1.3.0** (phase 32, 2026-10-25: the same unit
sha256 before and after, the bundle untouched, the guard silent, one file and one restart), and the
`/gateway` page counted **33 of 33** routes afterwards where it had counted 2 before.
Two things the first full rollout learned, both cheap to hit and expensive to guess at:
@@ -887,8 +955,10 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
`connections` graph (phase 30: one row per route the two config files declare, each naming its
inbound port, its outbound endpoint, and the config's own selector for it — a haproxy backend
name, an nginx `server_name`, a stream `map` key) with its **live count** — the socket table's
ESTABLISHED rows, inbound by the socket's local port and outbound only where a remote
`(address, port)` equals a target the config wrote as an address. Each row carries the `metric` it
ESTABLISHED rows, inbound by the socket's local port and outbound where a remote
`(address, port)` equals an endpoint the join can name — a target the config wrote as an address, or
(phase 32 task 03) one of the addresses the gateway's own `resolver` section published for the name
it wrote. Each row carries the `metric` it
is stored under, and `shared: true` when more than one route reaches its endpoint (one count, shown
twice, never split). `supported`/`sections`/`why` cover the degradation arms. `warnings` (phase 30
task 03) is the list of the two config files **not connecting to themselves** — seven kinds, each
@@ -903,16 +973,25 @@ certs.py ─ root CA + client cert, minted once at startup (unreadable row → n
or absent section warns about nothing — a host that cannot be read has not been found guilty of
anything — and a `warnings: []` is therefore the truth, not a shrug. Non-zero, `/` gets **one**
amber row (`N proxy config problems to address — <first message>`) and this page gets the list. **Nothing reconciles
to `sockets.established` and that is the design**: a connection to a target the config wrote as a
name (most of this fleet's fan-out) or on a port no config names is ignored — not counted, not
totalled, not warned about — so a pipe count means "this route has N live connections" and nothing
more. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
to `sockets.established` and that is the design**: a connection to a target nothing can name (a
name with no resolver entry behind it) or a socket on a port no config names is ignored — not
counted, not totalled, not warned about — so a pipe count means "this route has N live connections"
and nothing more. The payload also carries the agent's `resolver` block (`available`, `via`, and
the projected `entries`: name, port, status, addresses, ts — nothing an operator has not reviewed)
beside the `sections` pair that already says which halves answered, so a dash on one line is
explainable from the document the page already holds. On the recorded
ingress that section places every declared route, which makes the dash the exception rather than
the normal look of this page, and its meaning is *this name has no record* — and on the live fleet
it is the exception too: the gateway has run agent **1.3.0** since 2026-10-25, `resolver.available`
answers `true` with one entry per declared target, and that host's page counted **33 of 33** routes
with no dash at all. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
a `404 {"configured": false}` answered before any client is built (both arms spy-pinned), a host
that has never answered is the `unreachable` stub whose every count is `null`, and a 1.1.0 agent
is `supported: false` plus its one `why` sentence — three arms, no red among them (invariant 3).
`/history` serves the `gateway.in_conns@<port>` and `gateway.out_conns@<host>:<port>` gauges —
the port in the outbound name because two services on one host are two facts, and only ever for an
address endpoint — with `hosts=`
endpoint whose address is known: one the config wrote, or (since phase 32 task 03) one the gateway
resolved the name to — with `hosts=`
defaulting to the ingress alone, unknown names answering as empty series, and no dial either way
because history reads only SQLite.
* `web/` — six static dashboard pages (`/`, `/system`, `/disks`, `/llm`, `/ups`, `/gateway`) plus
@@ -1108,9 +1187,10 @@ more. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
Above the map, one **filter box** finds a domain among 33 routes (`stackexpected`, `brain`, `4443`,
`minecraft`) because the rows are in config order and the question is always about one name. It hides
what the payload already sent and asks for **nothing**: no request, no re-derivation, no new route.
It matches the strings the config wrote — the inbound word(s), the endpoint, the port, the haproxy
`server` name, and a trunk's own domains and client addresses — case-insensitively, so a label CSS
ellipsised still matches its whole text. A trunk that matches keeps its **whole fan-out**: typing
It matches the strings the payload carries — the inbound word(s), the endpoint, the port, the haproxy
`server` name, the **addresses the gateway resolved** for a name target, and a trunk's own domains
and client addresses — case-insensitively, so a label CSS ellipsised still matches its whole text and
typing `10.1.2.64` finds the route whose target that address belongs to. A trunk that matches keeps its **whole fan-out**: typing
`4443` asks for the port, and every route hanging off it is the answer. `Escape` clears it, a chip
beside the field counts `N of M routes`, and a query that matches nothing leaves both panels standing
with `no route matches "…"` in each. What it never touches is the header's four tallies, the warnings
@@ -1120,12 +1200,17 @@ more. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
than inventing a zone no proxy declared.
**Nothing on the page
is coloured by a verdict** — a busy ingress is the job, so `severity.py` rates no `gateway.*` metric
and no number gets a green/amber/red tint. A line to an endpoint the config wrote as a **name**
prints `—`, never `0`: counting it would need a DNS lookup and the join never asks one, and the row's
tooltip says which of the two facts it is looking at. Expect that on most of this ingress's routes —
31 of its 33 are written as domains — and read it as *drawn, not counted* rather than as a fault; the
footer says the same thing once, in plain terms, because a dash per row needs one explanation and not
thirty-two. The header's four tallies are `ports`, `connections`, `live` and
and no number gets a green/amber/red tint. A line prints `—`, never `0`, and its tooltip and its
accessible name say **which** unknown it is: *the gateway could not resolve this name in this
reading* when nothing is known for the endpoint, or *the socket table did not answer this reading*
when an address is known and nothing was measured — two different things to go and fix. A resolved
endpoint with nothing open is neither of them: it prints `0`, because that is a measurement. A
counted line carries its provenance on hover (`… · address from libc, resolved 3m`), and a line the
config wrote as an address claims no resolution at all. A dash on a domain-named route now means the
gateway had **no record** for that name in this reading (`nxdomain`/`error`, or no `resolver` section
at all on a pre-1.3.0 agent) rather than "this page cannot count names": the ingress here runs
**1.3.0**, so all 33 of its declared routes carry a number. The
footer says the same thing once, in plain terms, and points at the hover. The header's four tallies are `ports`, `connections`, `live` and
`warnings`, every one of them the payload's own figure. A `warnings` count above zero raises a panel
above the map with the config-integrity rows (`a backend no frontend routes to`, a `proxy_pass` with
no `upstream` behind it, an SNI entry that reaches no address) in the overview's alert vocabulary;
@@ -1142,7 +1227,8 @@ more. `GATEWAY_HOST` naming no `HOST_TARGETS` entry is
documented here as a thing you can have: name-based liveness needs the agent to report resolved
targets (a future agent version), and switching a route needs a second gated write path in front of
the proxy configs, which is a change to this app's read-only promise rather than a feature. What runs
today is **agent 1.2.0**, no name resolution, and no switching.
today is **agent 1.3.0** — name resolution on the gateway, counts on every declared route, and no
switching.
The four event feeds (`/system`, `/disks`, `/llm`, `/ups`) are a **paged window** over the `event`
table rather than a printout of it: the list is a bounded box that scrolls, 50 events to a
page, with `1–50 of 200` between `‹ prev` and `next ›` (disabled at the ends) underneath.
@@ -1326,7 +1412,7 @@ plus its included documents, and the two `/proc/net/tcp{,6}` captures — run th
`host_agent` parsers, so the test compares the join against the agent's own document rather than
against a copy of its field names. The assertions are the join's two rules and the limits around
them: the graph carries exactly the ports, `server_name`s and targets those documents declare (all
32 routes on this capture, as a **set**, so a missing or invented route fails by name); a socket is
33 routes on this capture, as a **set**, so a missing or invented route fails by name); a socket is
inbound by its local port and outbound only when its remote `(address, port)` equals a target the
config wrote as an address; **every other socket is ignored** — `established`, `live_in` and
`live_out` are all printed and deliberately never reconciled, because a connection to a name, or on
@@ -1337,6 +1423,13 @@ stands for "not counted"; and **no join ever asks a resolver** — `socket.getad
fails the test, because deciding "already an address, or a name?" is the only question that module
may ask, and a DNS lookup in the poll loop would be rule 2 with a round trip attached.
The seventh capture, `gateway_resolver.json`, is in that suite's reach too: it is the agent's own
`resolver` document from the real gateway, so `tests/test_agent.py` proves it has the section's shape
(one entry per name `target_names()` derives, `via: "libc"`, the config's own `resolver` line, no
globally routable address) and `tests/test_gateway_source.py` proves it reaches every route the
configs write as a **name** and shares addresses with the recorded socket table — the capability the
join needs, pinned before the join exists to use it.
`tests/rate_window_contract.py` is the same device for one number rather than a field table: the Δ
window each source measures its rates over — `RATE_WINDOW_SECONDS` for the host agents,
`LLM_INTERVAL_SECONDS` for the engines, one estimator per measuring cadence (phase 23). It holds