services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 28_gateway_agent_proxy

Browse Source 
**Phase 28 final pass — complete, 100 % green.** All five tasks were already `complete/`; this pass verified the criteria, re-ran the live half, and did the docs pass.

- Gate `bash .agents/validate.sh` **rc=0**: `1650 passed, 15 skipped`, coverage **98.97 %** (floor 94), `ruff` clean, `pyright` **0 errors**, `node --check` clean, hygiene clean (`/tmp/validate-done.log`).
- Live: `pytest -m live --live` **11 passed**; `./scripts/dev-server.sh status` nine sources `ok`; mTLS re-read of `gateway` → agent **1.2.0**, all three sections `available: true`, 6/6 frontends routed, 11 nginx servers + `$targetBackend`, **19** listeners, no core errors (`data/logs/28-finalpass-live.out`); installed agent file byte-identical to this tree (`b0e9ebf1b718…`).
- Criteria: 1 ✅ pytest/coverage · 2 ✅ three sections over a captured tree, `false`/`null` + one `errors` line each elsewhere with `cpu`/`mem`/`disk_io`/`system` intact · 3 ✅ `AGENT_VERSION` 1.2.0 + SCHEMA names all three · 4 ✅ six fixtures verbatim, capture refuses a bad read, 116 socket address columns hold no globally routable endpoint · 5 ✅ gateway runs 1.2.0 and answers over mTLS (transcript `data/logs/28-task05-rollout.md`, `28-finalpass-live.out`) · 6 ✅ validate · 7 ✅ additive: HEAD's 1.1.0 and this 1.2.0 `collect()`ed over one `--root` are key-for-key equal plus the three new sections, `--no-proxy` equal exactly (`data/logs/verify-28-final.py` → 41 PASS, `28-final-verify.out`).
- Docs fixed (README/AGENTS/Containerfile, all re-green): README's opening said "Four dashboards" above a five-row table and said the new sections are "stored in the snapshot" (they are in the **agent's** document — no `/api` payload, no `sample` row, nothing charts them until phase 29); `Containerfile`'s header still said "three dashboards"; AGENTS's coverage figure (98.94 → **98.97 %**), invariant 6 gained the captured-socket-table identifier rule, the agent bullet gained "phase 28 stops at the wire", and the fleet record names the stash-and-compare control plus the volatile listener count. Full record: `data/logs/NOTES-28-final.md`.
- Deviations: none functional; no git index/history touched, phase files untouched (`PLAN.md` and `complete/` unchanged). `uv pip list --outdated` still lists 4 packages (fastapi 0.153→0.161 etc.) — advisory only, dependencies untouched by this phase, left for a deliberate pass.
- Next pending phase: `.agents/phases/todo/29_gateway_dashboard_page/` (the page that consumes these three sections).
This commit is contained in:
ducoterra
2026-10-22 20:24:01 -04:00
parent 5d37444fb0
commit 44d61d3ab9
45 changed files with 4759 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+98 -2
View File
@@ -1,6 +1,7 @@
# Homelab Dashboards
Four dashboards, one domain, one process, one SQLite file:
Four dashboards and the overview page that points at them — one domain, one process, one SQLite
file:
| Page | What it answers | Source |
| ----------- | ------------------------------------------------------------------------------- | -------------------------------------------------- |
@@ -90,7 +91,8 @@ that exact line. upsd is dialled at
## The host agent
`agent/host_agent.py` is one file, **stdlib only**, Python 3.10+. It reads `/proc`,
`/sys/block`, `/sys/class/drm` and `statvfs` and answers `GET /snapshot` (everything, one
`/sys/block`, `/sys/class/drm` and `statvfs` — and, on the host that fronts the fleet, the
haproxy/nginx configs and `/proc/net/tcp{,6}` — and answers `GET /snapshot` (everything, one
JSON document; `?top=N` trims the process lists), `GET /info` (slow-changing identity: OS,
kernel, CPU model, GPU list), `GET /schema` (the schema version plus this document) and
`GET /health` (the one route that needs no credential) as JSON, with a plain HTML status
@@ -147,6 +149,81 @@ one installed before phase 09 has no SMART pass to run. Opt out silently with `-
The drive truth this is all developed against lives in `tests/fixtures/smartctl_driveripper_*.txt`:
16 real drives, recorded with serials scrubbed, each one parsed by a test.
The `haproxy` section is the same kind of best-effort add-on, for the host that fronts the fleet.
The agent reads `/etc/haproxy/haproxy.cfg` (`--haproxy-config` names another path) and reports
what the config says: every `frontend` block with its name, each `bind` address:port exactly as
written (`:::22` is the address `::` on port `22`) and the backend names it routes to
(`default_backend`, plus every named `use_backend`), and every `backend` block with each `server`'s
name, target and port — `port: null` when the line names none, never `0`. It interprets nothing:
the routing between the two lists is the **names**, so joining them is the page's job (the gateway
page is phase 29) and the agent never evaluates a rule. `available` answers three questions the way `smartctl_available` does —
`true` read and parsed, `false` no config at that path (this host runs no haproxy), `null` a file
is there but could not be read or understood — and each non-`true` answer costs exactly **one**
`errors` line for the pass. Like the SMART lines, those lines never make the *system* page amber:
a host that never installed haproxy is unconfigured, not broken, and the page that owns the add-on
reads the sentence out of `agent.errors`. A failing config read cannot fail `/snapshot` or
`/health`, and `--no-proxy` drops the section and the line both. The parser is pinned to
`tests/fixtures/gateway_haproxy.cfg`, the verbatim capture from gateway. Neither path is an env
var and `deploy/install-agent.sh` passes neither flag — the unit it writes runs on the defaults —
so a box whose configs live elsewhere means adding the flag to `$UNIT` after installing.
The `nginx` section is that add-on's other half, and it reads more than one file: the main config
(`/etc/nginx/nginx.conf`, `--nginx-config` names another) **plus the files its own `include` lines
name**, one level deep, because that is where nginx keeps its vhosts
(`include /etc/nginx/conf.d/*.conf`) and its SNI routing (`/etc/nginx/stream.d/*.conf`). It
reports every `server` block with its `listen` address:port pairs (`[::]:8443` is the address `::`
on port `8443`; a bare `listen 80` has no address, and the `ssl`/`proxy_protocol` flags are not
this section's business), its `server_name` list (the catch-all `_` included verbatim — "any
domain" is what the page says about it), and every `proxy_pass`; every `upstream` block with its
`server` targets; and every `map` block with its keys. A `proxy_pass $var` is served with the
variable's name and `target: null`: nginx resolves that per connection from the `map` that defines
it, so the agent reports both facts **unmerged** and leaves the joining to a page — a merged target
would be a number the wire never published. `files` names the main config and the included
documents actually merged, and the include pass is bounded (8 files / 1 MB): what it could not
take is said in one `errors` line naming the count, and the section still reads `true`. The same
three-way `available`, the same one line per arm, the same rule that a missing, unreadable or
unparseable config degrades only this section — and `--no-proxy` drops the whole family with it,
in silence.
The `sockets` section is that add-on's live half: what is actually open and carrying traffic, as
opposed to what a config claims. The agent reads `/proc/net/tcp` and `/proc/net/tcp6` and keeps
only the two states a gateway page can draw pipes from — kernel state `0A` LISTEN into `listens`
(`{family, addr, port}`) and `01` ESTABLISHED into `established` (`{family, local, remote}`, each
end an `{addr, port}`). Every other state is a socket in transit: a `TIME_WAIT` row is a
connection that stopped being true before the page painted, and a proxy accumulates thousands of
them, so they are skipped without a word. Each table is read through **at most 4096 lines**
(`/proc`'s own header among them, so 4095 sockets per table); a longer one reports the rows it read
and costs one `errors` line naming the bound, with `available` still `true`. `addr` is the address
itself, unrolled from `/proc`'s per-32-bit-word little-endian spelling — `0.0.0.0` and `::` keep
the literal wildcard form, because "open on every address" is the fact the inbound join keys on —
and a word that is not an address is `null`, never a guess. The tables are read independently: a
kernel without IPv6 has no `tcp6` file, which costs nothing and no line, while a table that is
there and denied costs one. `available` is `true` when either table answered and `null` when
neither did, with one line saying why; there is no `false`, because a table that did not answer is
"no answer" and not "this host has no sockets". Nothing here is joined to a config: a socket does
not know its domain or its backend, so phase 29's gateway page will be what joins these ports to
the `haproxy` and `nginx` sections — where a socket no frontend claims stays visible as one. Until
that page exists the three sections live only in the agent's own `/snapshot` document: no `/api`
payload grows them, no `sample` row is stored for them, and nothing charts them.
All three parsers are pinned to captures rather than to descriptions, and
`./scripts/capture-gateway-config.sh` re-records them over the `GATEWAY_SSH` alias (default
`gateway-ops`): `/etc/haproxy/haproxy.cfg`, `/etc/nginx/nginx.conf` plus every document its own
glob `include` lines name (`gateway_nginx_conf_d_*.conf`, `gateway_nginx_stream_d_*.conf`), and
`/proc/net/tcp{,6}`. No sudo — those paths are world-readable, which is exactly what the agent
reads as itself — and a read that fails, or answers empty, leaves the existing fixture alone and
exits 1, so a laptop with no route to the gateway cannot turn six captures into six empty files.
The configs are not rewritten (their domains are the ingress map, and key material appears only
as the paths nginx is told to read); the socket tables lose only their identifiers, in line with
the rule that scrubs a drive's serial: a globally routable address becomes a documentation one
(RFC 5737 / RFC 3849), because a global IPv6 on Linux is EUI-64 and an inbound visitor's address
is not ours to publish. Every port, state, column and width in those tables is verbatim, and
those files are volatile by nature — a proxy in service has a different connection set every
second — so the tests assert shape and the ports the configs claim, never a particular row.
`AGENT_VERSION` is **1.2.0**, the version that ships all three sections; a host still running
1.1.0 answers `GET /schema` with a block that names none of them, and its snapshot simply has no
such keys.
`deploy/install-agent.sh --fetch-bundle <url> --cert-name <name> --session <token>` is the same
TLS install with the download done inside the registration run: it asks
`GET <url>/api/certs/<name>/bundle`, writes the answer to a temp file, and hands it to the
@@ -338,6 +415,20 @@ for a bundle install, so step 3 is a copy rather than a guess. A host left on `h
working over its bearer token — the rollout is per host, and one revoked or expired certificate
never blinks the others.
**Upgrading an already-registered host to a newer agent is one file, not a rollout.** Re-stage
`agent/host_agent.py` into `/home/ops/agent-install/agent/` (the snapshot-first rule applies: a path
grant is only as current as the file it names, so copy what you are replacing to
`host_agent.py-<date>-backup` first — a name the sudo grant does not name, so sudo will not run it),
run the granted path with **the unit's own arguments and nothing else**
(`sudo /home/ops/agent-install/deploy/install-agent.sh --bind <addr> --port 9120`), and restart.
No `--cert-bundle`, no `--fetch-bundle`, no `--force`: the bundle already on disk is what the unit
render reads — the flag only replaces the file — so the render equals the installed unit, the re-run
guard prints nothing, and `/etc/dashboards-agent.certs` and the token file keep their bytes. Then
the same two checks the registration uses: the CA-only `--status` probe (with `--bind`) answers
`{"ok": true, …, "version": "1.2.0"}` on `/health`, and the dashboard's own `https://` dial answers
whatever the new version added — `agent_version` plus the `haproxy` / `nginx` / `sockets` sections
for a 1.2.0 agent — beside the sections it always had.
Two things the first full rollout learned, both cheap to hit and expensive to guess at:
* **A CA row older than the identifier fix must be rotated before any bundle is installed.** A
@@ -953,6 +1044,11 @@ uv run ruff check src tests agent && uv run pyright # lint + types
SMART_HOSTS="host-a-ops host-b-ops" ./scripts/capture-fixtures.sh
# ... and smartctl per physical disk: plain smartctl,
# `sudo -n` if that is denied; scrubs serials/WWNs
./scripts/capture-gateway-config.sh # ... and the gateway's proxy tree + socket tables:
# haproxy.cfg, nginx.conf + its glob includes,
# /proc/net/tcp{,6} over $GATEWAY_SSH (gateway-ops).
# World-readable paths only; a failed read never
# replaces a good fixture
bash .agents/validate.sh # all of the above + `node --check` on every page
# script + repo hygiene, with the 94% coverage floor
```