services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 28_gateway_agent_proxy

Browse Source 
**Phase 28 final pass — complete, 100 % green.** All five tasks were already `complete/`; this pass verified the criteria, re-ran the live half, and did the docs pass.

- Gate `bash .agents/validate.sh` **rc=0**: `1650 passed, 15 skipped`, coverage **98.97 %** (floor 94), `ruff` clean, `pyright` **0 errors**, `node --check` clean, hygiene clean (`/tmp/validate-done.log`).
- Live: `pytest -m live --live` **11 passed**; `./scripts/dev-server.sh status` nine sources `ok`; mTLS re-read of `gateway` → agent **1.2.0**, all three sections `available: true`, 6/6 frontends routed, 11 nginx servers + `$targetBackend`, **19** listeners, no core errors (`data/logs/28-finalpass-live.out`); installed agent file byte-identical to this tree (`b0e9ebf1b718…`).
- Criteria: 1 ✅ pytest/coverage · 2 ✅ three sections over a captured tree, `false`/`null` + one `errors` line each elsewhere with `cpu`/`mem`/`disk_io`/`system` intact · 3 ✅ `AGENT_VERSION` 1.2.0 + SCHEMA names all three · 4 ✅ six fixtures verbatim, capture refuses a bad read, 116 socket address columns hold no globally routable endpoint · 5 ✅ gateway runs 1.2.0 and answers over mTLS (transcript `data/logs/28-task05-rollout.md`, `28-finalpass-live.out`) · 6 ✅ validate · 7 ✅ additive: HEAD's 1.1.0 and this 1.2.0 `collect()`ed over one `--root` are key-for-key equal plus the three new sections, `--no-proxy` equal exactly (`data/logs/verify-28-final.py` → 41 PASS, `28-final-verify.out`).
- Docs fixed (README/AGENTS/Containerfile, all re-green): README's opening said "Four dashboards" above a five-row table and said the new sections are "stored in the snapshot" (they are in the **agent's** document — no `/api` payload, no `sample` row, nothing charts them until phase 29); `Containerfile`'s header still said "three dashboards"; AGENTS's coverage figure (98.94 → **98.97 %**), invariant 6 gained the captured-socket-table identifier rule, the agent bullet gained "phase 28 stops at the wire", and the fleet record names the stash-and-compare control plus the volatile listener count. Full record: `data/logs/NOTES-28-final.md`.
- Deviations: none functional; no git index/history touched, phase files untouched (`PLAN.md` and `complete/` unchanged). `uv pip list --outdated` still lists 4 packages (fastapi 0.153→0.161 etc.) — advisory only, dependencies untouched by this phase, left for a deliberate pass.
- Next pending phase: `.agents/phases/todo/29_gateway_dashboard_page/` (the page that consumes these three sections).
This commit is contained in:
ducoterra
2026-10-22 20:24:01 -04:00
parent 5d37444fb0
commit 44d61d3ab9
45 changed files with 4759 additions and 42 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/capture-gateway-config.sh
Executable
+352
View File
@@ -0,0 +1,352 @@
#!/usr/bin/env bash
# Re-record the gateway's proxy configs and kernel socket tables into tests/fixtures/.
#
# ./scripts/capture-gateway-config.sh # dials the `gateway-ops` ssh alias
# GATEWAY_SSH=lab-gw ./scripts/capture-gateway-config.sh
#
# What lands where — one fixture per remote document, in `tests/fixtures/`:
#
# gateway_haproxy.cfg cat /etc/haproxy/haproxy.cfg
# gateway_nginx_main.conf cat /etc/nginx/nginx.conf
# gateway_nginx_conf_d_<n>.conf cat /etc/nginx/conf.d/*.conf
# gateway_nginx_stream_d_<n>.conf cat /etc/nginx/stream.d/*.conf
# gateway_proc_net_tcp.txt cat /proc/net/tcp
# gateway_proc_net_tcp6.txt cat /proc/net/tcp6
#
# These documents are the ground truth behind `agent/host_agent.py`'s `haproxy`, `nginx` and
# `sockets` sections (phase 28) the way `tests/fixtures/nut_<ups>.vars` pins the UPS wire and
# `smartctl_driveripper_<dev>.txt` pins the drives. The include globs are read **out of the
# captured main config**, so a gateway that moves its vhosts to a third directory grows a third
# fixture on the next run instead of keeping a fixture set that quietly stopped being the tree
# the agent merges.
#
# Four rules, three of them `scripts/capture-fixtures.sh`'s:
#
# * **Verbatim, never hand-written** (AGENTS.md invariant 7). A fixture you typed is a test of
# your memory. Each file carries a `#` provenance block naming the remote path it came from.
# The header is comment-shaped because `#` is a comment in haproxy's config and in nginx's,
# and a `/proc` line whose 4th field is not a kernel state is skipped by the socket parser —
# provenance that costs no parser anything, which `tests/test_agent.py` pins by parsing the
# fixtures exactly as they sit on disk.
# * **The configs are not rewritten at all.** A domain is not an identifier to hide here: a
# `server_name` and the `proxy_pass` target beside it *are* the ingress map the gateway page
# exists to draw, and a scrubbed fixture is a fixture that tests a config this gateway does
# not run. Key material appears in these files only as the paths the proxy is told to read
# (`ssl_certificate_key /etc/letsencrypt/…`), which is a filename and not a key. No text
# filter of any kind sits between the read and a config fixture —
# `tests/test_deploy_scripts.py` runs this script against a fake remote holding a domain and
# demands the bytes back.
# * **A socket table keeps its shape and loses its identifiers** (AGENTS.md invariant 6, and
# the rule `capture-fixtures.sh` already applies to a drive: "Identifiers never go in the
# repo" — Serial, LU WWN, and the NVMe EUI-64 *which encodes the serial*). A Linux global
# IPv6 is EUI-64: `2603:…:5054:ff:fe84:b2e0` is this box's NIC address, and the other thing
# `/proc/net/tcp` carries is the public address of whoever was connected at the moment of
# the capture, which is not ours to publish. So any address `ipaddress` calls globally
# routable becomes a documentation address (RFC 5737 / RFC 3849) in first-seen order,
# re-encoded through the kernel's own per-32-bit-word little-endian spelling. Loopback,
# private, link-local, wildcard and already-documentation addresses are left exactly as they
# were, so the run is idempotent and the fixture still meets every shape the parser has to
# survive — `0.0.0.0`, `::`, `::ffff:a.b.c.d`, 32-hex-digit v6 words. Every port, state,
# queue, uid, inode and column width is byte-for-byte what the kernel answered, and only the
# two address *columns* are even looked at (a tx/rx queue pair is spelled like an address
# and is not one). The tables are volatile anyway — a proxy in service has a different
# connection set every second — so the tests assert shape, the two states kept, and the
# ports the captured configs claim to bind. Never a particular row.
# * **A bad read never replaces a good fixture.** Empty output, an ssh failure or a permission
# denial prints one line naming the file and the reason, leaves whatever is on disk alone,
# and exits 1 — the `slots_gufo.json` rule, so a laptop with no route to the gateway cannot
# turn six verbatim captures into six empty files.
#
# No sudo, deliberately: haproxy's config is 0644, nginx's tree is 0644 and `/proc/net/tcp{,6}`
# is world-readable, which is exactly what `dashboards-agent` reads as itself. A capture that
# needed root would record more than the agent can see — the mistake the smartctl pass in
# `capture-fixtures.sh` documents in the other direction.
#
# A file the host stops having is left in place rather than deleted: when the config moves,
# remove the orphan fixture in the same commit that updates the parser tests naming it.
set -uo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
FIX="$ROOT/tests/fixtures"
GATEWAY_SSH="${GATEWAY_SSH:-gateway-ops}"
mkdir -p "$FIX"
# `-n` matters: `capture_included` walks the remote listing on stdin, and an ssh that reads
# stdin would eat the rest of that list and quietly capture one vhost file instead of all of them.
SSH_OPTS=(-n -o BatchMode=yes -o ConnectTimeout=8)
failed=0
usage() {
cat <<'TXT'
Usage: ./scripts/capture-gateway-config.sh [-h]
Re-records the gateway's proxy configs and kernel socket tables into tests/fixtures/:
/etc/haproxy/haproxy.cfg, /etc/nginx/nginx.conf plus the documents its own glob includes
name, and /proc/net/tcp{,6}. Every read goes over the `$GATEWAY_SSH` ssh alias (default
`gateway-ops`) of a world-readable path, and a read that fails leaves the existing fixture
alone and exits 1. The configs land unedited; in the socket tables a globally routable
address becomes a documentation one and nothing else moves. See the block comment.
TXT
}
case "${1:-}" in
-h|--help) usage; exit 0 ;;
esac
# The socket tables' one edit, and the only place in this script that touches a captured byte.
# Kept as a here-doc in the house style (capture-fixtures.sh embeds two); it is pinned by
# running this script against a fake remote in tests/test_deploy_scripts.py.
SOCKET_IDENTIFIERS_OUT=$(cat <<'PY'
"""Move the globally-routable addresses in a /proc/net/tcp{,6} table to documentation space.
Every column, every port and every byte of spacing is the kernel's. What changes is an address
`ipaddress` calls global, which becomes a documentation address in first-seen order (RFC 5737
for v4, RFC 3849 for v6) re-encoded through the kernel's own per-32-bit-word little-endian
spelling — so the replacement is exactly as wide as the original and an IPv4-mapped v6 row stays
an IPv4-mapped v6 row. AGENTS.md invariant 6 is the reason: this repo's capture rule is that
identifiers never go in tracked files, and a global IPv6 on Linux is EUI-64 (it encodes the
NIC's MAC) while a visitor's public address belongs to the visitor. Loopback, private,
link-local, multicast, wildcard and already-documentation addresses are returned untouched,
which is what makes re-running this over its own output change nothing.
"""
import ipaddress
import re
import sys
DOC_V4 = int(ipaddress.IPv4Address("203.0.113.0"))
DOC_V6 = int(ipaddress.IPv6Address("2001:db8::"))
def unroll(word: str) -> bytes:
"""`/proc`'s per-word little-endian spelling -> the address's own bytes."""
raw = bytes.fromhex(word)
return b"".join(raw[i:i + 4][::-1] for i in range(0, len(raw), 4))
def roll(addr: ipaddress.IPv4Address | ipaddress.IPv6Address) -> str:
"""The inverse, in the kernel's uppercase."""
packed = addr.packed
return "".join(packed[i:i + 4][::-1].hex().upper() for i in range(0, len(packed), 4))
class Replacer:
def __init__(self) -> None:
self.counters: dict[int, int] = {4: 0, 6: 0}
self.mapping: dict[str, str] = {}
self.replacements = 0
def documentation(self, version: int) -> str:
self.counters[version] += 1
if version == 4:
return str(ipaddress.IPv4Address(DOC_V4 + self.counters[version]))
return str(ipaddress.IPv6Address(DOC_V6 + self.counters[version]))
def column(self, token: str) -> str:
"""One `<word>:<port>` column: rewrite the word, keep the port and its spelling."""
word, _, port = token.rpartition(":")
if len(word) not in (8, 32) or any(c not in "0123456789abcdefABCDEF" for c in word):
return token
try:
addr = ipaddress.ip_address(unroll(word))
except ValueError:
return token
# Only IPv6Address has `ipv4_mapped`; a v4 word is its own subject.
mapped = getattr(addr, "ipv4_mapped", None)
subject = mapped if mapped is not None else addr
if not subject.is_global:
return token
key = str(subject)
if key not in self.mapping:
self.mapping[key] = self.documentation(subject.version)
replacement = self.mapping[key]
self.replacements += 1
# A v4-mapped row keeps its mapped spelling, so the row that exercises that branch of
# the agent's decoder still exercises it after the capture.
if mapped is not None:
new: ipaddress.IPv4Address | ipaddress.IPv6Address = ipaddress.IPv6Address(
f"::ffff:{replacement}")
else:
new = ipaddress.ip_address(replacement)
return f"{roll(new)}:{port}"
def line(self, text: str) -> str:
"""Rewrite columns 1 and 2 only — the two address columns of a socket table."""
spans = [(m.start(), m.end(), m.group(0)) for m in re.finditer(r"\S+", text)]
if len(spans) < 4:
return text
out = []
last = 0
for index, (start, end, token) in enumerate(spans):
out.append(text[last:start])
out.append(self.column(token) if index in (1, 2) else token)
last = end
out.append(text[last:])
return "".join(out)
def main(argv: list[str]) -> int:
replacer = Replacer()
with open(argv[1], encoding="utf-8", errors="replace") as handle:
for line in handle:
sys.stdout.write(replacer.line(line))
if replacer.replacements:
print(f" ({replacer.replacements} globally-routable address column(s) moved to "
f"documentation space; {len(replacer.mapping)} distinct endpoint(s))",
file=sys.stderr)
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv))
PY
)
# header <kind> <remote path> — the provenance block every fixture carries.
header() {
local kind="$1" path="$2"
echo "# Verbatim $path from $GATEWAY_SSH, read read-only over the ssh key by"
echo "# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the"
echo "# deployed upstream, never guess its shape)."
case "$kind" in
config)
echo "# Nothing here is rewritten: it is this fleet's ingress map, and the domains in it are"
echo "# the data — an inbound name and the target it routes to are what these config sections"
echo "# exist to report. Key material appears only as the file paths the proxy reads."
;;
sockets)
echo "# One line per TCP socket, in the kernel's own spelling. The only edit is the repo's"
echo "# identifier rule (invariant 6, the drive-serial scrub in capture-fixtures.sh): a"
echo "# globally routable address became a documentation one (RFC 5737 / RFC 3849), because a"
echo "# global IPv6 here is EUI-64 and a visitor's address is not ours to publish. Every port,"
echo "# state, queue, uid, inode and column width is verbatim, as are the loopback, private"
echo "# and wildcard addresses the gateway's own frontends use."
echo "# Volatile by nature: a proxy in service has a different connection set every second, so"
echo "# a diff here on re-capture is expected. The tests assert shape, the two states the agent"
echo "# keeps, and the ports the captured configs claim to bind — never a particular row."
;;
esac
}
# capture <fixture> <kind> <remote path> — one ssh call, one document, written only if the read
# answered something. The remote command is a single `cat`: `scp` cannot read /proc at all (a
# procfs file reports size 0, so scp copies an empty file), and one mechanism for six fixtures
# is one mechanism to pin.
capture() {
local fixture="$1" kind="$2" path="$3"
local tmp="$FIX/.$fixture.tmp" err="$FIX/.$fixture.err"
local rc=0
ssh "${SSH_OPTS[@]}" "$GATEWAY_SSH" "cat $path" >"$tmp" 2>"$err" || rc=$?
if [ "$rc" -ne 0 ] || [ ! -s "$tmp" ]; then
local why
why="$(head -n 1 "$err" 2>/dev/null)"
[ -n "$why" ] || { [ "$rc" -eq 0 ] && why="answered an empty file" || why="ssh exited $rc"; }
rm -f "$tmp" "$err"
echo " ✗ $fixture — $GATEWAY_SSH:$path unreadable ($why); the fixture on disk is left alone"
failed=1
return 1
fi
rm -f "$err"
{
header "$kind" "$path"
case "$kind" in
sockets) python3 -c "$SOCKET_IDENTIFIERS_OUT" "$tmp" ;;
*) cat "$tmp" ;;
esac
} >"$FIX/$fixture"
rc=0
rm -f "$tmp"
echo " ✓ $fixture ($(wc -l <"$FIX/$fixture") lines from $GATEWAY_SSH:$path)"
}
# capture_included <fixture prefix> <remote dir> — every `*.conf` the main config's include line
# names, one fixture each. The prefix names the directory the way the parser spells it
# (`gateway_nginx_conf_d_` for /etc/nginx/conf.d), the rest is the file's own name:
# /etc/nginx/conf.d/http.conf → gateway_nginx_conf_d_http.conf.
capture_included() {
local prefix="$1" dir="$2"
local listing="$FIX/.$prefix.ls" err="$FIX/.$prefix.ls.err"
local rc=0
ssh "${SSH_OPTS[@]}" "$GATEWAY_SSH" "ls $dir/*.conf" >"$listing" 2>"$err" || rc=$?
if [ "$rc" -ne 0 ]; then
local why
why="$(head -n 1 "$err" 2>/dev/null)"
[ -n "$why" ] || { [ "$rc" -eq 0 ] && why="answered an empty list" || why="ssh exited $rc"; }
rm -f "$listing" "$err"
echo " ✗ ${prefix}* — $GATEWAY_SSH:$dir could not be listed ($why); the fixtures on disk are left alone"
failed=1
return 1
fi
rm -f "$err"
local path base
while IFS= read -r path; do
[ -n "$path" ] || continue
base="${path##*/}"
# Parameter expansion and nothing else builds the name, so there is no text filter of any
# kind between what the host said and what the fixture holds.
capture "${prefix}${base%.conf}.conf" config "$path"
done <"$listing"
rm -f "$listing"
}
# --- haproxy -----------------------------------------------------------------
# The file `parse_haproxy_config` is pinned to: frontends on `:::<port> v4v6`, one
# `server <name> <domain>:<port> check inter 5s` per backend, and a `use_backend` whose target
# is a fetch expression — which is why a name is reported only where the config names one.
capture gateway_haproxy.cfg config /etc/haproxy/haproxy.cfg
# --- nginx -------------------------------------------------------------------
# The main config, then the documents its own include lines name one level deep: the same set
# `Collector.nginx` merges. A literal include naming a file this host does not have (this
# gateway's `include /etc/nginx/mime.types;`) is neither captured nor an error — absence of a
# document the parser needs nothing from is the shape the agent already reports as "costs
# nothing and no line".
if capture gateway_nginx_main.conf config /etc/nginx/nginx.conf; then
globs=()
while IFS= read -r line; do
probe="${line#"${line%%[![:space:]]*}"}" # leading whitespace off, for the two tests below
case "$probe" in ''|'#'*) continue ;; esac
case "$probe" in include*) ;; *) continue ;; esac
spec="${probe#include}"
spec="${spec%%;*}"
spec="${spec//[[:space:]]/}"
# Anything with a `*` is a glob include, and the agent merges it — so it gets a fixture, or
# the report below says the script has no name for it. Silent skipping is what would let the
# fixture set stop being the tree the agent reads.
case "$spec" in *"*)"*) continue ;; esac # not a directive then: a fetch expression
case "$spec" in *"*"*) globs+=("$spec") ;; esac
done <"$FIX/gateway_nginx_main.conf"
if [ ${#globs[@]} -eq 0 ]; then
echo " ✗ gateway_nginx_*.conf — the captured main config names no glob include, so the"
echo " vhost fixtures were not refreshed: has the tree the agent merges moved?"
failed=1
fi
for spec in "${globs[@]}"; do
case "$spec" in
/etc/nginx/conf.d/*.conf) capture_included gateway_nginx_conf_d_ "${spec%/*}" ;;
/etc/nginx/stream.d/*.conf) capture_included gateway_nginx_stream_d_ "${spec%/*}" ;;
*)
echo " ✗ $spec — an include directory this script has no fixture name for"
echo " (add a case above and a fixture-pinned test in tests/test_agent.py)"
failed=1
;;
esac
done
fi
# --- the live sockets --------------------------------------------------------
# Both tables, always: `available` is decided across them, because a kernel built without IPv6
# answers tcp and not tcp6 and that is absence rather than a fault (no line, the include rule).
# Keeping only LISTEN and ESTABLISHED is what makes a capture of a busy gateway usable — the
# TIME_WAIT rows in here are the ones the parser must skip, which is exactly what
# tests/test_agent.py asserts against this file.
capture gateway_proc_net_tcp.txt sockets /proc/net/tcp
capture gateway_proc_net_tcp6.txt sockets /proc/net/tcp6
if [ "$failed" -ne 0 ]; then
echo
echo "capture FAILED — every fixture named above is unchanged on disk. Is $GATEWAY_SSH an ssh"
echo "alias that answers (ssh $GATEWAY_SSH true), and are the paths still where they were?"
exit 1
fi
echo "done — read the diff before committing the fixtures."