services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
219d7a7c3b36fed7a87a91c0b734bbdf47bc9839
dashboards/scripts/capture-gateway-config.sh
T
ducoterra 51772ffef2 phase: 32_gateway_resolved_target_liveness 
Fixture and live stream.conf are byte-identical (modulo header comment). Let me test the parser directly:
2026-10-26 14:03:42 -04:00

710 lines
36 KiB
Bash
Executable File
Raw Blame History

#!/usr/bin/env bash
# Re-record the gateway's proxy configs, kernel socket tables and libc resolutions into
# tests/fixtures/.
#
# ./scripts/capture-gateway-config.sh # dials the `gateway-ops` ssh alias
# GATEWAY_SSH=lab-gw ./scripts/capture-gateway-config.sh
#
# What lands where — one fixture per remote document, in `tests/fixtures/`:
#
# gateway_haproxy.cfg cat /etc/haproxy/haproxy.cfg
# gateway_nginx_main.conf cat /etc/nginx/nginx.conf
# gateway_nginx_conf_d_<n>.conf cat /etc/nginx/conf.d/*.conf
# gateway_nginx_stream_d_<n>.conf cat /etc/nginx/stream.d/*.conf
# gateway_proc_net_tcp.txt cat /proc/net/tcp
# gateway_proc_net_tcp6.txt cat /proc/net/tcp6
# gateway_resolver.json getent ahostsv4/ahostsv6, once per declared target
#
# These documents are the ground truth behind `agent/host_agent.py`'s `haproxy`, `nginx`,
# `sockets` and `resolver` sections (phases 28 and 32) the way `tests/fixtures/nut_<ups>.vars`
# pins the UPS wire and `smartctl_driveripper_<dev>.txt` pins the drives. The include globs are
# read **out of the captured main config**, so a gateway that moves its vhosts to a third
# directory grows a third fixture on the next run instead of keeping a fixture set that quietly
# stopped being the tree the agent merges.
#
# The resolver document is the one fixture whose *name list* comes from the repo rather than from
# the remote: the targets are `agent.host_agent.target_names()` over the configs captured above,
# so the capture can never ask for more names than the agent will resolve, and a fixture naming a
# target the parser no longer derives is a fixture that fails a test. Each name is then read on
# the gateway itself — `getent ahostsv4 <name>` and `getent ahostsv6 <name>`, the same libc
# resolution `via: "libc"` claims and the one haproxy's `init-addr last,libc,none` performs — so
# the addresses in the fixture are the answers the *routing* box gives, not this laptop's. A
# `config_directive` read out of the already captured nginx text costs no round trip, so the
# fixture's provenance line and the agent's own answer come from the same bytes.
#
# Four rules, three of them `scripts/capture-fixtures.sh`'s:
#
# * **Verbatim, never hand-written** (AGENTS.md invariant 7). A fixture you typed is a test of
# your memory. Each file carries a `#` provenance block naming the remote path it came from.
# The header is comment-shaped because `#` is a comment in haproxy's config and in nginx's,
# and a `/proc` line whose 4th field is not a kernel state is skipped by the socket parser —
# provenance that costs no parser anything, which `tests/test_agent.py` pins by parsing the
# fixtures exactly as they sit on disk. `gateway_resolver.json` is the one fixture that cannot
# carry it: it is JSON the tests `json.load`, and a `#` line is not JSON, so its provenance is
# this block plus the `via`/`config_directive`/`ts` keys inside it (the same reason
# `slots_gufo.json` and `agent_<host>.json` carry none). Its `ts` is a fixed constant, not the
# capture's clock: the value is a *shape* (an entry carries the time it was answered) and a
# live timestamp would make every re-run diff for a difference no test can read.
# * **The configs are not rewritten at all.** A domain is not an identifier to hide here: a
# `server_name` and the `proxy_pass` target beside it *are* the ingress map the gateway page
# exists to draw, and a scrubbed fixture is a fixture that tests a config this gateway does
# not run. Key material appears in these files only as the paths the proxy is told to read
# (`ssl_certificate_key /etc/letsencrypt/…`), which is a filename and not a key. No text
# filter of any kind sits between the read and a config fixture —
# `tests/test_deploy_scripts.py` runs this script against a fake remote holding a domain and
# demands the bytes back.
# * **A socket table keeps its shape and loses its identifiers** (AGENTS.md invariant 6, and
# the rule `capture-fixtures.sh` already applies to a drive: "Identifiers never go in the
# repo" — Serial, LU WWN, and the NVMe EUI-64 *which encodes the serial*). A Linux global
# IPv6 is EUI-64: `2603:…:5054:ff:fe84:b2e0` is this box's NIC address, and the other thing
# `/proc/net/tcp` carries is the public address of whoever was connected at the moment of
# the capture, which is not ours to publish. So any address `ipaddress` calls globally
# routable becomes a documentation address (RFC 5737 / RFC 3849) in first-seen order,
# re-encoded through the kernel's own per-32-bit-word little-endian spelling. Loopback,
# private, link-local, wildcard and already-documentation addresses are left exactly as they
# were, so the run is idempotent and the fixture still meets every shape the parser has to
# survive — `0.0.0.0`, `::`, `::ffff:a.b.c.d`, 32-hex-digit v6 words. Every port, state,
# queue, uid, inode and column width is byte-for-byte what the kernel answered, and only the
# two address *columns* are even looked at (a tx/rx queue pair is spelled like an address
# and is not one). The tables are volatile anyway — a proxy in service has a different
# connection set every second — so the tests assert shape, the two states kept, and the
# ports the captured configs claim to bind. Never a particular row.
# * **A bad read never replaces a good fixture.** Empty output, an ssh failure or a permission
# denial prints one line naming the file and the reason, leaves whatever is on disk alone,
# and exits 1 — the `slots_gufo.json` rule, so a laptop with no route to the gateway cannot
# turn seven verbatim captures into seven empty files. The resolver document keeps the rule with
# one refinement: **one** name answering nothing is a fact and is recorded as
# `status: "nxdomain"`, while a *broken vantage* is not — an ssh transport failure, a missing
# `getent`, or not one declared name answering an address leaves the fixture untouched and
# exits 1, because a document of nothing-but-failures would turn every live route on the page
# into a permanent dash and every test of the join into a test of nothing.
#
# Two documents, one scrub. The socket tables and the resolver document each rewrite their
# globally routable addresses through the **same** `Replacer` (one rule, two shapes, below), in
# first-seen order per document. That order is per-document, so if a name ever resolves to a
# globally routable address its replacement in `gateway_resolver.json` need not be the number the
# socket tables gave that same address — which would silently unjoin `tests/test_gateway_source.py`'s
# captured routes. `test_the_captured_resolver_document_needed_no_scrub_at_all` is the pin on that
# hazard: it holds today because every declared target here answers a private address.
#
# No sudo, deliberately: haproxy's config is 0644, nginx's tree is 0644 and `/proc/net/tcp{,6}`
# is world-readable, which is exactly what `dashboards-agent` reads as itself. A capture that
# needed root would record more than the agent can see — the mistake the smartctl pass in
# `capture-fixtures.sh` documents in the other direction.
#
# A file the host stops having is left in place rather than deleted: when the config moves,
# remove the orphan fixture in the same commit that updates the parser tests naming it.
set -uo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
FIX="$ROOT/tests/fixtures"
GATEWAY_SSH="${GATEWAY_SSH:-gateway-ops}"
mkdir -p "$FIX"
# `-n` matters: `capture_included` walks the remote listing on stdin, and an ssh that reads
# stdin would eat the rest of that list and quietly capture one vhost file instead of all of them.
SSH_OPTS=(-n -o BatchMode=yes -o ConnectTimeout=8)
failed=0
usage() {
cat <<'TXT'
Usage: ./scripts/capture-gateway-config.sh [-h]
Re-records the gateway's proxy configs, kernel socket tables and libc resolutions into
tests/fixtures/: /etc/haproxy/haproxy.cfg, /etc/nginx/nginx.conf plus the documents its own
glob includes name, /proc/net/tcp{,6}, and one `getent ahostsv4`/`getent ahostsv6` read on
the gateway per target name `agent.host_agent.target_names()` derives from those configs
(written as `gateway_resolver.json`, the agent's own `resolver` section shape). Every read
goes over the `$GATEWAY_SSH` ssh alias (default `gateway-ops`) of a world-readable path or
command, and a read that fails leaves the existing fixture alone and exits 1. The configs
land unedited; in the socket tables and the resolver document a globally routable address
becomes a documentation one and nothing else moves. See the block comment.
TXT
}
case "${1:-}" in
-h|--help) usage; exit 0 ;;
esac
# The interpreter for the two steps that import the repo's own parser.
# `uv run python` is the same interpreter the tests use, which is the point: a capture whose name
# list came from a different parser than the one the suite pins is a fixture that agrees with
# nothing. A copy of this script with no project beside it (a test sandbox) has nothing for uv to
# resolve, so it falls back to `python3` — `agent/host_agent.py` is stdlib-only, and PYTHONPATH
# below is what makes it importable either way.
export PYTHONPATH="$ROOT${PYTHONPATH:+:$PYTHONPATH}"
run_py() {
if command -v uv >/dev/null 2>&1 && [ -f "$ROOT/pyproject.toml" ]; then
(cd "$ROOT" && uv run python "$@")
else
python3 "$@"
fi
}
# The identifier rule (AGENTS.md invariant 6) and the resolver document, in one module with three
# subcommands, and the only place in this script that touches a captured byte. One `Replacer` for
# both documents is the point: two copies of "globally routable becomes documentation space, in
# first-seen order" is two rules. Kept as a here-doc in the house style (capture-fixtures.sh
# embeds two); it is pinned by running this script against a fake remote in
# tests/test_deploy_scripts.py.
SOCKET_IDENTIFIERS_OUT=$(cat <<'PY'
"""The gateway capture's identifier rule and resolver document.
sockets <table> a /proc/net/tcp{,6} table on stdout, scrubbed
plan <fixtures dir> <plan file> every target the captured configs declare
resolver <plan> <raw dir> <fixture> <ts> the resolver document, or a refusal
**The scrub** (sockets, resolver). Every column, every port and every byte of spacing is the
kernel's or `getent`'s. What changes is an address `ipaddress` calls global, which becomes a
documentation address in first-seen order (RFC 5737 for v4, RFC 3849 for v6); in a socket table
it is re-encoded through the kernel's own per-32-bit-word little-endian spelling, so the
replacement is exactly as wide as the original and an IPv4-mapped v6 row stays an IPv4-mapped v6
row. AGENTS.md invariant 6 is the reason: this repo's capture rule is that identifiers never go in
tracked files, and a global IPv6 on Linux is EUI-64 (it encodes the NIC's MAC) while a visitor's
public address belongs to the visitor — and a name that resolves to one is the same exposure by
another route. Loopback, private, link-local, multicast, wildcard and already-documentation
addresses are returned untouched, which is what makes re-running this over its own output change
nothing.
**The plan** is derived from the repo's own parser and never by re-implementing one —
`from agent.host_agent import target_names` — so the capture can never out-ask the agent. The
name list, the order, the literal/name split and the `resolver` directive all come out of the
agent's own reading of the config fixtures this run captured.
"""
import ipaddress
import json
import os
import pathlib
import re
import sys
import tempfile
from typing import Any
DOC_V4 = int(ipaddress.IPv4Address("203.0.113.0"))
DOC_V6 = int(ipaddress.IPv6Address("2001:db8::"))
def unroll(word: str) -> bytes:
"""`/proc`'s per-word little-endian spelling -> the address's own bytes."""
raw = bytes.fromhex(word)
return b"".join(raw[i:i + 4][::-1] for i in range(0, len(raw), 4))
def roll(addr: ipaddress.IPv4Address | ipaddress.IPv6Address) -> str:
"""The inverse, in the kernel's uppercase."""
packed = addr.packed
return "".join(packed[i:i + 4][::-1].hex().upper() for i in range(0, len(packed), 4))
class Replacer:
def __init__(self) -> None:
self.counters: dict[int, int] = {4: 0, 6: 0}
self.mapping: dict[str, str] = {}
self.replacements = 0
def documentation(self, version: int) -> str:
self.counters[version] += 1
if version == 4:
return str(ipaddress.IPv4Address(DOC_V4 + self.counters[version]))
return str(ipaddress.IPv6Address(DOC_V6 + self.counters[version]))
def column(self, token: str) -> str:
"""One `<word>:<port>` column: rewrite the word, keep the port and its spelling."""
word, _, port = token.rpartition(":")
if len(word) not in (8, 32) or any(c not in "0123456789abcdefABCDEF" for c in word):
return token
try:
addr = ipaddress.ip_address(unroll(word))
except ValueError:
return token
# Only IPv6Address has `ipv4_mapped`; a v4 word is its own subject.
mapped = getattr(addr, "ipv4_mapped", None)
subject = mapped if mapped is not None else addr
if not subject.is_global:
return token
key = str(subject)
if key not in self.mapping:
self.mapping[key] = self.documentation(subject.version)
replacement = self.mapping[key]
self.replacements += 1
# A v4-mapped row keeps its mapped spelling, so the row that exercises that branch of
# the agent's decoder still exercises it after the capture.
if mapped is not None:
new: ipaddress.IPv4Address | ipaddress.IPv6Address = ipaddress.IPv6Address(
f"::ffff:{replacement}")
else:
new = ipaddress.ip_address(replacement)
return f"{roll(new)}:{port}"
def line(self, text: str) -> str:
"""Rewrite columns 1 and 2 only — the two address columns of a socket table."""
spans = [(m.start(), m.end(), m.group(0)) for m in re.finditer(r"\S+", text)]
if len(spans) < 4:
return text
out = []
last = 0
for index, (start, end, token) in enumerate(spans):
out.append(text[last:start])
out.append(self.column(token) if index in (1, 2) else token)
last = end
out.append(text[last:])
return "".join(out)
def address(self, value: str) -> str:
"""One dotted/colon address in its own spelling: scrub it, keep the spelling.
The resolver document's shape, where the socket table's is hex words. Same rule and the
same mapping: a globally routable address becomes a documentation one and everything else
— private, loopback, link-local, wildcard, already-documentation — is returned as it came.
A `::ffff:`-mapped address keeps its mapped spelling, because the spelling is a fact about
the answer, not an identifier.
"""
try:
candidate = ipaddress.ip_address(value)
except ValueError:
return value
mapped = getattr(candidate, "ipv4_mapped", None)
subject = mapped if mapped is not None else candidate
if not subject.is_global:
return value
key = str(subject)
if key not in self.mapping:
self.mapping[key] = self.documentation(subject.version)
replacement = self.mapping[key]
self.replacements += 1
if mapped is not None:
return f"::ffff:{replacement}"
return replacement
def sockets(argv: list[str]) -> int:
"""`sockets <table>` — the table on stdout with its two address columns scrubbed."""
replacer = Replacer()
with open(argv[2], encoding="utf-8", errors="replace") as handle:
for line in handle:
sys.stdout.write(replacer.line(line))
if replacer.replacements:
print(f" ({replacer.replacements} globally-routable address column(s) moved to "
f"documentation space; {len(replacer.mapping)} distinct endpoint(s))",
file=sys.stderr)
return 0
def plan(argv: list[str]) -> int:
"""`plan <fixtures dir> <plan file>` — every declared target, in the agent's own order.
Reads the config fixtures this run just captured, from a tree shaped the way the agent's
`--root` shapes it, so the plan is the answer to "what would the agent on this box ask?" and
not a second opinion about what a config declares. The first line is the nginx `resolver`
directive read out of the same bytes the agent reads them from; the rest is one tab-separated
line per target: `target`, `name` or `literal`, the name, the port (empty when the config
wrote none).
"""
from agent.host_agent import Collector, address_family, target_names
fixtures, out = pathlib.Path(argv[2]), pathlib.Path(argv[3])
tree = pathlib.Path(tempfile.mkdtemp(prefix="gateway-plan-"))
for rel, fixture in (("etc/haproxy/haproxy.cfg", "gateway_haproxy.cfg"),
("etc/nginx/nginx.conf", "gateway_nginx_main.conf")):
if (fixtures / fixture).is_file():
body = "".join(line for line in (fixtures / fixture).read_text().splitlines(keepends=True)
if not line.startswith("#"))
target = tree / rel
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(body)
else:
print(f"plan: {fixture} is not on disk — capture the configs before the resolver",
file=sys.stderr)
return 1
for prefix, rel in (("gateway_nginx_conf_d_", "etc/nginx/conf.d"),
("gateway_nginx_stream_d_", "etc/nginx/stream.d")):
for found in sorted(fixtures.glob(f"{prefix}*.conf")):
target = tree / rel / f"{found.name[len(prefix):]}"
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text("".join(line for line in found.read_text().splitlines(keepends=True)
if not line.startswith("#")))
collector = Collector(str(tree), smartctl="/nonexistent/smartctl")
errors: list[str] = []
haproxy = collector.haproxy(errors)
nginx, directive = collector.nginx_document(errors)
targets = target_names(haproxy, nginx)
if not targets:
print("plan: the captured configs declare no target, so there is nothing to resolve — "
"is the config capture intact?", file=sys.stderr)
return 1
# The name is the one part of the plan that goes back out over ssh, inside a remote command
# string. A config target is written by the operator, so this is not a hostile input today —
# but a fixture step that interpolated arbitrary config text into a shell command would be a
# remote-command hole waiting for somebody to write a semicolon in a `server_name`, and the
# honest answer to a name the script cannot safely ask about is to refuse the run, not to ask
# it anyway. Names, dots, dashes, underscores and scoped literals only; no whitespace, quote,
# semicolon, dollar or backtick.
unsafe = [name for name, _port in targets
if not re.fullmatch(r"[A-Za-z0-9._:.-]{1,253}", name)]
if unsafe:
print(f"plan: {len(unsafe)} target name(s) the script cannot ask about safely, starting "
f"with {unsafe[0]!r} — nothing was read and nothing was written", file=sys.stderr)
return 1
with out.open("w", encoding="utf-8") as handle:
handle.write(f"directive\t{directive if directive is not None else ''}\n")
for name, port in targets:
# The agent's own test, imported rather than restated: a target that is already an
# address is `status: "literal"` and costs no read on the remote, exactly as the
# agent's pass skips its own lookup.
kind = "literal" if address_family(name) else "name"
handle.write(f"target\t{kind}\t{name}\t{'' if port is None else port}\n")
return 0
def resolver(argv: list[str]) -> int:
"""`resolver <plan file> <raw dir> <fixture> <ts>` — the document, or a refusal.
Written to the fixture path only when at least one declared name answered an address; every
name failing means the vantage is broken (a resolver down, a firewall between, a `getent` that
answers nothing), and a fixture of failures would make every live route on the page a
permanent dash and every test of the join a test of nothing. One name failing among successes
is a fact, and is kept as `status: "nxdomain"`.
"""
# The shape's single source of truth: the same `via` and the same address order the agent
# publishes, imported rather than remembered, so the fixture is the agent's document.
from agent.host_agent import RESOLVE_VIA, sort_resolver_addresses
plan_path = pathlib.Path(argv[2])
raw = argv[3]
out = pathlib.Path(argv[4])
ts = float(argv[5])
replacer = Replacer()
directive: str | None = None
entries: list[dict[str, Any]] = []
names = 0
answered = 0
index = 0
with open(plan_path, encoding="utf-8") as handle:
for line in handle:
fields = line.rstrip("\n").split("\t")
if fields[0] == "directive":
directive = fields[1] if len(fields) > 1 and fields[1] else None
continue
if fields[0] != "target":
continue
kind, name = fields[1], fields[2]
port: int | None = int(fields[3]) if len(fields) > 3 and fields[3] else None
if kind == "literal":
found, is_literal = [name], True
else:
names += 1
is_literal = False
# `getent ahostsv6` answers `::ffff:a.b.c.d` for a name with no AAAA record — the
# spelling glibc's AF_INET6 view synthesises, and the one `getaddrinfo` the agent
# performs never publishes it. Keeping it would claim an address the agent does
# not, so a mapped answer whose v4 form is in the same set is dropped. A mapped
# answer with no plain one beside it survives, because then it is the answer.
found = [field[0] for path in (f"{index}.v4", f"{index}.v6")
for field in (line.split() for line in read_raw(raw, path).splitlines())
if field]
plain = {f"::ffff:{address}" for address in found
if not address.startswith("::ffff:")}
found = [address for address in found if address not in plain]
index += 1
# Scrub first, then sort: `addresses` is the published list, so the agent's ordering
# rule belongs to what the fixture prints and not to what the resolver answered.
addresses = sort_resolver_addresses(replacer.address(a) for a in found)
if is_literal:
status = "literal"
else:
status = "ok" if addresses else "nxdomain"
if addresses:
answered += 1
entries.append({"name": name, "port": port, "status": status,
"addresses": addresses, "ts": ts})
if names and not answered:
print(f"resolver: none of the {names} declared name(s) answered an address — the vantage "
"is broken, not the fixture; nothing was written", file=sys.stderr)
return 1
document = {"available": True, "via": RESOLVE_VIA, "config_directive": directive,
"ts": ts, "entries": entries, "errors": []}
tmp = out.with_name(f".{out.name}.tmp")
tmp.write_text(json.dumps(document, indent=2) + "\n", encoding="utf-8")
tmp.replace(out)
tallies: dict[str, int] = {}
for row in entries:
tallies[row["status"]] = tallies.get(row["status"], 0) + 1
summary = ", ".join(f"{count} {status}" for status, count in sorted(tallies.items()))
if replacer.replacements:
summary += (f"; {replacer.replacements} address(es) moved to documentation space "
f"({len(replacer.mapping)} distinct)")
print(f"{len(entries)} target(s): {summary}")
return 0
def read_raw(directory: str, name: str) -> str:
"""One `getent` reply, or empty when the name never got that far.
A missing file is not an error here: the transport failure that would have left a hole is
fatal in the shell and stops the run before this step, so a hole means a `literal` target,
which is read out of the plan instead.
"""
path = os.path.join(directory, name)
if not os.path.isfile(path):
return ""
with open(path, encoding="utf-8", errors="replace") as handle:
return handle.read()
def main(argv: list[str]) -> int:
commands = {"sockets": sockets, "plan": plan, "resolver": resolver}
mode = argv[1] if len(argv) > 1 else "sockets"
if mode not in commands:
print(f"this module does sockets, plan or resolver, not '{mode}'", file=sys.stderr)
return 2
return commands[mode](argv)
if __name__ == "__main__":
raise SystemExit(main(sys.argv))
PY
)
# header <kind> <remote path> — the provenance block every fixture carries.
header() {
local kind="$1" path="$2"
echo "# Verbatim $path from $GATEWAY_SSH, read read-only over the ssh key by"
echo "# ./scripts/capture-gateway-config.sh (AGENTS.md invariant 7: verify against the"
echo "# deployed upstream, never guess its shape)."
case "$kind" in
config)
echo "# Nothing here is rewritten: it is this fleet's ingress map, and the domains in it are"
echo "# the data — an inbound name and the target it routes to are what these config sections"
echo "# exist to report. Key material appears only as the file paths the proxy reads."
;;
sockets)
echo "# One line per TCP socket, in the kernel's own spelling. The only edit is the repo's"
echo "# identifier rule (invariant 6, the drive-serial scrub in capture-fixtures.sh): a"
echo "# globally routable address became a documentation one (RFC 5737 / RFC 3849), because a"
echo "# global IPv6 here is EUI-64 and a visitor's address is not ours to publish. Every port,"
echo "# state, queue, uid, inode and column width is verbatim, as are the loopback, private"
echo "# and wildcard addresses the gateway's own frontends use."
echo "# Volatile by nature: a proxy in service has a different connection set every second, so"
echo "# a diff here on re-capture is expected. The tests assert shape, the two states the agent"
echo "# keeps, and the ports the captured configs claim to bind — never a particular row."
;;
esac
}
# capture <fixture> <kind> <remote path> — one ssh call, one document, written only if the read
# answered something. The remote command is a single `cat`: `scp` cannot read /proc at all (a
# procfs file reports size 0, so scp copies an empty file), and one mechanism for six fixtures
# is one mechanism to pin.
capture() {
local fixture="$1" kind="$2" path="$3"
local tmp="$FIX/.$fixture.tmp" err="$FIX/.$fixture.err"
local rc=0
ssh "${SSH_OPTS[@]}" "$GATEWAY_SSH" "cat $path" >"$tmp" 2>"$err" || rc=$?
if [ "$rc" -ne 0 ] || [ ! -s "$tmp" ]; then
local why
why="$(head -n 1 "$err" 2>/dev/null)"
[ -n "$why" ] || { [ "$rc" -eq 0 ] && why="answered an empty file" || why="ssh exited $rc"; }
rm -f "$tmp" "$err"
echo " ✗ $fixture — $GATEWAY_SSH:$path unreadable ($why); the fixture on disk is left alone"
failed=1
return 1
fi
rm -f "$err"
{
header "$kind" "$path"
case "$kind" in
sockets) run_py -c "$SOCKET_IDENTIFIERS_OUT" sockets "$tmp" ;;
*) cat "$tmp" ;;
esac
} >"$FIX/$fixture"
rc=0
rm -f "$tmp"
echo " ✓ $fixture ($(wc -l <"$FIX/$fixture") lines from $GATEWAY_SSH:$path)"
}
# capture_included <fixture prefix> <remote dir> — every `*.conf` the main config's include line
# names, one fixture each. The prefix names the directory the way the parser spells it
# (`gateway_nginx_conf_d_` for /etc/nginx/conf.d), the rest is the file's own name:
# /etc/nginx/conf.d/http.conf → gateway_nginx_conf_d_http.conf.
capture_included() {
local prefix="$1" dir="$2"
local listing="$FIX/.$prefix.ls" err="$FIX/.$prefix.ls.err"
local rc=0
ssh "${SSH_OPTS[@]}" "$GATEWAY_SSH" "ls $dir/*.conf" >"$listing" 2>"$err" || rc=$?
if [ "$rc" -ne 0 ]; then
local why
why="$(head -n 1 "$err" 2>/dev/null)"
[ -n "$why" ] || { [ "$rc" -eq 0 ] && why="answered an empty list" || why="ssh exited $rc"; }
rm -f "$listing" "$err"
echo " ✗ ${prefix}* — $GATEWAY_SSH:$dir could not be listed ($why); the fixtures on disk are left alone"
failed=1
return 1
fi
rm -f "$err"
local path base
while IFS= read -r path; do
[ -n "$path" ] || continue
base="${path##*/}"
# Parameter expansion and nothing else builds the name, so there is no text filter of any
# kind between what the host said and what the fixture holds.
capture "${prefix}${base%.conf}.conf" config "$path"
done <"$listing"
rm -f "$listing"
}
# --- haproxy -----------------------------------------------------------------
# The file `parse_haproxy_config` is pinned to: frontends on `:::<port> v4v6`, one
# `server <name> <domain>:<port> check inter 5s` per backend, and a `use_backend` whose target
# is a fetch expression — which is why a name is reported only where the config names one.
capture gateway_haproxy.cfg config /etc/haproxy/haproxy.cfg
# --- nginx -------------------------------------------------------------------
# The main config, then the documents its own include lines name one level deep: the same set
# `Collector.nginx` merges. A literal include naming a file this host does not have (this
# gateway's `include /etc/nginx/mime.types;`) is neither captured nor an error — absence of a
# document the parser needs nothing from is the shape the agent already reports as "costs
# nothing and no line".
if capture gateway_nginx_main.conf config /etc/nginx/nginx.conf; then
globs=()
while IFS= read -r line; do
probe="${line#"${line%%[![:space:]]*}"}" # leading whitespace off, for the two tests below
case "$probe" in ''|'#'*) continue ;; esac
case "$probe" in include*) ;; *) continue ;; esac
spec="${probe#include}"
spec="${spec%%;*}"
spec="${spec//[[:space:]]/}"
# Anything with a `*` is a glob include, and the agent merges it — so it gets a fixture, or
# the report below says the script has no name for it. Silent skipping is what would let the
# fixture set stop being the tree the agent reads.
case "$spec" in *"*)"*) continue ;; esac # not a directive then: a fetch expression
case "$spec" in *"*"*) globs+=("$spec") ;; esac
done <"$FIX/gateway_nginx_main.conf"
if [ ${#globs[@]} -eq 0 ]; then
echo " ✗ gateway_nginx_*.conf — the captured main config names no glob include, so the"
echo " vhost fixtures were not refreshed: has the tree the agent merges moved?"
failed=1
fi
for spec in "${globs[@]}"; do
case "$spec" in
/etc/nginx/conf.d/*.conf) capture_included gateway_nginx_conf_d_ "${spec%/*}" ;;
/etc/nginx/stream.d/*.conf) capture_included gateway_nginx_stream_d_ "${spec%/*}" ;;
*)
echo " ✗ $spec — an include directory this script has no fixture name for"
echo " (add a case above and a fixture-pinned test in tests/test_agent.py)"
failed=1
;;
esac
done
fi
# --- the live sockets --------------------------------------------------------
# Both tables, always: `available` is decided across them, because a kernel built without IPv6
# answers tcp and not tcp6 and that is absence rather than a fault (no line, the include rule).
# Keeping only LISTEN and ESTABLISHED is what makes a capture of a busy gateway usable — the
# TIME_WAIT rows in here are the ones the parser must skip, which is exactly what
# tests/test_agent.py asserts against this file.
capture gateway_proc_net_tcp.txt sockets /proc/net/tcp
capture gateway_proc_net_tcp6.txt sockets /proc/net/tcp6
# --- the resolver document ---------------------------------------------------
# The names the two configs declare, read on the box that dials them. This is the only fixture
# whose *question* comes from the repo: the plan is `target_names()` over the configs captured
# above, so the capture asks exactly what the agent will ask and never one name more.
#
# One world-readable, unprivileged read per name — `getent ahostsv4 <name>` then
# `getent ahostsv6 <name>` — because that is libc resolution on the routing box, which is what
# `via: "libc"` claims and what haproxy's `init-addr last,libc,none` performs. No sudo, no `dig`,
# no `host`, no copy tool: a capture that needed a tool the gateway does not ship would record
# something the agent cannot see.
#
# The exit statuses are read as three different sentences. `0` answered; `2` is getent's "no such
# record", which is a fact about the name and becomes `status: "nxdomain"`; `255` is ssh itself
# failing and `127` is a missing `getent`, either of which means the vantage is broken rather than
# the name, so the run stops and the fixture on disk stays exactly as it was.
capture_resolver() {
local scratch rc_plan=0 rc_doc=0
scratch="$(mktemp -d "${TMPDIR:-/tmp}/gateway-resolver.XXXXXX")" || {
echo " ✗ gateway_resolver.json — no scratch directory; the fixture on disk is left alone"
failed=1
return 1
}
if ! run_py -c "$SOCKET_IDENTIFIERS_OUT" plan "$FIX" "$scratch/plan" 2>"$scratch/plan.err"; then
# Last line, not first: an exception's own sentence is the last one, and this is the line the
# operator reads to decide what to fix.
echo " ✗ gateway_resolver.json — $(tail -n 1 "$scratch/plan.err" 2>/dev/null)"
echo " (the name list comes from agent.host_agent.target_names(); the fixture on disk is "
echo " left alone)"
rm -rf "$scratch"
failed=1
return 1
fi
local kind name port index=0 rc4=0 rc6=0 broken=""
while IFS=$'\t' read -r line f1 f2 f3; do
[ "$line" = target ] || continue # the directive line needs no read: it is in the text
kind="$f1"; name="$f2"; port="$f3"
[ "$kind" = name ] || continue # a literal is already an address; the agent asks nothing
rc4=0; rc6=0
ssh "${SSH_OPTS[@]}" "$GATEWAY_SSH" "getent ahostsv4 $name" >"$scratch/$index.v4" \
2>"$scratch/$index.err" || rc4=$?
ssh "${SSH_OPTS[@]}" "$GATEWAY_SSH" "getent ahostsv6 $name" >"$scratch/$index.v6" \
2>"$scratch/$index.err6" || rc6=$?
# `2` on its own is getent's "no such record", which the document records as nxdomain. These
# two are not: one means the box never answered, the other means it has no getent to ask.
if [ "$rc4" -eq 255 ] || [ "$rc6" -eq 255 ]; then
broken="$GATEWAY_SSH did not answer (ssh exited 255)"
break
fi
if [ "$rc4" -eq 127 ] || [ "$rc6" -eq 127 ]; then
broken="getent is not available on $GATEWAY_SSH"
break
fi
index=$((index + 1))
done <"$scratch/plan"
if [ -n "$broken" ]; then
echo " ✗ gateway_resolver.json — $broken; a broken vantage is not a fixture, and the one on "
echo " disk is left alone"
rm -rf "$scratch"
failed=1
return 1
fi
# Fixed, not `date`: the fixture's `ts` is a shape (an entry carries the time it was answered)
# and a live clock would make every re-run diff for a difference no test can read. 2026-10-25
# 00:00:00Z, the date this document was first recorded.
local summary
if ! summary="$(run_py -c "$SOCKET_IDENTIFIERS_OUT" resolver "$scratch/plan" "$scratch" \
"$FIX/gateway_resolver.json" 1792886400 2>"$scratch/doc.err")"; then
echo " ✗ gateway_resolver.json — $(tail -n 1 "$scratch/doc.err" 2>/dev/null)"
rm -rf "$scratch"
failed=1
return 1
fi
echo " ✓ gateway_resolver.json ($summary, on $GATEWAY_SSH)"
rm -rf "$scratch"
}
# The resolver rides on the configs: no config capture means no name list, and a document derived
# from a half-captured tree would be a fixture of the wrong question. Say so rather than being
# silent about the one fixture this run did not refresh.
if [ "$failed" -eq 0 ]; then
capture_resolver
else
echo " - gateway_resolver.json — not attempted: the configs it derives its name list from did "
echo " not all capture"
fi
if [ "$failed" -ne 0 ]; then
echo
echo "capture FAILED — every fixture named above is unchanged on disk. Is $GATEWAY_SSH an ssh"
echo "alias that answers (ssh $GATEWAY_SSH true), and are the paths still where they were?"
exit 1
fi
echo "done — read the diff before committing the fixtures."
Reference in New Issue View Git Blame Copy Permalink