phase: 28_gateway_agent_proxy

**Phase 28 final pass — complete, 100 % green.** All five tasks were already `complete/`; this pass verified the criteria, re-ran the live half, and did the docs pass.

- Gate `bash .agents/validate.sh` **rc=0**: `1650 passed, 15 skipped`, coverage **98.97 %** (floor 94), `ruff` clean, `pyright` **0 errors**, `node --check` clean, hygiene clean (`/tmp/validate-done.log`).
- Live: `pytest -m live --live` **11 passed**; `./scripts/dev-server.sh status` nine sources `ok`; mTLS re-read of `gateway` → agent **1.2.0**, all three sections `available: true`, 6/6 frontends routed, 11 nginx servers + `$targetBackend`, **19** listeners, no core errors (`data/logs/28-finalpass-live.out`); installed agent file byte-identical to this tree (`b0e9ebf1b718…`).
- Criteria: 1 ✅ pytest/coverage · 2 ✅ three sections over a captured tree, `false`/`null` + one `errors` line each elsewhere with `cpu`/`mem`/`disk_io`/`system` intact · 3 ✅ `AGENT_VERSION` 1.2.0 + SCHEMA names all three · 4 ✅ six fixtures verbatim, capture refuses a bad read, 116 socket address columns hold no globally routable endpoint · 5 ✅ gateway runs 1.2.0 and answers over mTLS (transcript `data/logs/28-task05-rollout.md`, `28-finalpass-live.out`) · 6 ✅ validate · 7 ✅ additive: HEAD's 1.1.0 and this 1.2.0 `collect()`ed over one `--root` are key-for-key equal plus the three new sections, `--no-proxy` equal exactly (`data/logs/verify-28-final.py` → 41 PASS, `28-final-verify.out`).
- Docs fixed (README/AGENTS/Containerfile, all re-green): README's opening said "Four dashboards" above a five-row table and said the new sections are "stored in the snapshot" (they are in the **agent's** document — no `/api` payload, no `sample` row, nothing charts them until phase 29); `Containerfile`'s header still said "three dashboards"; AGENTS's coverage figure (98.94 → **98.97 %**), invariant 6 gained the captured-socket-table identifier rule, the agent bullet gained "phase 28 stops at the wire", and the fleet record names the stash-and-compare control plus the volatile listener count. Full record: `data/logs/NOTES-28-final.md`.
- Deviations: none functional; no git index/history touched, phase files untouched (`PLAN.md` and `complete/` unchanged). `uv pip list --outdated` still lists 4 packages (fastapi 0.153→0.161 etc.) — advisory only, dependencies untouched by this phase, left for a deliberate pass.
- Next pending phase: `.agents/phases/todo/29_gateway_dashboard_page/` (the page that consumes these three sections).
This commit is contained in:
2026-10-22 20:24:01 -04:00
parent 5d37444fb0
commit 44d61d3ab9
45 changed files with 4759 additions and 42 deletions
+386 -1
View File
@@ -1,4 +1,12 @@
"""Contract tests for the scripts an operator runs: `deploy/` and `scripts/dev-*.sh`/`.py`.
"""Contract tests for the scripts an operator runs: `deploy/`, `scripts/dev-*.sh`/`.py`, and
`scripts/capture-gateway-config.sh`.
The capture script gets the same treatment as the installer because it can do the same kind of
damage quietly: pointed at a gateway it cannot read, it is six empty files where six verbatim
captures were, and every parser test that reads them then passes against nothing. So it is run
here, against a stub `ssh` and a fake remote tree, and the assertions are about what lands on
disk — including the one edit it makes (a globally routable socket address becomes documentation
space) and the thing it never does (rewrite a config).
There is no shell test framework here — and adding one would buy nothing a subprocess does
not — but phase 10 rests on a sentence that lives in **two** files: the sudoers grant in
@@ -46,6 +54,7 @@ from dashboards.store import SampleStore
ROOT = Path(__file__).resolve().parent.parent
DEPLOY = ROOT / "deploy"
SCRIPTS = ROOT / "scripts"
INSTALLER = DEPLOY / "install-agent.sh"
SUDOERS = DEPLOY / "agent-sudoers"
BASH = shutil.which("bash") or "/bin/bash"
@@ -2334,3 +2343,379 @@ def test_a_local_bundle_still_validates_with_its_own_flag_name(tmp_path: Path) -
assert result.returncode == 2
assert result.stderr.startswith("--cert-bundle"), result.stderr
assert "--fetch-bundle" not in result.stderr
# --- the gateway capture (phase 28 task 04) ------------------------------------------------
#: The script that records `tests/fixtures/gateway_*` — the ground truth the agent's `haproxy`,
#: `nginx` and `sockets` parsers are pinned to. It has no Python-side coverage to speak of (the
#: project's coverage source is `src/dashboards`), so the pins here **run it**, against a fake
#: remote reached through a stubbed `ssh`, the way the installer sandbox stubs `install`.
CAPTURE = SCRIPTS / "capture-gateway-config.sh"
#: The remote documents, and the fixture each one becomes. A new include directory on the gateway
#: means a new entry here *and* a new `case` in the script — the script refuses silently-unknown
#: globs for exactly this reason.
CAPTURE_TARGETS = (
"/etc/haproxy/haproxy.cfg",
"/etc/nginx/nginx.conf",
"/etc/nginx/conf.d/*.conf",
"/etc/nginx/stream.d/*.conf",
"/proc/net/tcp",
"/proc/net/tcp6",
)
#: The stub `ssh`. It drops the options the script passes, records every call, and answers `cat`
#: and `ls` from `$FAKE_REMOTE` as if that directory were the remote filesystem's `/`. Nothing
#: here can reach a network: the point is to exercise the script's own branches (the include
#: globs, the refusal rule, the socket-table edit) with no gateway in sight.
CAPTURE_SSH_STUB = """#!/bin/sh
printf '%s\\n' "$*" >> "$STUB_LOG"
while [ $# -gt 0 ]; do
case "$1" in
-o) shift 2 ;;
-n) shift ;;
*) break ;;
esac
done
host="$1"; shift
command="$*"
if [ "${FAIL_SSH:-0}" = "1" ]; then
echo "ssh: connect to host $host port 22: Connection refused" >&2
exit 255
fi
case "$command" in
"cat "*)
path="${command#cat }"
file="$FAKE_REMOTE$path"
[ -f "$file" ] || { echo "cat: $path: No such file or directory" >&2; exit 1; }
cat "$file"
;;
"ls "*)
spec="${command#ls }"
dir="${spec%/*}"
glob="${spec##*/}"
found=0
for file in "$FAKE_REMOTE$dir"/$glob; do
[ -e "$file" ] || continue
found=1
printf '%s\\n' "${file#"$FAKE_REMOTE"}"
done
[ "$found" = 1 ] || { echo "ls: cannot access '$spec': No such file or directory" >&2; exit 2; }
;;
*)
echo "stub ssh: unsupported remote command: $command" >&2
exit 2
;;
esac
"""
#: A gateway tree small enough to assert on whole, with the two shapes the parsers exist for: a
#: `use_backend` whose target is a fetch expression, and a stream `proxy_pass` of a map variable.
CAPTURE_FAKE_GATEWAY: dict[str, str] = {
"/etc/haproxy/haproxy.cfg": (
"global\n maxconn 4096\n"
"defaults\n mode tcp\n"
"frontend haproxy-22\n bind :::22 v4v6\n default_backend gitea-2222\n"
"frontend haproxy-8443\n bind :::8443 v4v6\n"
" use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/maps,b0)\n"
"backend gitea-2222\n server gitea gitea.reeselink.com:2222 check inter 5s\n"
),
"/etc/nginx/nginx.conf": (
"user nginx;\n"
"stream {\n include /etc/nginx/stream.d/*.conf;\n}\n"
"http {\n"
" include /etc/nginx/mime.types;\n"
" server { listen 80; listen [::]:80; server_name _; return 301 https://$host; }\n"
" include /etc/nginx/conf.d/*.conf;\n}\n"
),
"/etc/nginx/conf.d/http.conf": (
"server {\n"
" listen 4443 ssl proxy_protocol;\n"
" server_name nextcloud.reeseapps.com;\n"
" ssl_certificate_key /etc/letsencrypt/live/nextcloud.reeseapps.com/privkey.pem;\n"
" location / { proxy_pass http://nextcloud.reeselink.com:11000; }\n}\n"
),
"/etc/nginx/conf.d/tls.conf": (
"server {\n listen 8443 ssl;\n server_name brain.reeseapps.com;\n"
" location / { proxy_pass http://reeseapps-brain-of-reese.reeselink.com:443; }\n}\n"
),
"/etc/nginx/stream.d/stream.conf": (
"map $ssl_preread_server_name $targetBackend {\n"
" default 127.0.0.1:4443;\n"
" brain.example reeseapps-brain-of-reese.reeselink.com:443;\n}\n"
"server {\n listen 443;\n proxy_pass $targetBackend;\n}\n"
),
# 8.8.8.8 and 2603:6013::1 are globally routable, `::ffff:8.8.8.8` is a visitor behind a v6
# socket, and 203.0.113.7 is already documentation space. The tx/rx queue pair on the last row
# is spelled like an address column and is not one.
"/proc/net/tcp": (
" sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid"
" timeout inode\n"
" 0: 00000000:1EFF 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0"
" 0 1 1 0000000000000000 100 0 0 10 0\n"
" 1: 1B01010A:01BB 08080808:911A 01 00000000:00000000 00:00000000 00000000 101"
" 0 2 1 00000000d22dedba 20 4 28 10 -1\n"
" 2: 0100007F:23A0 017100CB:C0A8 06 08080808:08080808 00:00000000 00000000 0"
" 0 3 3 0000000000000000 20 0 0 10 0\n"
),
"/proc/net/tcp6": (
" sl local_address remote_address"
" st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode\n"
" 0: 00000000000000000000000000000000:01BB 00000000000000000000000000000000:0000 0A"
" 00000000:00000000 00:00000000 00000000 0 0 4 1 0000000000000000 100 0 0 10 0\n"
" 1: 1360032601010031FF005450E0B284FE:01BB 0000000000000000FFFF000008080808:C0AC 01"
" 00000000:00000000 00:00000000 00000000 101 0 5 1 0000000000000000 63 4 26 10 -1\n"
),
}
def _capture_sandbox(
tmp_path: Path,
gateway: dict[str, str] | None = None,
*,
recorded: dict[str, str] | None = None,
fail_ssh: bool = False,
alias: str | None = None,
) -> tuple[Path, Path, Path, dict[str, str]]:
"""A copy of the capture script, a fake gateway, and a PATH whose `ssh` reaches only the fake.
Returns (script, fixtures dir, stub call log, env). The script is copied into
`<tmp>/scripts/` because it derives `tests/fixtures/` from its own location, which is what
keeps the repository's fixtures out of reach of a test.
"""
script = tmp_path / "scripts" / CAPTURE.name
script.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(CAPTURE, script)
fake = tmp_path / "gateway"
for path, body in {**CAPTURE_FAKE_GATEWAY, **(gateway or {})}.items():
target = fake / path.lstrip("/")
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(body, encoding="utf-8")
fixtures = tmp_path / "tests" / "fixtures"
fixtures.mkdir(parents=True, exist_ok=True)
for name, body in (recorded or {}).items():
(fixtures / name).write_text(body, encoding="utf-8")
stubs = tmp_path / "stubs"
stubs.mkdir()
stub = stubs / "ssh"
stub.write_text(CAPTURE_SSH_STUB, encoding="utf-8")
stub.chmod(0o755)
call_log = tmp_path / "ssh-calls.log"
call_log.write_text("", encoding="utf-8")
env = {**os.environ,
"PATH": f"{stubs}{os.pathsep}{os.environ['PATH']}",
"STUB_LOG": str(call_log),
"FAKE_REMOTE": str(fake),
"FAIL_SSH": "1" if fail_ssh else "0"}
if alias is not None:
env["GATEWAY_SSH"] = alias
return script, fixtures, call_log, env
def _capture(tmp_path: Path, **kwargs: object) -> subprocess.CompletedProcess[str]:
script, _fixtures, _log, env = _capture_sandbox(tmp_path, **kwargs) # type: ignore[arg-type]
return subprocess.run([BASH, str(script)], capture_output=True, text=True, env=env, check=False)
def _fixture_body(path: Path) -> str:
"""A fixture without its `#` provenance block — the bytes the host answered."""
return "".join(line for line in path.read_text().splitlines(keepends=True)
if not line.startswith("#"))
def test_the_gateway_capture_exists_and_asks_for_no_privilege() -> None:
"""Six world-readable reads over an ssh key: nothing here needs root or a rewrite tool."""
assert CAPTURE.is_file(), "scripts/capture-gateway-config.sh is missing (phase 28 task 04)"
assert os.access(CAPTURE, os.X_OK)
text = CAPTURE.read_text()
for path in CAPTURE_TARGETS:
assert path in text, f"{path} is no longer captured"
code = "\n".join(line for line in text.splitlines() if not line.lstrip().startswith("#"))
assert "sudo" not in code, (
"the capture must read exactly what dashboards-agent can read; a pass that needed root "
"would record more than the agent can see (the smartctl note in capture-fixtures.sh)"
)
for tool in ("sed ", "awk ", " tr ", "cut "):
assert tool not in text, f"{tool.strip()} would be a text filter between the read and a fixture"
def test_the_gateway_capture_dials_the_alias_and_nothing_else(tmp_path: Path) -> None:
"""No hardcoded address: every remote command goes to `$GATEWAY_SSH`, default `gateway-ops`."""
text = CAPTURE.read_text()
assert 'GATEWAY_SSH="${GATEWAY_SSH:-gateway-ops}"' in text
# Comments may name an address (`0.0.0.0` is a shape the parser has to survive); code may not.
code = "\n".join(line for line in text.splitlines() if not line.lstrip().startswith("#"))
stray = [a for a in re.findall(r"\b(?:\d{1,3}\.){3}\d{1,3}\b", code)
if not a.startswith("203.0.113.")]
assert not stray, f"the script dials or embeds an address: {stray}"
_script, _fixtures, log, env = _capture_sandbox(tmp_path)
subprocess.run([BASH, str(_script)], capture_output=True, text=True, env=env, check=False)
assert _dialled_hosts(log) == {"gateway-ops"}
other = _capture_sandbox(tmp_path / "alias", alias="lab-gw")
subprocess.run([BASH, str(other[0])], capture_output=True, text=True, env=other[3], check=False)
assert _dialled_hosts(other[2]) == {"lab-gw"}
def _dialled_hosts(log: Path) -> set[str]:
"""The host argument of every ssh the stub was called for — options are not hosts."""
hosts = set()
for line in log.read_text().splitlines():
tokens = [token for token in line.split() if not token.startswith("-") and "=" not in token]
if tokens:
hosts.add(tokens[0])
return hosts
def test_the_gateway_capture_records_every_document_of_the_tree_it_read(tmp_path: Path) -> None:
"""The happy path, run: one fixture per remote document, include globs and all.
Two files in `conf.d` is deliberate. The script reads the include globs out of the captured
main config rather than a hardcoded pair of directories, so a gateway that adds a vhost file
gets it captured, and `files` in the payload stays the tree the config actually merges.
"""
result = _capture(tmp_path)
assert result.returncode == 0, result.stdout + result.stderr
fixtures = tmp_path / "tests" / "fixtures"
expected = {"gateway_haproxy.cfg", "gateway_nginx_main.conf",
"gateway_nginx_conf_d_http.conf", "gateway_nginx_conf_d_tls.conf",
"gateway_nginx_stream_d_stream.conf",
"gateway_proc_net_tcp.txt", "gateway_proc_net_tcp6.txt"}
assert {p.name for p in fixtures.iterdir()} == expected, result.stdout
assert _fixture_body(fixtures / "gateway_haproxy.cfg") == \
CAPTURE_FAKE_GATEWAY["/etc/haproxy/haproxy.cfg"]
assert _fixture_body(fixtures / "gateway_nginx_conf_d_tls.conf") == \
CAPTURE_FAKE_GATEWAY["/etc/nginx/conf.d/tls.conf"]
# `mime.types` is included by the config and absent on the host: no fixture, no complaint —
# the same absence-the-agent-reports rule, one level up.
assert not (fixtures / "gateway_nginx_mime.types").exists()
def test_the_gateway_capture_writes_the_domains_it_read_verbatim(tmp_path: Path) -> None:
"""Nothing is scrubbed from a config, and the domains are the proof.
The `server_name` and the `proxy_pass` target beside it are the ingress map, and the
`ssl_certificate_key` is a path rather than a key — so the file the fixture holds is the file
the gateway runs, byte for byte. The one exception in this script is the socket tables'
globally routable addresses, which `capture-fixtures.sh`'s identifier rule requires.
"""
_capture(tmp_path)
vhosts = (tmp_path / "tests" / "fixtures" / "gateway_nginx_conf_d_http.conf").read_text()
assert "server_name nextcloud.reeseapps.com;" in vhosts
assert "proxy_pass http://nextcloud.reeselink.com:11000;" in vhosts
assert "ssl_certificate_key /etc/letsencrypt/live/nextcloud.reeseapps.com/privkey.pem;" in vhosts
assert "SCRUB" not in vhosts and "REDACTED" not in vhosts
assert next((line for line in vhosts.splitlines()
if line.startswith("# Verbatim /etc/nginx/conf.d/http.conf from gateway-ops")),
None), "every fixture says which remote document it is"
def test_a_dead_gateway_leaves_the_recorded_fixtures_alone(tmp_path: Path) -> None:
"""The `slots_gufo.json` rule: a capture script must never turn six good fixtures into six
empty files, because then the parser tests pass against nothing (or fail for the wrong
reason) and the ground truth is gone."""
keep = {"gateway_haproxy.cfg": "# old\nfrontend kept\n bind :::22 v4v6\n",
"gateway_proc_net_tcp.txt": " 0: 00000000:1EFF 00000000:0000 0A x\n"}
result = _capture(tmp_path, recorded=keep, fail_ssh=True)
assert result.returncode == 1, result.stdout
for name, body in keep.items():
assert (tmp_path / "tests" / "fixtures" / name).read_text() == body, \
f"{name} was overwritten by a read that failed"
for name in keep:
assert f"✗ {name}" in result.stdout, result.stdout
assert "Connection refused" in result.stdout, "one line names the reason, per fixture"
assert sorted(p.name for p in (tmp_path / "tests" / "fixtures").iterdir()) == sorted(keep), \
"a failed capture leaves no temp files behind"
def test_an_empty_read_is_refused_the_way_a_failed_one_is(tmp_path: Path) -> None:
"""`cat` exiting 0 on an empty file is still not a capture."""
keep = {"gateway_proc_net_tcp.txt": " 0: 00000000:1EFF 00000000:0000 0A keep me\n"}
result = _capture(tmp_path, gateway={"/proc/net/tcp": ""}, recorded=keep)
assert result.returncode == 1, result.stdout
assert "gateway_proc_net_tcp.txt" in result.stdout
assert "empty" in result.stdout
assert (tmp_path / "tests" / "fixtures" / "gateway_proc_net_tcp.txt").read_text() == \
next(iter(keep.values()))
def test_an_include_dir_with_no_fixture_name_is_reported_and_fails(tmp_path: Path) -> None:
"""A config that moved its vhosts is a loud failure, not a fixture set that quietly stopped
being the tree the agent merges."""
moved = {"/etc/nginx/nginx.conf": (
"http {\n include /etc/nginx/sites-enabled/*.conf;\n}\n"
"stream {\n include /etc/nginx/stream.d/*.conf;\n}\n")}
result = _capture(tmp_path, gateway=moved)
assert result.returncode == 1, result.stdout
assert "/etc/nginx/sites-enabled/*.conf" in result.stdout
assert not (tmp_path / "tests" / "fixtures" / "gateway_nginx_sites_enabled_web.conf").exists()
def test_a_config_with_no_glob_include_says_the_vhost_fixtures_were_not_refreshed(
tmp_path: Path) -> None:
keep = {"gateway_nginx_conf_d_http.conf": "# keep\nserver {\n listen 4443;\n}\n"}
result = _capture(tmp_path, gateway={"/etc/nginx/nginx.conf": "user nginx;\nevents { }\n"},
recorded=keep)
assert result.returncode == 1, result.stdout
assert "no glob include" in result.stdout
assert (tmp_path / "tests" / "fixtures" / "gateway_nginx_conf_d_http.conf").read_text() == \
next(iter(keep.values()))
def test_the_socket_capture_publishes_no_globally_routable_endpoint(tmp_path: Path) -> None:
"""The one edit the capture makes, run rather than asserted in prose.
Two rules at once, which is what makes the socket fixtures both usable and publishable: a
globally routable address (including one behind `::ffff:`) becomes documentation space, and
everything else — every port, every private/loopback/wildcard word, every other column, and
the column widths themselves — is byte-for-byte what the kernel answered. A tx/rx queue pair
is spelled exactly like an address column and must survive untouched, which is why the edit
reads two columns and not the line.
"""
_capture(tmp_path)
fixtures = tmp_path / "tests" / "fixtures"
v4 = _fixture_body(fixtures / "gateway_proc_net_tcp.txt").splitlines()
v6 = _fixture_body(fixtures / "gateway_proc_net_tcp6.txt").splitlines()
raw_v4 = CAPTURE_FAKE_GATEWAY["/proc/net/tcp"].splitlines()
raw_v6 = CAPTURE_FAKE_GATEWAY["/proc/net/tcp6"].splitlines()
assert len(v4) == len(raw_v4) and len(v6) == len(raw_v6), "a row was added or lost"
assert v4[1] == raw_v4[1], "a wildcard listener must survive untouched"
assert "08080808:911A" not in "".join(v4), "8.8.8.8 is still in the fixture"
assert "017100CB:911A" in v4[2], v4[2] # 203.0.113.1, the port kept
assert len(v4[2]) == len(raw_v4[2]), "the replacement must be exactly as wide as the original"
assert v4[3] == raw_v4[3], (
"an already-documentation address and the tx/rx queue pair on that row both moved: only "
"the two address columns may be looked at")
assert "08080808:08080808" in v4[3], v4[3]
assert "1360032601010031FF005450E0B284FE" not in "".join(v6), "a global v6 is left in"
assert "B80D0120000000000000000001000000:01BB" in v6[2], v6[2] # 2001:db8::1, port kept
assert "FFFF0000017100CB:C0AC" in v6[2] or "FFFF0000" in v6[2], \
"a v4-mapped row must stay v4-mapped, or the parser's branch for it stops being exercised"
assert v6[1] == raw_v6[1], "the wildcard v6 listener moved"
assert len(v6[2]) == len(raw_v6[2])
def test_the_socket_capture_is_idempotent_so_a_re_run_does_not_churn(tmp_path: Path) -> None:
"""Re-recording the same moment must not diff, or every capture moves the fixture twice.
Documentation space is not globally routable, so the second pass leaves it alone — the same
property that makes the drive-serial scrub re-runnable.
"""
once = tmp_path / "once"
_capture(once)
again = _capture_sandbox(tmp_path / "twice",
gateway={"/proc/net/tcp": _fixture_body(
once / "tests" / "fixtures" / "gateway_proc_net_tcp.txt")})
result = subprocess.run([BASH, str(again[0])], capture_output=True, text=True,
env=again[3], check=False)
assert result.returncode == 0, result.stdout + result.stderr
assert _fixture_body(again[1] / "gateway_proc_net_tcp.txt") == _fixture_body(
once / "tests" / "fixtures" / "gateway_proc_net_tcp.txt")