services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 33_gateway_route_switching

Browse Source 
Phase 33 completion pass is done. Final report:

## Phase 33 — completion report

**All 8 task files complete** (`.agents/phases/complete/33_gateway_route_switching/`), full gate green, docs verified.

### Work in this pass

1. **`tests/test_routectl.py` added** (17 tests, all green) — exercises the *real* helper binary unprivileged against a fake tree (`--root`), the real agent parser, the captured gateway fixtures, and stubbed `nginx`/`haproxy`/`systemctl` on `PATH`. Pins the full protocol (locate → state → dry-run → temp-file/fsync/`os.replace` write → checker → restore on checker failure → reload-never-restart → re-read proof → sidecar record), all three route shapes (haproxy `server`, nginx `proxy_pass`, nginx stream `map` via its sink-value id), `drift` / unknown-id / `map-default` refusals, `list`/`verify`, and the `--serve` peer gate (`SO_PEERCRED` uid check, one JSON line per request, exact body shape).

2. **Real helper defect fixed** — `deploy/dashboards-routectl.py`: `main()`'s early-return paths returned `({"ok": False, …}, EXIT_*)` tuples, so `sys.exit()` printed a Python repr instead of a JSON line (usage errors, agent-import refusal). Now prints `json.dumps(…)` and returns only the code, keeping the one-JSON-line contract.

3. **Documentation consistency fixes** (task 08 criterion 6):
   - `README.md`: garbled verdict-colour sentence on the switch ("green and to the position") → "in the **on** position and tinted `--gate-on`" (the existence-pair contract).
   - `README.md`: audit sentence now names all three event kinds — `route-disabled` / `route-enabled` / **`route-denied`** (the code stores one per refused call; dry runs store none).
   - `page-gateway.js`: replaced the stale pre-33 module docstring ("the switch is not wired") with the two-arm `GATEWAY_ALLOW_WRITES` contract.
   - `AGENTS.md`: every phase-33 claim re-verified against code (the ten gates, the 35 s/30 s budget pair, `gateway.writes_enabled`, `/api/summary`'s `disabled`, the 1.4.0 agent bullet, the gap list) — no changes needed.

### Verification
- `uv run pytest --cov`: **1970 passed**, 18 skipped — coverage **98.74%** (floor 94%)
- `ruff` clean · `pyright` 0 errors · `bash .agents/validate.sh` **exit 0** (incl. `node --check` on every page script)
- Dev server running; all 10 sources `ok`; bundles exported

### Live infrastructure
**No live changes made.** The live drill remains unexecuted per `data/logs/NOTES-33-switching.md` (no operator approval; checker binaries unavailable in the podman proxy units). All routes stay enabled, config hashes unchanged, and the sandbox rehearsal + backup tree are in place for the operator's own `DANGER` call. All changes are in the working tree only.
This commit is contained in:
ducoterra
2026-11-01 11:50:51 -05:00
parent 0271387053
commit 86c8bf4992
91 changed files with 7568 additions and 519 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_agent.py
+593 -41
View File
@@ -27,6 +27,7 @@ import threading
import time
import urllib.error
import urllib.request
from contextlib import suppress
from datetime import UTC, datetime, timedelta
from http.server import ThreadingHTTPServer
from pathlib import Path
@@ -838,8 +839,10 @@ def test_the_captured_haproxy_shape_parses_into_frontends_and_backends() -> None
[{"addr": "::", "port": port}] for port in (22, 2000, 5201, 8080, 25565, 25566)]
assert [frontend["default_backend"] for frontend in parsed["frontends"]] == [
"svc-2222", "svc-2000", "svc-5201", "svc-8080", "svc-25565", "svc-25566"]
# no line carries the helper's marker, so none is off (the flag is present, not inferred)
assert [backend["servers"] for backend in parsed["backends"]] == [
[{"name": f"svc{port}", "target": f"svc{port}.example.com", "port": port}]
[{"name": f"svc{port}", "target": f"svc{port}.example.com", "port": port,
"disabled": False}]
for port in (2222, 2000, 5201, 8080, 25565, 25566)]
# routing is the names themselves, and the config joins: every frontend's default_backend is
# one of the backends listed beside it (phase 29 draws its pipes from exactly this)
@@ -865,9 +868,10 @@ def test_the_recorded_gateway_haproxy_config_parses_too() -> None:
"testcraft-25565", "nimcraft-25566"]
assert [(backend["name"], backend["servers"]) for backend in parsed["backends"]][:2] == [
("gitea-2222", [{"name": "gitea", "target": "reeseapps-gitea.reeselink.com",
"port": 2222}]),
("sec-gitea-2000", [{"name": "sec-gitea", "target": "stackexpected-gitea.reeselink.com",
"port": 2222}]),
"port": 2222, "disabled": False}]),
("sec-gitea-2000", [{"name": "sec-gitea",
"target": "stackexpected-gitea.reeselink.com",
"port": 2222, "disabled": False}]),
]
# every frontend's target exists beside it — the join phase 29 draws its pipes from
names = {backend["name"] for backend in parsed["backends"]}
@@ -912,9 +916,10 @@ def test_a_server_that_names_no_port_keeps_null_and_unknown_lines_are_ignored()
" http-request deny deny_status 403\n"
)
assert parsed["backends"][0]["servers"] == [
{"name": "withport", "target": "10.1.0.9", "port": 8080},
{"name": "noport", "target": "10.1.0.10", "port": None},
{"name": "servicename", "target": "host.example.com:http", "port": None},
{"name": "withport", "target": "10.1.0.9", "port": 8080, "disabled": False},
{"name": "noport", "target": "10.1.0.10", "port": None, "disabled": False},
{"name": "servicename", "target": "host.example.com:http", "port": None,
"disabled": False},
]
# a `backend` body never grows frontend fields, however many directives it holds
assert "binds" not in parsed["backends"][0]
@@ -933,7 +938,7 @@ def test_a_listen_section_is_reported_as_the_pair_haproxy_says_it_is() -> None:
"default_backend": "stats", "backends": ["stats"]}]
assert parsed["backends"] == [{"name": "stats",
"servers": [{"name": "own", "target": "127.0.0.1",
"port": 8405}]}]
"port": 8405, "disabled": False}]}]
def test_a_backend_computed_per_request_contributes_no_name() -> None:
@@ -1249,27 +1254,30 @@ def test_the_captured_nginx_shape_parses_into_servers_maps_and_targets() -> None
assert [server["proxy_passes"] for server in parsed["servers"]] == [
[],
# the stream form: no `location` exists, and the target is a variable, not an address
[{"location": None, "scheme": None, "target": None, "variable": "targetBackend"}],
[{"location": None, "scheme": None, "target": None, "variable": "targetBackend",
"disabled": False}],
[{"location": "/", "scheme": "http", "variable": None,
"target": {"host": "nextcloud.internal.test", "port": 11000}}],
"target": {"host": "nextcloud.internal.test", "port": 11000}, "disabled": False}],
[{"location": "/", "scheme": "http", "variable": None,
"target": {"host": "127.0.0.1", "port": 8081}}],
"target": {"host": "127.0.0.1", "port": 8081}, "disabled": False}],
]
maps = by_variable(parsed["maps"])
assert set(maps) == {"connection_upgrade", "targetBackend"}
# the variable name is spelled one way on both sides of the join, and the `$` is nginx's
# syntax rather than part of the name — so it is dropped here once, not in the page
assert maps["targetBackend"]["entries"] == [
{"key": "default", "target": {"host": "127.0.0.1", "port": 4443}},
{"key": "default", "target": {"host": "127.0.0.1", "port": 4443},
"disabled": False},
{"key": "brain.example.com", "target": {"host": "brain-of-reese.internal.test",
"port": 443}},
"port": 443}, "disabled": False},
]
# …and the `proxy_pass $targetBackend` above still carries `target: null`: the map decides
# per connection, so merging it here would be a number the wire never published
assert parsed["servers"][1]["proxy_passes"][0]["target"] is None
# `$connection_upgrade`'s values are header words, not hosts: keys reported, targets null
assert maps["connection_upgrade"]["entries"] == [
{"key": "default", "target": None}, {"key": "", "target": None}]
{"key": "default", "target": None, "disabled": False},
{"key": "", "target": None, "disabled": False}]
# the include lines, as written, in document order — including the one file this tree has no
# opinion about because it is not a config (`mime.types`)
assert parsed["includes"] == ["/etc/nginx/stream.d/*.conf", "/etc/nginx/mime.types",
@@ -1325,15 +1333,18 @@ def test_a_proxy_pass_reports_the_address_the_variable_or_neither() -> None:
assert parsed["servers"][0]["proxy_passes"] == [
# the trailing path is dropped the way haproxy's `check inter 5s` is: it is not a place
{"location": "/a/", "scheme": "http", "target": {"host": "h1.test", "port": 8080},
"variable": None},
"variable": None, "disabled": False},
# an `upstream` name is a host with no port — the name is the fact, the port is not
{"location": "/b/", "scheme": "http", "target": {"host": "pool_name", "port": None},
"variable": None},
{"location": "/c/", "scheme": None, "target": None, "variable": "backend"},
"variable": None, "disabled": False},
{"location": "/c/", "scheme": None, "target": None, "variable": "backend",
"disabled": False},
# a unix socket is neither a host nor a variable: the line is reported with nulls
{"location": "/d/", "scheme": None, "target": None, "variable": None},
{"location": "/d/", "scheme": None, "target": None, "variable": None,
"disabled": False},
# a colon that is not a port either: the scheme is a fact, the host would be a guess
{"location": "/e/", "scheme": "http", "target": None, "variable": None},
{"location": "/e/", "scheme": "http", "target": None, "variable": None,
"disabled": False},
]
@@ -1371,9 +1382,9 @@ def test_an_upstream_block_keeps_its_server_lines_as_targets() -> None:
"http { server { listen 80; } }"
)
assert parsed["upstreams"] == [{"name": "pool_name", "servers": [
{"host": "10.1.0.9", "port": 8080},
{"host": "backend.test", "port": None},
{"host": "::1", "port": 8080},
{"host": "10.1.0.9", "port": 8080, "disabled": False},
{"host": "backend.test", "port": None, "disabled": False},
{"host": "::1", "port": 8080, "disabled": False},
]}]
# an upstream's `server` lines are not server blocks, however many of them there are
assert len(parsed["servers"]) == 1
@@ -2179,8 +2190,11 @@ def agent_server():
host_agent.Handler.collector = host_agent.Collector("/")
host_agent.Handler.token = token
# Cleared on purpose: `Handler` is process-wide state, so a token-mode server started
# after a TLS test in this file must not inherit that test's bundle.
# after a TLS test in this file must not inherit that test's bundle — or the write
# path's gate, which the bundle check is the first line of.
host_agent.Handler.bundle = None
host_agent.Handler.allow_writes = False
host_agent.Handler.routectl_socket = None
httpd = ThreadingHTTPServer(("127.0.0.1", 0), host_agent.Handler)
servers.append(httpd)
threading.Thread(target=httpd.serve_forever, daemon=True).start()
@@ -2253,15 +2267,6 @@ def test_token_is_required_when_configured_but_health_stays_open(agent_server) -
assert _get(base + "/health")[0] == 200
def test_agent_has_no_write_path(agent_server) -> None:
"""No do_POST/do_PUT exists: read-only is structural, not a policy toggle."""
base, _port = agent_server()
request = urllib.request.Request(base + "/snapshot", data=b"{}", method="POST")
with pytest.raises(urllib.error.HTTPError) as excinfo:
urllib.request.urlopen(request, timeout=5)
assert excinfo.value.code in (405, 501)
def test_refuses_to_bind_off_loopback_without_a_token(monkeypatch) -> None:
"""0.0.0.0 with no token would hand host inventory to the whole LAN."""
monkeypatch.delenv("AGENT_TOKEN", raising=False)
@@ -2452,6 +2457,10 @@ def tls_agent_server():
host_agent.Handler.collector = host_agent.Collector("/", smart=False)
host_agent.Handler.token = token
host_agent.Handler.bundle = bundle
# Cleared on purpose, like `bundle` above: a writes-off server started after a
# `POST /route` test must not inherit that test's write gate or helper socket.
host_agent.Handler.allow_writes = False
host_agent.Handler.routectl_socket = None
httpd = ThreadingHTTPServer(("127.0.0.1", 0), host_agent.Handler)
httpd.socket = bundle.context.wrap_socket(httpd.socket, server_side=True)
servers.append(httpd)
@@ -2463,6 +2472,8 @@ def tls_agent_server():
httpd.shutdown()
httpd.server_close()
host_agent.Handler.bundle = None
host_agent.Handler.allow_writes = False
host_agent.Handler.routectl_socket = None
def _json_or_raw(body: bytes) -> dict:
@@ -2757,14 +2768,19 @@ def test_a_bundle_allows_the_off_loopback_bind_without_a_token(
tls_fleet: SimpleNamespace, tmp_path: Path, monkeypatch) -> None:
"""The client certificate *is* the protection, so the exit-2 refusal does not apply."""
served: list[tuple] = []
seen_kwargs: list[dict] = []
monkeypatch.setattr(host_agent, "serve",
lambda *args: served.append(args))
lambda *args, **kwargs: served.append(args) or seen_kwargs.append(kwargs))
path = write_bundle(tmp_path, tls_fleet.bundle)
assert host_agent.main(["--host", "0.0.0.0", "--port", "0", "--cert-bundle", str(path)]) == 0
assert served, "a bundle-configured agent must start"
host_arg, _port, _collector, token_arg, bundle_arg = served[0]
assert (host_arg, token_arg) == ("0.0.0.0", None)
assert isinstance(bundle_arg, host_agent.TlsBundle)
# 1.4.0's defaults: the write path is off, and the helper's home is the standard socket
# path unless the flag names another one (an empty value is the 503 arm: no helper here).
assert seen_kwargs[0]["allow_writes"] is False
assert seen_kwargs[0]["routectl_socket"] == "/run/dashboards-routectl.sock"
@pytest.mark.parametrize("extra", [[], ["--cert-bundle", ""]])
@@ -2819,22 +2835,157 @@ def test_the_schema_names_the_three_proxy_sections() -> None:
assert "--no-proxy" in doc and "--haproxy-config" in doc and "--nginx-config" in doc
def test_the_agent_version_is_1_3_0_and_the_schema_names_the_resolver_section() -> None:
"""1.3.0 is the version that resolves the names the configs declare.
def test_the_agent_version_is_1_4_0_and_the_schema_documents_the_route_marker() -> None:
"""1.4.0 is the version that reads the route switch's marker off the route line.
The bump is how the fleet tells an agent that publishes `resolver` from one that does not —
`--status`, `/health` and this test all read the same constant, and `sources/gateway.py`'s
"the agent is too old" arm keys on the sections the version belongs to. The SCHEMA block is
`GET /schema`'s body, so it names the section, its three-way `available`, and `--no-resolve`,
the flag that removes it in silence.
The bump is how the fleet tells an agent that publishes `disabled` from one that does not —
`--status`, `/health` and this test all read the same constant. The SCHEMA block is
`GET /schema`'s body, so it documents the flag on both config sections and the marker's
spelling, and `--no-resolve` stays named: the flag that removes the resolver section in
silence.
"""
assert host_agent.AGENT_VERSION == "1.3.0"
assert host_agent.AGENT_VERSION == "1.4.0"
doc = host_agent.__doc__ or ""
assert schema_documents("resolver"), "resolver is not documented in the SCHEMA block"
block = doc.split(" resolver ", 1)[-1]
assert "available" in block, "the resolver block does not document its three-way answer"
assert "--no-resolve" in doc, "the flag that removes the section is undocumented"
assert "libc" in block, "how the addresses were obtained is not published in the schema"
assert "disabled" in doc, "the marker read is not documented in the SCHEMA block"
assert "#off" in doc, "the marker's spelling is not documented in the schema"
def test_the_marker_grammar_is_one_constant_shared_by_writer_and_reader() -> None:
"""`split_marker` is the read side of the switch and `marker_for` the write side; both
spell the same grammar, which is the whole of the read side (phase 33 task 04).
A marked line keeps its text and gains a name; anything that is not exactly `#off <name>`
— an extra word, a bare `#off`, or the operator's own comment — is not a switch, and a line
without a comment is a line.
"""
assert host_agent.ROUTE_MARKER == "#off"
line, name = host_agent.split_marker("server s1 10.1.1.9:443 #off s1")
assert line == "server s1 10.1.1.9:443" and name == "s1"
# trailing whitespace after the name is the operator's formatting, not part of the name
assert host_agent.split_marker("server s1 10.1.1.9:443 #off s1 ") == (
"server s1 10.1.1.9:443", "s1")
line, name = host_agent.split_marker("server s1 10.1.1.9:443 #off")
assert name is None, "a bare #off with no name is a comment, not a switch"
line, name = host_agent.split_marker("server s1 10.1.1.9:443 #off s1 extra")
assert name is None, "two words after the marker is a comment, not a switch"
line, name = host_agent.split_marker("server s1 10.1.1.9:443 # off s1")
assert name is None, "a comment that merely mentions the word is not a switch"
line, name = host_agent.split_marker("server s1 10.1.1.9:443")
assert name is None
# the write side spells what the read side parses: the round trip is exact
for name_ in ("s1", "pool:8080", "web.example.com"):
marked, marker = host_agent.split_marker(f"line {host_agent.marker_for(name_)}")
assert marker == name_ and marked == "line"
def test_config_line_index_finds_the_line_the_marker_names() -> None:
"""The helper edits by line number, and the line is found by the marker, never the route's
text (a target string appears in many lines; the marker names exactly one of them)."""
text = "backend be\n server s1 10.1.1.9:443\n server s2 10.1.1.10:443 #off s1\n"
assert host_agent.config_line_index(text, "s1") == 3
assert host_agent.config_line_index(text, "s2") == -1
# a commented-out line is not a route: its marker is part of the comment, not a switch
text = "# server s3 10.1.1.11:443 #off s3\n"
assert host_agent.config_line_index(text, "s3") == -1
def test_the_haproxy_parser_reports_a_marked_server_line_disabled() -> None:
"""The read side of the switch: a `server` line carrying the helper's marker is
`disabled: true` and every other line `disabled: false` — and the marked line is still a
line: its target is parsed and its text kept, because the helper rewrites the value, not
the line, and a parser that dropped it would lose the route the switch is about."""
parsed = host_agent.parse_haproxy_config(
"backend be\n"
" server on 10.1.1.9:443\n"
" server off 10.1.1.10:443 #off off\n"
" server note 10.1.1.11:443 # a comment that is not the switch\n"
)
assert parsed["backends"][0]["servers"] == [
{"name": "on", "target": "10.1.1.9", "port": 443, "disabled": False},
{"name": "off", "target": "10.1.1.10", "port": 443, "disabled": True},
{"name": "note", "target": "10.1.1.11", "port": 443, "disabled": False},
]
def test_the_nginx_parser_reports_a_marked_line_disabled_in_all_three_shapes() -> None:
"""One marker, three route lines: a `proxy_pass`, an `upstream` member, a `map` entry.
The stream form is the fleet's own: a `proxy_pass $var` with no location, a `map` whose
entries are the SNI fan-out, and an `upstream` the http vhosts name. Each line's marker is
its own line's `disabled`, and an unmarked line beside a marked one is still on."""
parsed = host_agent.parse_nginx_config(
"stream {\n"
" server { listen 443; proxy_pass $targetBackend; }\n"
" server { listen 127.0.0.1:44450; return 502; }\n"
" map $ssl_preread_server_name $targetBackend {\n"
" default 10.1.0.1:8008;\n"
" on.example.com 10.1.0.1:8008;\n"
" off.example.com 127.0.0.1:44450; #off off.example.com\n"
" }\n"
"}"
"http {\n"
" upstream pool {\n"
" server 10.1.0.9:8080; #off 10.1.0.9:8080\n"
" server 10.1.0.10:8080;\n"
" }\n"
" server { listen 80; location / { proxy_pass http://pool; } }\n"
"}"
)
assert parsed["servers"][0]["proxy_passes"] == [
{"location": None, "scheme": None, "target": None, "variable": "targetBackend",
"disabled": False}]
assert parsed["servers"][1]["proxy_passes"] == [], "a `return` block proxies nowhere"
entries = parsed["maps"][0]["entries"]
assert [entry["key"] for entry in entries] == ["default", "on.example.com", "off.example.com"]
assert [entry["disabled"] for entry in entries] == [False, False, True]
assert entries[2]["target"] == {"host": "127.0.0.1", "port": 44450}, \
"the marked entry keeps its value: the state is the flag, not a rewritten line"
assert parsed["upstreams"] == [{"name": "pool", "servers": [
{"host": "10.1.0.9", "port": 8080, "disabled": True},
{"host": "10.1.0.10", "port": 8080, "disabled": False},
]}]
def test_the_nginx_section_exposes_the_marker_flag_on_the_wire(tmp_path: Path) -> None:
"""The section the dashboard reads is where the flag must land: a marked line in the file
is a `disabled: true` row in the `haproxy` and `nginx` sections, and an unmarked config
reports `false`, never a missing key (the dashboard's `enabled: false` is the flag's
absence of absence, and a row that could not answer the question would read as on)."""
write_file(tmp_path / "root", "opt/haproxy/etc/haproxy.cfg",
"backend be\n server on 10.1.1.9:443\n"
" server off 10.1.1.10:443 #off off\n")
write_file(tmp_path / "root", "opt/nginx/conf/nginx.conf",
"stream {\n"
" server {\n"
" listen 443;\n"
" proxy_pass $v; #off v\n"
" }\n"
" map $x $v {\n"
" default 10.1.0.1:8008;\n"
" a.test 10.1.0.1:8008;\n"
" }\n"
"}\n"
"http {\n"
" upstream p {\n"
" server 10.1.0.9:8080; #off 10.1.0.9:8080\n"
" }\n"
"}\n")
errors: list[str] = []
collector = host_agent.Collector(
str(tmp_path / "root"), haproxy_config="/opt/haproxy/etc/haproxy.cfg",
nginx_config="/opt/nginx/conf/nginx.conf")
haproxy = collector.haproxy(errors)
assert [row["disabled"] for row in haproxy["backends"][0]["servers"]] == [False, True]
nginx = collector.nginx(errors)
assert nginx["servers"][0]["proxy_passes"][0]["disabled"] is True
assert [row["disabled"] for row in nginx["upstreams"][0]["servers"]] == [True]
assert [row["disabled"] for row in nginx["maps"][0]["entries"]] == [False, False]
assert errors == []
# ---- the recorded gateway: the captures themselves (phase 28 task 04) -------------------- #
@@ -3732,3 +3883,404 @@ def test_the_target_list_is_the_two_sections_and_survives_a_section_that_is_not_
("vhost.test", 443), ("map.test", 443)]
assert host_agent.target_names(None, None) == []
assert host_agent.target_names({"backends": "nonsense"}, {"maps": [7]}) == []
# ---- POST /route (the one write, agent 1.4.0) -------------------------------- #
class _Peer:
"""A routectl stub: an AF_UNIX socket that counts dials, reads one request line and
answers one line. `hang` holds the connection open without answering, which is what
makes the agent's 30 s budget (patched down in the test) fire. A dial it never sees
is a refusal that happened before the socket, so the counters are the pins."""
def __init__(self, tmp_path: Path, reply: str | None = None, *, hang: bool = False) -> None:
self.path = str(tmp_path / "peer.sock")
self.reply = reply
self.hang = hang
self.requests: list[dict[str, Any]] = []
self.connections = 0
self._server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
self._server.bind(self.path)
self._server.listen(8)
def serve(self, window: float) -> None:
deadline = time.monotonic() + window
self._server.settimeout(0.05)
while time.monotonic() < deadline:
try:
conn, _addr = self._server.accept()
except TimeoutError:
continue
except OSError:
return # stop() closed the server; the window is over
self.connections += 1
data = b""
try:
while b"\n" not in data:
chunk = conn.recv(65536)
if not chunk:
break
data += chunk
except OSError:
pass
if data:
self.requests.append(json.loads(data.decode()))
if self.hang:
continue # never answer; the agent's own timeout fires
if self.reply is not None:
with suppress(OSError):
conn.sendall(self.reply.encode())
conn.close()
def stop(self) -> None:
self._server.close()
def _start_peer(tmp_path: Path, **kwargs: Any) -> _Peer:
peer = _Peer(tmp_path, **kwargs)
threading.Thread(target=peer.serve, args=(6.0,), daemon=True).start()
return peer
def _https_post(url: str, context: ssl.SSLContext, body: bytes) -> tuple[int, dict]:
request = urllib.request.Request(url, data=body, method="POST")
try:
with urllib.request.urlopen(request, timeout=15, context=context) as response:
return response.status, _json_or_raw(response.read())
except urllib.error.HTTPError as error:
return error.code, _json_or_raw(error.read())
def _post(url: str, body: bytes, headers: dict[str, str] | None = None) -> tuple[int, dict]:
request = urllib.request.Request(url, data=body, headers=headers or {}, method="POST")
try:
with urllib.request.urlopen(request, timeout=5) as response:
return response.status, json.loads(response.read())
except urllib.error.HTTPError as error:
return error.code, json.loads(error.read())
def _route_body() -> bytes:
return json.dumps({"id": "nginx:443->brain.test:443", "action": "disable",
"dry_run": True}).encode()
def test_the_agent_is_1_4_0_and_the_write_path_is_a_documented_flag() -> None:
"""The version moves with the write; the two flags are on the command line, not in
config, and the status page and the docstring say the write exists and is gated."""
assert host_agent.AGENT_VERSION == "1.4.0"
assert host_agent.ROUTE_TIMEOUT_SECONDS == 30
help_text = subprocess.run([sys.executable, str(AGENT_PATH), "--help"],
capture_output=True, text=True, timeout=30).stdout
for flag in ("--allow-writes", "--routectl-socket", "--once"):
assert flag in help_text, flag
assert "POST /route" in (host_agent.__doc__ or "")
def test_a_post_without_a_client_certificate_is_refused(
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path) -> None:
"""The token is not a write credential: a probe that holds only the CA (the same
identity `install-agent.sh --status` uses) gets the certificate refusal, on a POST too."""
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle))
status, body = _https_post(base + "/route", client_context(tls_fleet.root_cert), _route_body())
assert status == 403
assert body["detail"] == "client certificate required"
def test_a_revoked_serial_is_refused_on_the_write_path_too(
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path) -> None:
"""A serial on the denylist is refused on every path, including the write — the
revoke in `/certs` bites a `POST /route` no later than it bites a `GET`."""
text = tls_fleet.bundle + f"{host_agent.REVOKED_LINE}{tls_fleet.stolen.serial}\n"
base = tls_agent_server(load_bundle(tmp_path, text))
status, body = _https_post(base + "/route",
client_context(tls_fleet.root_cert, str(tls_fleet.stolen.cert),
str(tls_fleet.stolen.key)),
_route_body())
assert status == 403
assert body["detail"] == "client certificate revoked"
def test_token_mode_cannot_write_and_token_is_not_a_write_credential(
agent_server, tmp_path: Path) -> None:
"""A token-mode agent (no bundle) answers the write 403 even when the bearer token is
right: `Authorization` is a read credential, and the write's gate is the certificate.
The 405 arm is the same handler: a POST to any other path is refused before anything
else is checked."""
base, _port = agent_server("k8s-ops-token")
bearer = {"Authorization": "Bearer k8s-ops-token"}
status, body = _post(base + "/route", _route_body(), bearer)
assert status == 403
assert body["detail"] == "writes require a certificate"
for path in ("/health", "/snapshot", "/info", "/schema", "/"):
status, body = _post(base + path, b"{}", bearer)
assert status == 405, path
assert body["detail"] == "no such write path", path
@pytest.mark.parametrize("body, detail", [
('{"id": "x", "action": "disable", "dry_run": true, "extra": 1}',
"body carries a key the write path does not take"),
('{"id": "x", "action": "restart", "dry_run": true}',
"action must be disable or enable"),
(json.dumps({"id": "y" * 257, "action": "disable"}),
"id must be a string of at most 256 characters"),
("[1, 2]", "body is not a JSON object"),
("not json at all", "body is not a JSON object"),
])
def test_a_body_outside_the_vocabulary_is_refused_before_the_peer_is_touched(
body: str, detail: str, tls_fleet: SimpleNamespace, tls_agent_server,
tmp_path: Path) -> None:
"""The helper decides what a route id means; the agent decides what a body is. A body
outside `{id, action, dry_run}` is refused before any socket opens, so a bad body can
never reach a proxy config through this door."""
peer = _start_peer(tmp_path, reply='{"ok": true, "applied": true}')
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle))
host_agent.Handler.allow_writes = True
host_agent.Handler.routectl_socket = peer.path
status, answer = _https_post(base + "/route",
client_context(tls_fleet.root_cert, tls_fleet.client_cert,
tls_fleet.client_key),
body.encode())
assert status == 422
assert answer["detail"] == detail
peer.stop()
assert peer.connections == 0
def test_an_oversized_body_is_refused_before_the_peer_is_touched(
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path) -> None:
peer = _start_peer(tmp_path, reply='{"ok": true, "applied": true}')
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle))
host_agent.Handler.allow_writes = True
host_agent.Handler.routectl_socket = peer.path
oversized = json.dumps({"id": "x" * 5000, "action": "disable"})
assert len(oversized) > host_agent._ROUTE_MAX_BODY
status, answer = _https_post(base + "/route",
client_context(tls_fleet.root_cert, tls_fleet.client_cert,
tls_fleet.client_key),
oversized.encode())
assert status == 422
assert answer["detail"] == "body longer than the write path accepts"
peer.stop()
assert peer.connections == 0
def test_writes_off_refuses_before_the_peer_is_touched(
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path) -> None:
"""The certificate verifies and the helper is listening, but the agent was started
without `--allow-writes`: the refusal is the agent's own, and the dial never happens —
a host that is not armed must not be reachable by an armed dashboard."""
peer = _start_peer(tmp_path, reply='{"ok": true, "applied": true}')
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle))
# allow_writes stays False (the fixture's default)
host_agent.Handler.routectl_socket = peer.path
status, answer = _https_post(base + "/route",
client_context(tls_fleet.root_cert, tls_fleet.client_cert,
tls_fleet.client_key),
_route_body())
assert status == 403
assert answer["detail"] == "writes are not enabled on this agent"
peer.stop()
assert peer.connections == 0
def test_a_host_without_the_helper_answers_503_not_a_write(
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path) -> None:
"""`--routectl-socket` empty (the flag's documented arm): armed host, no helper unit.
The answer is 503, not a write and not a 200 — a dashboard can tell "not installed"
from "refused" without guessing."""
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle))
host_agent.Handler.allow_writes = True
# routectl_socket stays None (the fixture's default)
status, answer = _https_post(base + "/route",
client_context(tls_fleet.root_cert, tls_fleet.client_cert,
tls_fleet.client_key),
_route_body())
assert status == 503
assert answer["detail"] == "no route helper on this host"
@pytest.mark.parametrize("action, dry_run", [("disable", True), ("enable", False)])
def test_a_good_request_reaches_the_peer_once_and_the_answer_is_passed_through(
action: str, dry_run: bool, tls_fleet: SimpleNamespace, tls_agent_server,
tmp_path: Path) -> None:
"""The agent is a forward: the body that reaches the helper is exactly `{id, action,
dry_run}` — no header, no token, no invented fields — and the helper's answer (a
refusal here, `applied` stays false) is the response, verbatim. One dial, one request:
a retry would be a second reload."""
peer = _start_peer(tmp_path, reply='{"ok": false, "detail": "ambiguous: whatever"}')
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle))
host_agent.Handler.allow_writes = True
host_agent.Handler.routectl_socket = peer.path
payload = {"id": "nginx:443->brain.test:443", "action": action, "dry_run": dry_run}
status, answer = _https_post(base + "/route",
client_context(tls_fleet.root_cert, tls_fleet.client_cert,
tls_fleet.client_key),
json.dumps(payload).encode())
assert status == 200
assert answer == {"ok": False, "detail": "ambiguous: whatever"}
assert peer.connections == 1
assert peer.requests == [payload]
peer.stop()
def test_the_peer_is_called_once_even_when_it_times_out(
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path,
monkeypatch: pytest.MonkeyPatch) -> None:
"""The helper hangs: the agent's own 30 s budget (patched to 0.3 s here) fires and the
answer is 502 with the reason, never a hang and never a second dial — a retry would be
a second reload of the fleet's only ingress."""
peer = _start_peer(tmp_path, hang=True)
monkeypatch.setattr(host_agent, "ROUTE_TIMEOUT_SECONDS", 0.3)
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle))
host_agent.Handler.allow_writes = True
host_agent.Handler.routectl_socket = peer.path
status, answer = _https_post(base + "/route",
client_context(tls_fleet.root_cert, tls_fleet.client_cert,
tls_fleet.client_key),
_route_body())
assert status == 502
assert answer["detail"] == "route helper timed out"
assert peer.connections == 1
assert len(peer.requests) == 1
peer.stop()
def test_an_unreadable_peer_answer_is_502_and_the_peer_is_not_retried(
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path) -> None:
"""A helper that answers bytes that are not a JSON object is a broken helper: 502 with
the reason, one dial. The agent invents no success and no detail of its own."""
peer = _start_peer(tmp_path, reply="nonsense is not json")
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle))
host_agent.Handler.allow_writes = True
host_agent.Handler.routectl_socket = peer.path
status, answer = _https_post(base + "/route",
client_context(tls_fleet.root_cert, tls_fleet.client_cert,
tls_fleet.client_key),
_route_body())
assert status == 502
assert answer["detail"] == "route helper answered something unreadable"
assert peer.connections == 1
peer.stop()
def test_every_other_path_is_405_on_the_write_path_and_a_query_string_is_not_a_second_path(
tls_fleet: SimpleNamespace, tls_agent_server, tmp_path: Path) -> None:
"""The write path is `/route` and nothing else, on a TLS host with writes on: every
other path is 405 before the certificate, the flag or the body are even looked at. A
query string on `/route` is the same path — the helper decides what the id means."""
peer = _start_peer(tmp_path, reply='{"ok": true, "applied": true}')
base = tls_agent_server(load_bundle(tmp_path, tls_fleet.bundle))
host_agent.Handler.allow_writes = True
host_agent.Handler.routectl_socket = peer.path
context = client_context(tls_fleet.root_cert, tls_fleet.client_cert, tls_fleet.client_key)
for path in ("/health", "/snapshot", "/info", "/schema", "/", "/nope"):
status, answer = _https_post(base + path, context, b"{}")
assert status == 405, path
assert answer["detail"] == "no such write path", path
status, answer = _https_post(base + "/route?from=drill", context, _route_body())
assert status == 200
assert peer.connections == 1
peer.stop()
# ---- the `disabled` field (the read side of the switch) ---------------------- #
def _haproxy_server(snapshot: dict[str, Any], name: str) -> dict[str, Any] | None:
for backend in snapshot["haproxy"]["backends"]:
for server in backend["servers"]:
if server["name"] == name:
return server
return None
def _nginx_proxy_pass(snapshot: dict[str, Any], host: str, port: int) -> dict[str, Any] | None:
for server in snapshot["nginx"]["servers"]:
for entry in server["proxy_passes"]:
target = entry.get("target")
if target is not None and target["host"] == host and target["port"] == port:
return entry
return None
def _nginx_map_entry(snapshot: dict[str, Any], key: str) -> dict[str, Any] | None:
for mapped in snapshot["nginx"]["maps"]:
for entry in mapped["entries"]:
if entry["key"] == key:
return entry
return None
def test_a_marked_haproxy_server_is_disabled_and_stays_in_the_document(gateway_snapshot) -> None:
"""`#off <name>` is a trailing comment, not a deletion: the marked line stays in the
config (the enable half needs it) and the agent says `disabled: true` on the node.
The unmarked neighbours keep `disabled: false` — the field is the marker, nothing else."""
snap = gateway_snapshot("marked")
gitea = _haproxy_server(snap, "gitea")
assert gitea is not None, "the marked line must stay in the document"
assert gitea["disabled"] is True
assert gitea["target"] == "reeseapps-gitea.reeselink.com" and gitea["port"] == 2222
others = [server for backend in snap["haproxy"]["backends"]
for server in backend["servers"] if server["name"] != "gitea"]
assert others, "the fixture's other servers must still be parsed"
assert all(server["disabled"] is False for server in others)
def test_an_unmarked_config_has_disabled_false_everywhere(gateway_snapshot) -> None:
snap = gateway_snapshot("full")
for backend in snap["haproxy"]["backends"]:
for server in backend["servers"]:
assert server["disabled"] is False, server["name"]
for server in snap["nginx"]["servers"]:
for entry in server["proxy_passes"]:
assert entry["disabled"] is False
for pool in snap["nginx"]["upstreams"]:
for member in pool["servers"]:
assert member["disabled"] is False
for mapped in snap["nginx"]["maps"]:
for entry in mapped["entries"]:
assert entry["disabled"] is False
def test_a_marked_vhost_proxy_pass_is_disabled(gateway_snapshot) -> None:
snap = gateway_snapshot("marked")
entry = _nginx_proxy_pass(snap, "nextcloud.reeselink.com", 11000)
assert entry is not None, "the marked proxy_pass must stay in the document"
assert entry["disabled"] is True
plain = gateway_snapshot("full")
plain_entry = _nginx_proxy_pass(plain, "nextcloud.reeselink.com", 11000)
assert plain_entry is not None
assert plain_entry["disabled"] is False
def test_a_marked_map_entry_is_disabled_and_its_value_is_the_sink(gateway_snapshot) -> None:
"""The helper's third arm: the map entry's value is rewritten to the sink address and
the line marked. `disabled` is the marker (true even though the target is a loopback),
and the unmarked capture keeps the real target with `disabled: false`."""
snap = gateway_snapshot("marked")
entry = _nginx_map_entry(snap, "brain.reeseapps.com")
assert entry is not None
assert entry["disabled"] is True
assert entry["target"] == {"host": "127.0.0.1", "port": 44450}
plain = gateway_snapshot("full")
plain_entry = _nginx_map_entry(plain, "brain.reeseapps.com")
assert plain_entry is not None
target = plain_entry["target"]
assert target is not None and target["host"] == "reeseapps-brain-of-reese.reeselink.com"
assert target["port"] == 443
def test_a_hand_edit_is_an_absence_not_a_marker(gateway_snapshot) -> None:
"""`hand_edited` is the operator's other half of the switch: the haproxy line is gone
(the route is off, nothing to mark), the vhost line is plain (on, `disabled: false`),
and the map value is deleted (the entry is there, its target is null, not marked)."""
snap = gateway_snapshot("hand_edited")
assert _haproxy_server(snap, "gitea") is None, "the deleted line is gone"
entry = _nginx_proxy_pass(snap, "nextcloud.reeselink.com", 11000)
assert entry is not None
assert entry["disabled"] is False
mapped = _nginx_map_entry(snap, "brain.reeseapps.com")
assert mapped is not None
assert mapped["target"] is None
assert mapped["disabled"] is False