phase: 33_gateway_route_switching
Phase 33 completion pass is done. Final report:
## Phase 33 — completion report
**All 8 task files complete** (`.agents/phases/complete/33_gateway_route_switching/`), full gate green, docs verified.
### Work in this pass
1. **`tests/test_routectl.py` added** (17 tests, all green) — exercises the *real* helper binary unprivileged against a fake tree (`--root`), the real agent parser, the captured gateway fixtures, and stubbed `nginx`/`haproxy`/`systemctl` on `PATH`. Pins the full protocol (locate → state → dry-run → temp-file/fsync/`os.replace` write → checker → restore on checker failure → reload-never-restart → re-read proof → sidecar record), all three route shapes (haproxy `server`, nginx `proxy_pass`, nginx stream `map` via its sink-value id), `drift` / unknown-id / `map-default` refusals, `list`/`verify`, and the `--serve` peer gate (`SO_PEERCRED` uid check, one JSON line per request, exact body shape).
2. **Real helper defect fixed** — `deploy/dashboards-routectl.py`: `main()`'s early-return paths returned `({"ok": False, …}, EXIT_*)` tuples, so `sys.exit()` printed a Python repr instead of a JSON line (usage errors, agent-import refusal). Now prints `json.dumps(…)` and returns only the code, keeping the one-JSON-line contract.
3. **Documentation consistency fixes** (task 08 criterion 6):
- `README.md`: garbled verdict-colour sentence on the switch ("green and to the position") → "in the **on** position and tinted `--gate-on`" (the existence-pair contract).
- `README.md`: audit sentence now names all three event kinds — `route-disabled` / `route-enabled` / **`route-denied`** (the code stores one per refused call; dry runs store none).
- `page-gateway.js`: replaced the stale pre-33 module docstring ("the switch is not wired") with the two-arm `GATEWAY_ALLOW_WRITES` contract.
- `AGENTS.md`: every phase-33 claim re-verified against code (the ten gates, the 35 s/30 s budget pair, `gateway.writes_enabled`, `/api/summary`'s `disabled`, the 1.4.0 agent bullet, the gap list) — no changes needed.
### Verification
- `uv run pytest --cov`: **1970 passed**, 18 skipped — coverage **98.74%** (floor 94%)
- `ruff` clean · `pyright` 0 errors · `bash .agents/validate.sh` **exit 0** (incl. `node --check` on every page script)
- Dev server running; all 10 sources `ok`; bundles exported
### Live infrastructure
**No live changes made.** The live drill remains unexecuted per `data/logs/NOTES-33-switching.md` (no operator approval; checker binaries unavailable in the podman proxy units). All routes stay enabled, config hashes unchanged, and the sandbox rehearsal + backup tree are in place for the operator's own `DANGER` call. All changes are in the working tree only.
This commit is contained in:
@@ -33,6 +33,7 @@ from dashboards.collectors import (
|
||||
fetch_snapshot,
|
||||
gateway_job,
|
||||
host_jobs,
|
||||
post_route,
|
||||
)
|
||||
from dashboards.config import Settings, Target
|
||||
from dashboards.poller import Result
|
||||
@@ -891,6 +892,206 @@ async def test_a_token_mode_snapshot_gains_no_tls_fields() -> None:
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# post_route (the app's second upstream write, phase 33 task 05)
|
||||
# --------------------------------------------------------------------------- #
|
||||
def recorded_post(handler: Any) -> tuple[httpx.AsyncClient, list[httpx.Request]]:
|
||||
"""The `post_route` shape of `client_for`: a scripted answer, every request kept."""
|
||||
seen: list[httpx.Request] = []
|
||||
|
||||
def recording(request: httpx.Request) -> httpx.Response:
|
||||
seen.append(request)
|
||||
return handler(request)
|
||||
|
||||
return httpx.AsyncClient(transport=httpx.MockTransport(recording)), seen
|
||||
|
||||
|
||||
async def test_post_route_posts_the_wire_body_and_returns_the_helpers_object_verbatim() -> None:
|
||||
"""The wire is task 01's `--serve` contract: `id`, `action`, `dry_run`, and nothing else.
|
||||
|
||||
`danger` is the dashboard's typed confirmation, not a wire key — the agent's writer takes a
|
||||
body with no `danger`, and a header or field smuggled in is a contract break the agent would
|
||||
answer with its own 422. The answer comes back untouched: an `applied: false` must not
|
||||
become a success on the way over (rule 2), so the object is handed over, not rebuilt.
|
||||
"""
|
||||
client, seen = recorded_post(lambda request: httpx.Response(
|
||||
200, json={"applied": True, "changed": True, "reloaded": True}))
|
||||
async with client:
|
||||
answer = await post_route(client, Target("gateway", "http://agent.test"),
|
||||
{"id": "brain.reeseapps.com", "action": "disable",
|
||||
"dry_run": False},
|
||||
timeout=35.0)
|
||||
assert answer == {"applied": True, "changed": True, "reloaded": True}
|
||||
assert len(seen) == 1
|
||||
request = seen[0]
|
||||
assert request.method == "POST" and request.url.path == "/route"
|
||||
assert json.loads(request.content) == {"id": "brain.reeseapps.com",
|
||||
"action": "disable", "dry_run": False}
|
||||
assert "authorization" not in {key.lower() for key in request.headers}, (
|
||||
"the plain path carries only the target's own headers")
|
||||
|
||||
|
||||
async def test_post_route_sends_the_token_on_the_token_path() -> None:
|
||||
"""A host nobody has bundled dials the compatibility way: the header, exactly as the read sends it.
|
||||
|
||||
`parse_targets` is what moves the userinfo token into the header; a directly built target
|
||||
hands `post_route` the same shape, so the pin is the header arriving on the wire.
|
||||
"""
|
||||
client, seen = recorded_post(lambda request: httpx.Response(
|
||||
200, json={"applied": False, "changed": False}))
|
||||
target = Target("gateway", "http://agent.test",
|
||||
headers={"Authorization": "Bearer SEKRIT-AGENT-TOKEN"})
|
||||
async with client:
|
||||
await post_route(client, target, {"id": "r", "action": "enable", "dry_run": True},
|
||||
timeout=35.0)
|
||||
assert seen[0].headers["Authorization"] == "Bearer SEKRIT-AGENT-TOKEN"
|
||||
|
||||
|
||||
async def test_post_route_sends_no_authorization_on_the_certificate_path() -> None:
|
||||
"""The bundle is the credential on a `https://` target, and the write is stricter than the read.
|
||||
|
||||
`parse_targets` has already moved any token into the header, so a bundled host's target has a
|
||||
clean URL and no header — and the write must add nothing of its own: a header the read would
|
||||
have sent (a compatibility host's token) is a second opinion the write refuses, so the
|
||||
certificate path sends the body and the certificate and nothing else.
|
||||
"""
|
||||
seen: list[list[tuple[bytes, bytes]]] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
seen.append(request.headers.raw)
|
||||
return httpx.Response(200, json={"applied": False, "dry_run": True, "changed": True})
|
||||
|
||||
tls = AgentTls(verify="/unused/root.pem", cert=("/unused/cert.pem", "/unused/key.pem"),
|
||||
status=lambda: certs.ACTIVE, transport=httpx.MockTransport(handler))
|
||||
async with client_for(lambda request: httpx.Response(200, json={})) as shared:
|
||||
answer = await post_route(shared, Target("gateway", "https://agent.test"),
|
||||
{"id": "r", "action": "disable", "dry_run": True},
|
||||
timeout=35.0, tls=tls)
|
||||
assert answer["dry_run"] is True
|
||||
assert len(seen) == 1
|
||||
assert not any(key == b"authorization" for key, _ in seen[0]), (
|
||||
"a token next to a certificate is the second opinion the write refuses")
|
||||
|
||||
|
||||
class PostTimeoutRecorder(TimeoutRecordingClient):
|
||||
"""The same seam as `TimeoutRecordingClient`, for the write's own budget argument."""
|
||||
|
||||
async def post(self, url: Any, **kwargs: Any) -> httpx.Response: # type: ignore[override]
|
||||
self.timeouts.append(kwargs.get("timeout", "<unset>"))
|
||||
return await super().post(url, **kwargs)
|
||||
|
||||
|
||||
async def test_post_route_carries_the_write_budget_and_never_a_bare_none() -> None:
|
||||
"""`GATEWAY_WRITE_TIMEOUT_SECONDS` is the write's own 35 s, passed per request.
|
||||
|
||||
A `None` timeout is *no timeout at all* to httpx: a helper stuck on a wedged `systemctl
|
||||
reload` would hold a request open for good. The sentinel keeps the client default, a number
|
||||
keeps the write's own budget.
|
||||
"""
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
return httpx.Response(200, json={"applied": False, "changed": False})
|
||||
|
||||
client = PostTimeoutRecorder(transport=httpx.MockTransport(handler),
|
||||
timeout=10.0, base_url="http://agent.test")
|
||||
async with client:
|
||||
await post_route(client, Target("gw", "http://agent.test"),
|
||||
{"id": "r", "action": "disable", "dry_run": False}, timeout=35.0)
|
||||
await post_route(client, Target("gw", "http://agent.test"),
|
||||
{"id": "r", "action": "disable", "dry_run": False})
|
||||
assert client.timeouts[-2:] == [35.0, httpx.USE_CLIENT_DEFAULT]
|
||||
|
||||
|
||||
async def test_post_route_timeout_is_a_504_that_names_the_redacted_url() -> None:
|
||||
"""The message lands in the event table and the browser reads it (invariant 6): the URL is
|
||||
the redacted one and the status is the 504 a timeout is, so a slow helper is not a 502.
|
||||
"""
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
raise httpx.ReadTimeout("read timed out")
|
||||
|
||||
target = Target("gw", "http://SEKRIT-AGENT-TOKEN@agent.test")
|
||||
async with client_for(handler) as client:
|
||||
with pytest.raises(UpstreamError) as caught:
|
||||
await post_route(client, target, {"id": "r", "action": "disable", "dry_run": False})
|
||||
assert caught.value.status == 504
|
||||
assert "SEKRIT-AGENT-TOKEN" not in caught.value.message
|
||||
assert "agent.test/route" in caught.value.message
|
||||
assert "timed out" in caught.value.message
|
||||
|
||||
|
||||
async def test_post_route_unreachable_is_a_502_with_the_redacted_url() -> None:
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
raise httpx.ConnectError("no route to host")
|
||||
|
||||
target = Target("gw", "https://SEKRIT-AGENT-TOKEN@agent.test")
|
||||
tls = AgentTls(verify="/unused/root.pem", cert=("/unused/cert.pem", "/unused/key.pem"),
|
||||
status=lambda: certs.ACTIVE, transport=httpx.MockTransport(handler))
|
||||
async with client_for(lambda request: httpx.Response(200, json={})) as client:
|
||||
with pytest.raises(UpstreamError) as caught:
|
||||
await post_route(client, target, {"id": "r", "action": "enable", "dry_run": False},
|
||||
tls=tls)
|
||||
assert caught.value.status == 502
|
||||
assert "SEKRIT-AGENT-TOKEN" not in caught.value.message
|
||||
|
||||
|
||||
async def test_post_route_401_on_the_token_path_is_about_the_token() -> None:
|
||||
"""A 401 the token path gets is the rotated-token case: the sentence says so, and the URL
|
||||
is still the redacted one."""
|
||||
async with client_for(lambda request: httpx.Response(401, json={"detail": "bad token"})) as client:
|
||||
with pytest.raises(UpstreamError) as caught:
|
||||
await post_route(client, Target("gw", "http://SEKRIT-AGENT-TOKEN@agent.test"),
|
||||
{"id": "r", "action": "disable", "dry_run": False})
|
||||
assert "rejected the token (401)" in caught.value.message
|
||||
assert "SEKRIT-AGENT-TOKEN" not in caught.value.message
|
||||
|
||||
|
||||
async def test_post_route_refusal_carries_the_helpers_own_word() -> None:
|
||||
"""A `>= 400` answer is the agent's own refusal — a `403` here is the write gate's own word
|
||||
(`writes are not enabled on this agent`), and paraphrasing it would be the dashboard
|
||||
claiming to know more than the peer said."""
|
||||
async with client_for(lambda request: httpx.Response(
|
||||
409, json={"detail": "no such id: brain.reeseapps.com"})) as client:
|
||||
with pytest.raises(UpstreamError) as caught:
|
||||
await post_route(client, Target("gateway", "http://agent.test"),
|
||||
{"id": "brain.reeseapps.com", "action": "disable", "dry_run": False})
|
||||
assert caught.value.status == 502
|
||||
assert "HTTP 409" in caught.value.message
|
||||
assert "no such id: brain.reeseapps.com" in caught.value.message
|
||||
|
||||
|
||||
async def test_post_route_an_unreadable_answer_is_an_error_with_a_bounded_slice() -> None:
|
||||
async with client_for(lambda request: httpx.Response(
|
||||
200, text="not json at all " + "x" * 200)) as client:
|
||||
with pytest.raises(UpstreamError, match="unreadable") as caught:
|
||||
await post_route(client, Target("gateway", "http://agent.test"),
|
||||
{"id": "r", "action": "disable", "dry_run": False})
|
||||
assert "x" * 200 not in caught.value.message, "the slice is bounded, not the whole body"
|
||||
async with client_for(lambda request: httpx.Response(200, json=[1, 2])) as client:
|
||||
with pytest.raises(UpstreamError, match="non-object"):
|
||||
await post_route(client, Target("gateway", "http://agent.test"),
|
||||
{"id": "r", "action": "disable", "dry_run": False})
|
||||
|
||||
|
||||
async def test_post_route_refuses_a_revoked_agent_before_any_dial() -> None:
|
||||
"""The revocation gate is in front of the network, for the write as for the read.
|
||||
|
||||
The scripted transport records what it receives, and a `status` callable that answers
|
||||
`REVOKED` is what `certs.agent_status` would answer after the operator's click: the
|
||||
refusal must come before the transport sees a request, not after.
|
||||
"""
|
||||
seen: list[httpx.Request] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
seen.append(request)
|
||||
return httpx.Response(200, json={"applied": True})
|
||||
|
||||
tls = AgentTls(verify="/unused/root.pem", cert=("/unused/cert.pem", "/unused/key.pem"),
|
||||
status=lambda: certs.REVOKED, transport=httpx.MockTransport(handler))
|
||||
async with client_for(lambda request: httpx.Response(200, json={})) as client:
|
||||
with pytest.raises(UpstreamError, match="agent certificate revoked") as caught:
|
||||
await post_route(client, Target("gateway", "https://agent.test"),
|
||||
{"id": "r", "action": "disable", "dry_run": False}, tls=tls)
|
||||
assert seen == [], "a revoked host must not be written"
|
||||
assert "revoked" in caught.value.message
|
||||
|
||||
# LLM engines
|
||||
# --------------------------------------------------------------------------- #
|
||||
GUFO = (FIXTURES / "metrics_gufo.txt").read_text()
|
||||
|
||||
Reference in New Issue
Block a user