services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 15_agent_mtls

Browse Source 
All green. Report:

**Verified / implemented (final pass of `15_agent_mtls`)**
- Wrote `data/logs/verify-15-tls-e2e.py`: a real `host_agent.py --cert-bundle` + a real dashboard process over a scratch DB (`HOST_TARGETS=loop=https://…`) — **27/27 PASS**, transcript `verify-15-tls-e2e.out`, notes `data/logs/NOTES-tls-rollout.md`.
- Read-only checks against the running dev store: 5 `certs.*` rows byte-identical across a restart; served bundle == the `meta` row (1430 B); 0 events containing `-----BEGIN`; unsigned `/certs`→`303 ?next=`, unsigned `/api/certs`→`401`, signed-in `200`; screenshots `data/logs/15-final-{certs,system}.png`.
- Docs (step 6, full review): README — added *Protecting an agent, host by host* (the rollout task-07 left out), a `tests/test_certs.py`/TLS-matrix/sandbox-installer paragraph, and fixed two stale “the one write endpoint in the app” claims (`NUT_ALLOW_ACTIONS`, Battery-test buttons). AGENTS.md — test count 1424→1427, bundle-fetch command in *Commands*, a Testing-policy bullet on structurally pinning a key-holding module, fleet transport bullet (token mode is now the compatibility path), gap 11 final state.

**Tests / lint (full suite 100% green)**
- `uv run pytest` → **1427 passed, 15 skipped**, exit 0 · `bash .agents/validate.sh` → **exit 0** (coverage **98.94%** vs 94% floor, `certs.py` 100%; ruff “All checks passed”; pyright 0 errors; `node --check` clean; hygiene clean)
- `uv run pytest -m live --live` → **11 passed / 0 failed** · `./scripts/dev-server.sh status` → all 9 sources `ok`
- `uv pip list --outdated` → empty after bumping `fastapi 0.152.0 → 0.153.0` (`uv.lock` only); full suite re-run green.

**Completion criteria**
- Fresh-DB root CA, idempotent: ✅ (tests + live store hashes) · agent TLS matrix incl. wrong-CA/certless 403/`/health` CA-only/revoked 403 + live denylist re-read: ✅
- `https` `host:`/`disks:` `ok` + revoked says “agent certificate revoked” with no dial: ✅ on loopback (real agent + real dashboard) and spy-pinned hermetically; **not** on the fleet (below)
- Signed-in `/certs` + rotate/revoke/root-rotate + byte-identical bundle download: ✅ · installer bundle/unit/`--status` over TLS: ✅ (0640, see deviation)
- `pytest`/`validate.sh`/`--outdated`: ✅ · token-mode byte-identical, engine/UPS untouched, public surface unchanged (route walk): ✅
- **Live fleet TLS (deskwork+gateway flip): NOT done** — no live changes. It needs root on hosts without the installer grant (`deskwork`), has no both-ways-working window (`http://` dashboard vs https-only agent), and would deploy the agent from this not-yet-committed tree. Exact procedure recorded in README + `data/logs/NOTES-tls-rollout.md`.

**Deviations**: bundle mode `0640 root:dashboards-agent` instead of the task file’s `0600` (the agent re-reads its own bundle for the denylist; 0600 breaks the unit — already recorded in AGENTS.md); the loopback dashboard substitutes for the fleet flip. Incidental: a stray `pkill` of mine killed the dev server mid-pass; restarted and verified healthy.

**Next pending phase**: `.agents/phases/todo/16_agent_smart_pass_budget` (then `17_installer_device_filter`).
This commit is contained in:
ducoterra
2026-10-20 10:02:16 -04:00
parent 569ca25fdb
commit cffbac5771
72 changed files with 7168 additions and 216 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_collectors.py
+406
View File
@@ -10,21 +10,30 @@ transitions); the socket protocol itself is in test_nut.py.
from __future__ import annotations
import json
import threading
from http.server import ThreadingHTTPServer
from pathlib import Path
from types import SimpleNamespace
from typing import Any, ClassVar
import httpx
import pytest
from dashboards import certs
from dashboards.collectors import (
AgentTls,
NutMonitor,
UpstreamError,
agent_tls_map,
disks_jobs,
fetch_disk_snapshot,
fetch_engine,
fetch_host_snapshot,
fetch_snapshot,
host_jobs,
)
from dashboards.config import Target
from dashboards.poller import Result
from dashboards.rates import RateEstimator
from dashboards.sources import nut
from dashboards.sources.disks import DisksCollector
@@ -351,6 +360,403 @@ async def test_smart_addon_errors_reach_the_disk_payload_verbatim(
assert "degraded" not in system.payload, "and /system stays unbothered by it"
# --------------------------------------------------------------------------- #
# TLS: the dashboard dials a protected exporter with its own client certificate
# --------------------------------------------------------------------------- #
#
# The certificates come from `dashboards.certs` over a dict store and the server is the real
# `agent/host_agent.py` in the TLS mode task 02 added, so this proves the dial itself: the
# client certificate is presented, the agent is verified against the root this dashboard minted,
# and a host the operator revoked is refused before a socket opens. `httpx.MockTransport` is
# kept for the sentences (a 403 body is the agent's words, not a handshake).
class DictMeta:
"""The three-method `certs.MetaStore` slice, backed by a dict (the `test_certs.py` idiom)."""
def __init__(self) -> None:
self.rows: dict[str, str] = {}
def get_meta(self, key: str) -> str | None:
return self.rows.get(key)
def set_meta(self, key: str, value: str) -> None:
self.rows[key] = value
def delete_meta(self, key: str) -> int:
return 1 if self.rows.pop(key, None) is not None else 0
@pytest.fixture
def tls_fleet(tmp_path: Path, host_agent_module: Any) -> Any:
"""One dashboard CA + client pair, one issued agent bundle, and the agent serving it.
Everything production has: the material written to a private directory (`httpx` takes
paths), the bundle installed where `install-agent.sh` will put it, and the `AgentTls` the
jobs get — built by `agent_tls_map`, the same factory `app.build_hub` is handed, so the
tests dial with the production wiring and not a hand-built guess of it.
"""
store = DictMeta()
state = certs.ensure_certs(store)
state = certs.issue_agent_cert(store, state, "localhost", ["127.0.0.1", "localhost"])
entry = state.agent("localhost")
assert entry is not None
material = certs.write_material(state, tmp_path)
bundle_path = tmp_path / "dashboards-agent.certs"
bundle_path.write_text(entry.bundle, encoding="ascii")
bundle, error = host_agent_module.TlsBundle.load(str(bundle_path))
assert error is None and bundle is not None
host_agent_module.Handler.collector = host_agent_module.Collector("/", smart=False)
host_agent_module.Handler.token = None
host_agent_module.Handler.bundle = bundle
httpd = ThreadingHTTPServer(("127.0.0.1", 0), host_agent_module.Handler)
httpd.socket = bundle.context.wrap_socket(httpd.socket, server_side=True)
thread = threading.Thread(target=httpd.serve_forever, daemon=True)
thread.start()
base = f"https://127.0.0.1:{httpd.server_address[1]}"
target = Target("localhost", base)
try:
yield SimpleNamespace(
store=store,
state=state,
material=material,
bundle=bundle,
bundle_rev=entry.bundle_rev,
base=base,
target=target,
tls=agent_tls_map(material, store, [target])["localhost"],
)
finally:
httpd.shutdown()
httpd.server_close()
# `Handler` is process-wide state: a later test must not inherit this bundle.
host_agent_module.Handler.bundle = None
host_agent_module.Handler.token = None
certs.remove_material(material)
async def test_an_https_agent_answers_the_dashboard_that_presents_its_certificate(
tls_fleet: SimpleNamespace) -> None:
"""The whole point of the phase, collector side: our cert opens the agent.
The two fields the agent adds in TLS mode come back too, because task 05's page compares the
revision `/certs` issued with the revision the host is actually running.
"""
async with httpx.AsyncClient(timeout=15.0) as client:
body = await fetch_snapshot(client, tls_fleet.target, tls=tls_fleet.tls)
assert body["ok"] is True
assert body["tls"] is True
assert body["bundle_rev"] == tls_fleet.bundle_rev
result = await fetch_host_snapshot(client, tls_fleet.target,
HostCollector(RateEstimator()), tls=tls_fleet.tls)
assert result.payload["agent"]["tls"] is True
assert result.payload["agent"]["bundle_rev"] == tls_fleet.bundle_rev
assert result.payload["memory"]["total_bytes"] > 0, "the snapshot is a real one"
async def test_an_agent_signed_by_another_authority_is_not_trusted(
tls_fleet: SimpleNamespace, tmp_path: Path) -> None:
"""A root that is not ours verifies nothing here, so the dial fails instead of accepting.
Trusting any certificate would make the whole phase a no-op with extra steps, and the
failure has to reach the poller as an `UpstreamError` naming the host rather than as a
traceback out of `ssl`.
"""
stranger_base = tmp_path / "stranger"
stranger_base.mkdir()
stranger = certs.write_material(certs.ensure_certs(DictMeta()), stranger_base)
foreign = agent_tls_map(stranger, tls_fleet.store, [tls_fleet.target])["localhost"]
async with httpx.AsyncClient(timeout=15.0) as client:
with pytest.raises(UpstreamError) as caught:
await fetch_snapshot(client, tls_fleet.target, tls=foreign)
assert "unreachable" in caught.value.message
assert tls_fleet.base in caught.value.message
certs.remove_material(stranger)
async def test_dashboard_material_that_cannot_be_read_fails_the_poll(
tls_fleet: SimpleNamespace, tmp_path: Path) -> None:
"""A missing client-cert file is a red source with a sentence, not a dead poll loop.
`httpx` loads the CA and the client chain when the client is *built*, so this arrives as a
`FileNotFoundError` from inside the fetch. A TLS dial that can only fail with a non-HTTP
exception is the invariant-4 failure mode: the task would die and the source would look
healthy while its charts stopped.
"""
broken = AgentTls(verify=str(tls_fleet.material.ca),
cert=(str(tmp_path / "gone-cert.pem"), str(tmp_path / "gone-key.pem")),
status=lambda: certs.ACTIVE)
async with httpx.AsyncClient(timeout=15.0) as client:
with pytest.raises(UpstreamError, match="unreachable") as caught:
await fetch_snapshot(client, tls_fleet.target, tls=broken)
assert "FileNotFoundError" in caught.value.message
assert tls_fleet.base in caught.value.message, "the failure still names the host's URL"
async def test_an_https_agent_is_not_readable_without_the_material(tls_fleet: SimpleNamespace) -> None:
"""A shared client with no client certificate gets nowhere — "only this dashboard can read
an agent" has to hold for the collector's own client too, not just for a stranger's.
"""
async with httpx.AsyncClient(timeout=15.0) as client:
with pytest.raises(UpstreamError) as caught:
await fetch_snapshot(client, tls_fleet.target)
assert "unreachable" in caught.value.message
async def test_a_revoked_agent_is_refused_before_anything_is_dialled(
tls_fleet: SimpleNamespace, monkeypatch: pytest.MonkeyPatch) -> None:
"""The revocation gate is in front of the network, not behind it.
The spy replaces `httpx.AsyncClient.get` on the class, so it catches the shared client and
the per-dial TLS client alike: what is being proved is that no request is built at all, not
that one particular client was unused. A dial-then-refuse would still put a socket on a host
the operator just struck out.
"""
calls: list[str] = []
async def spy_get(self: Any, url: Any, **kwargs: Any) -> Any:
calls.append(str(url))
raise AssertionError(f"a revoked agent was dialled at {url}")
monkeypatch.setattr(httpx.AsyncClient, "get", spy_get)
certs.revoke(tls_fleet.store, tls_fleet.state, "localhost")
async with httpx.AsyncClient(timeout=15.0) as client:
with pytest.raises(UpstreamError, match="agent certificate revoked") as caught:
await fetch_snapshot(client, tls_fleet.target, tls=tls_fleet.tls)
with pytest.raises(UpstreamError, match="agent certificate revoked"):
await fetch_disk_snapshot(client, tls_fleet.target, DisksCollector(),
tls=tls_fleet.tls)
assert calls == [], "a revoked host must not be dialled on either cadence"
assert "revoked" in caught.value.message
async def test_the_revocation_answer_is_a_live_read_of_the_store(tls_fleet: SimpleNamespace) -> None:
"""`status` is a callable because the operator clicks revoke while this process is running.
A status string read at startup would keep dialling a struck-out host until the next
restart, and a dashboard that needs a restart to enforce a security decision does not
enforce it.
"""
tls = tls_fleet.tls
assert tls.status() == certs.ACTIVE
assert tls.revoked is False
certs.revoke(tls_fleet.store, tls_fleet.state, "localhost")
assert tls.status() == certs.REVOKED
assert tls.revoked is True, "the same object, with no rebuild and no restart"
def test_the_revocation_is_per_host_and_the_material_is_shared(tls_fleet: SimpleNamespace) -> None:
"""One identity for every dial, one revocation answer per name.
`certs.py` gives the dashboard a single client certificate, so `verify`/`cert` are the same
paths for every host; what differs is whose identity the operator struck out. A name with no
entry is active rather than refused — never-issued is not the same fact as revoked, and
answering revoked for it would take a working host red with no operator action behind it
(invariant 3).
"""
targets = [tls_fleet.target, Target("deskwork", "https://10.1.0.51:9120")]
mapping = agent_tls_map(tls_fleet.material, tls_fleet.store, targets)
assert mapping["localhost"].verify == str(tls_fleet.material.ca)
assert mapping["localhost"].cert == mapping["deskwork"].cert == (
str(tls_fleet.material.client_cert), str(tls_fleet.material.client_key))
certs.revoke(tls_fleet.store, tls_fleet.state, "localhost")
assert mapping["localhost"].status() == certs.REVOKED
assert mapping["deskwork"].status() == certs.ACTIVE
assert agent_tls_map(None, tls_fleet.store, targets) == {}, (
"no material means no TLS dial, exactly the pre-phase-15 path")
def scripted_refusal(body: Any, *, status: int = 403) -> AgentTls:
"""An agent that answers every `/snapshot` with a refusal, in whatever shape it sent the body."""
def handler(request: httpx.Request) -> httpx.Response:
if isinstance(body, dict):
return httpx.Response(status, json=body)
return httpx.Response(status, text=body)
return AgentTls(verify="/unused/root.pem", cert=("/unused/cert.pem", "/unused/key.pem"),
status=lambda: certs.ACTIVE, transport=httpx.MockTransport(handler))
@pytest.mark.parametrize("body,expected", [
({"ok": False, "detail": "client certificate required"}, "client certificate required"),
({"ok": False, "detail": "client certificate revoked"}, "client certificate revoked"),
("<html><body>403 Forbidden</body></html>", "403 Forbidden"),
("", "no reason given"),
])
async def test_a_refusal_from_a_protected_agent_quotes_its_own_words(
body: Any, expected: str) -> None:
"""The 403 sentence names the refusal the *agent* made, redacted and never blank.
An `UpstreamError` string goes into the health record and the event table, so it has to say
which of the agent's two refusals this was (a required certificate and a revoked one are two
different operator actions) while quoting nothing but the URL it was dialling
(AGENTS.md invariant 6). The `verify`/`cert` paths here do not exist: a scripted refusal
proves the wording, and the tests above prove the handshake.
"""
tls = scripted_refusal(body)
target = Target("gw", "https://SEKRIT-AGENT-TOKEN@agent.test")
async with httpx.AsyncClient(timeout=15.0) as client:
with pytest.raises(UpstreamError) as caught:
await fetch_snapshot(client, target, tls=tls)
message = caught.value.message
assert "agent refused the client certificate" in message
assert expected in message
assert "SEKRIT-AGENT-TOKEN" not in message, "the token leaked into a browser-readable string"
assert "https://agent.test/snapshot" in message, "and it still says where"
@pytest.mark.parametrize("status", [401, 403, 500])
async def test_a_refusal_from_an_unprotected_agent_keeps_the_status_wording(status: int) -> None:
"""The certificate sentence is only for a dial where a certificate was presented.
A token-mode host answers 401 for a bad token and would answer 403 for something that has
nothing to do with certificates; calling that "refused the client certificate" would send
the operator to `/certs` for a token problem.
"""
def handler(request: httpx.Request) -> httpx.Response:
return httpx.Response(status, text="nope")
target = Target("gw", "http://agent.test")
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
with pytest.raises(UpstreamError) as caught:
await fetch_snapshot(client, target,
tls=AgentTls(verify="/unused/root.pem",
cert=("/unused/c.pem", "/unused/k.pem"),
status=lambda: certs.ACTIVE))
if status == 401:
assert "token" in caught.value.message
else:
assert f"HTTP {status}" in caught.value.message
assert "client certificate" not in caught.value.message
async def test_a_token_mode_host_still_uses_the_shared_client(tls_fleet: SimpleNamespace) -> None:
"""An `http://` entry is dialled exactly as it was before phase 15.
The material exists in this deployment (one CA, one client pair, every host), so the scheme
is the only thing that may decide the path — a rollout is per host, and the plain hosts
cannot be allowed to pick up a client certificate they never asked for.
"""
seen: list[str] = []
def handler(request: httpx.Request) -> httpx.Response:
seen.append(str(request.url))
return httpx.Response(200, json=agent_body())
plain = Target("localhost", "http://agent.test")
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
body = await fetch_snapshot(client, plain, tls=tls_fleet.tls)
assert body["ok"] is True
assert seen == ["http://agent.test/snapshot"], "the shared client, the shared transport"
async def test_a_revoked_agent_fails_both_jobs_with_one_sentence(store: Any,
tls_fleet: SimpleNamespace,
monkeypatch: pytest.MonkeyPatch) -> None:
"""The phase's criterion, at the level the health record is written: `host:` and `disks:`.
The two jobs dial one agent on two cadences, so a revocation has to fail both with the same
sentence (one dead source must not blink the other, and both must stop talking to it).
"""
calls: list[str] = []
async def spy_get(self: Any, url: Any, **kwargs: Any) -> Any:
calls.append(str(url))
raise AssertionError(f"a revoked agent was dialled at {url}")
monkeypatch.setattr(httpx.AsyncClient, "get", spy_get)
certs.revoke(tls_fleet.store, tls_fleet.state, "localhost")
async with httpx.AsyncClient(timeout=15.0) as client:
jobs = host_jobs([tls_fleet.target], client, HostCollector(RateEstimator()), 10.0,
tls={"localhost": tls_fleet.tls})
jobs += disks_jobs([tls_fleet.target], client, DisksCollector(), 300.0,
tls={"localhost": tls_fleet.tls})
assert [job.key for job in jobs] == ["host:localhost", "disks:localhost"]
for job in jobs:
assert await job.run_once(store) is False
assert [job.state.error for job in jobs] == [
"UpstreamError: localhost: agent certificate revoked"] * 2
assert calls == []
async def test_host_and_disk_jobs_hand_each_host_its_own_material(
monkeypatch: pytest.MonkeyPatch) -> None:
"""The plumbing, pinned: a host in the map gets its entry, a host outside it gets `None`.
Two hosts and one wrong-key miss is the minimum that shows the lookup is by name rather than
"whatever was built first", which is how a revoked host would keep being dialled with
another host's revocation answer.
"""
seen: dict[str, Any] = {}
def record(name: str) -> Any:
async def scrape(client: Any, target: Target, *_args: Any, **kwargs: Any) -> Result:
seen[f"{name}:{target.name}"] = kwargs.get("tls")
return Result(payload={}, samples=[], events=[])
return scrape
fetch_host = record("system")
fetch_disk = record("disks")
monkeypatch.setattr("dashboards.collectors.fetch_host_snapshot", fetch_host)
monkeypatch.setattr("dashboards.collectors.fetch_disk_snapshot", fetch_disk)
entry = AgentTls(verify="/ca.pem", cert=("/c.pem", "/k.pem"),
status=lambda: certs.ACTIVE)
targets = [Target("gw", "https://gw.test"), Target("desk", "http://desk.test")]
async with httpx.AsyncClient() as client:
for job in (host_jobs(targets, client, HostCollector(RateEstimator()), 10.0,
tls={"gw": entry})
+ disks_jobs(targets, client, DisksCollector(), 300.0, tls={"gw": entry})):
await job.fn()
assert seen == {"system:gw": entry, "system:desk": None,
"disks:gw": entry, "disks:desk": None}
async def test_the_tls_self_report_reaches_both_payloads(disk_snapshot: Any) -> None:
"""`tls` + `bundle_rev` travel into `/api/system/snapshot` and `/api/disks/snapshot`.
Task 05's page has to answer "is the installed bundle the one I just issued?", and both
dashboards read the same snapshot, so both carry the deployed revision under the same
`agent` key. Nothing grades it: a revision is an identity counter, not a threshold.
"""
def stamped(snapshot: dict[str, Any]) -> dict[str, Any]:
"""What a TLS-mode agent's `/snapshot` looks like on the wire."""
snapshot["tls"] = True
snapshot["bundle_rev"] = 3
return snapshot
target = Target("driveripper", "http://agent.test")
async with client_for(lambda request: httpx.Response(200,
json=stamped(agent_body()))) as client:
system = (await fetch_host_snapshot(client, target,
HostCollector(RateEstimator()))).payload
assert system["agent"]["tls"] is True
assert system["agent"]["bundle_rev"] == 3
# the `/disks` payload is built from a capture that also carries SMART blocks, so it gets
# its own stamped snapshot rather than sharing the one above
with_smart = disk_snapshot(mutate=stamped)
async with client_for(lambda request: httpx.Response(200, json=with_smart)) as client:
disks = (await fetch_disk_snapshot(client, target, DisksCollector())).payload
assert disks["agent"] == {"tls": True, "bundle_rev": 3}
async def test_a_token_mode_snapshot_gains_no_tls_fields() -> None:
"""The captures are token-mode agents, and their payloads must stay byte-identical.
`false` and `0` would be numbers the agent never published (rule 2), and an empty block
would be a container that means nothing.
"""
def handler(request: httpx.Request) -> httpx.Response:
return httpx.Response(200, json=agent_body())
target = Target("driveripper", "http://agent.test")
async with client_for(handler) as client:
system = (await fetch_host_snapshot(client, target, HostCollector(RateEstimator()))).payload
assert "tls" not in system["agent"] and "bundle_rev" not in system["agent"]
assert system["agent"]["version"], "the block is still the block it was"
# --------------------------------------------------------------------------- #
# LLM engines
# --------------------------------------------------------------------------- #