services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 24_dev_mtls_rollout

Browse Source 
**Phase 24 final verification — all green**

- **Verified (read-only re-run, 62/62 PASS)** via `data/logs/verify-24-mtls.py`: nine sources `ok`; `/api/certs` all three `active` with `deployed_bundle_rev == bundle_rev == 2`; `agent.tls: true` + `agent.version 1.1.0` on every host; driveripper 16/16 drives live; gateway absent from `/disks` but `disks:gateway ok`; all served target URLs `https://`, no userinfo; pre-flip backup present; no transport/cert alert.
- **Criterion 1 (export):** `data/certs/{driveripper,deskwork,gateway}.bundle` re-checked byte-identical (sha256) to `GET /api/certs/<name>/bundle`, mode `0600`, under gitignored `data/`; live arms re-probed (issued→file, unknown/`issued:false`→one-line note + no file, wrong password→exit 2, nothing written).
- **Criterion 2 (`--fetch-bundle`):** 54 sandbox pins in `tests/test_deploy_scripts.py` pass — byte-identical `0640` install + TLS unit lines, refusals (unissued / 401 / non-bundle body / dead dashboard) before `need_root` with no write, token absent from token file, unit, stdout and sudoers hint.
- **Criterion 3–4:** `./scripts/dev-server.sh status` nine `ok` + bundles block; screenshots `data/logs/24-task04-{certs,disks,system}.png` exist.
- **Criterion 7:** no behavior change — `agent/` untouched, `src/` diff is a `certs.ensure_certs` docstring only; token-mode, revocation-before-dial and bundle-route tests green.
- **Tests/lint:** `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1542 passed, 15 skipped, coverage 98.94%**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; `uv run pytest -m live --live` → **11 passed**.
- **Docs:** README.md (all 1144 lines) + AGENTS.md reviewed against the live project — every config.py env var is in README and both templates, route set / severity thresholds / test counts / coverage figure match, all repo paths, cited test names and `Dash.*`/`DashCharts.*` helpers resolve, phase-24 doc pins green. **Verified consistent — no changes needed.**
- **Notable:** left `fastapi 0.153.0 → 0.158.0` (validate reports it as advisory; no dependency was touched, rule 8 conditions the check on deps moving). Killed two abandoned `/tmp/fbtest` loopback stub processes left by task 02's manual probe; dev server unaffected.
- **Next pending phase:** `.agents/phases/todo/25_masthead_control_placement/`.
This commit is contained in:
ducoterra
2026-10-21 13:47:45 -04:00
parent 5f3ed9bcd4
commit 89d9fb9f89
36 changed files with 2378 additions and 146 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_live_fleet.py
+89 -14
View File
@@ -22,12 +22,14 @@ from __future__ import annotations
import asyncio
import os
import sqlite3
from collections.abc import Iterator
import httpx
import pytest
from card_contract import assert_card_contract
from dashboards import collectors
from dashboards import certs, collectors
from dashboards.config import Settings, load
from dashboards.rates import RateEstimator
from dashboards.severity import classify
@@ -46,25 +48,88 @@ def settings() -> Settings:
return load()
class _StoredAuthority:
"""`certs.MetaStore` over an existing database, opened **read-only**.
The live suite has to dial the fleet the way the app does, and since the phase-24 rollout that
means presenting this dashboard's own client certificate. The authority lives in the `meta`
table of the store the dev server owns, so this is the smallest honest reader of it: `mode=ro`
on the URI, `get_meta` the only method that does anything, and the two write methods a
*refusal* rather than a no-op — this file's docstring says nothing in here may write, and
minting or repairing an authority from a test would be the worst write it could make.
"""
def __init__(self, path: str) -> None:
self._conn = sqlite3.connect(f"file:{os.path.abspath(path)}?mode=ro", uri=True)
def close(self) -> None:
self._conn.close()
def get_meta(self, key: str) -> str | None:
row = self._conn.execute("SELECT value FROM meta WHERE key = ?", (key,)).fetchone()
return None if row is None else str(row[0])
def set_meta(self, key: str, value: str) -> None:
raise AssertionError(f"live tests are read-only: refused to write {key!r}")
def delete_meta(self, key: str) -> int:
raise AssertionError(f"live tests are read-only: refused to delete {key!r}")
@pytest.fixture(scope="module")
def agent_tls(settings: Settings) -> Iterator[dict[str, collectors.AgentTls]]:
"""One :class:`collectors.AgentTls` per `https://` host, from the authority in the store.
Empty when no `HOST_TARGETS` entry is https — every host is then dialled on the pre-phase-15
token path, which is what an `http://` fleet still is — and skipped when the store or its
`certs.*` rows are absent, because a box running the live suite without the dev database is a
normal partial environment rather than a failure (the same rule that lets half a fleet be
reachable).
`certs.read_state` is the request-time read and never `ensure_certs`: a live test may not mint
a root CA, and the material this writes is removed on teardown.
"""
targets = settings.host_targets
if not any(target.url.startswith("https://") for target in targets):
yield {}
return
if not os.path.exists(settings.db_path):
pytest.skip(f"no store at {settings.db_path} to read the CA from")
authority = _StoredAuthority(settings.db_path)
try:
state = certs.read_state(authority)
except (RuntimeError, sqlite3.Error) as exc:
authority.close()
pytest.skip(f"no readable authority in {settings.db_path}: {type(exc).__name__}")
material = certs.write_material(state)
try:
yield collectors.agent_tls_map(material, authority, targets)
finally:
certs.remove_material(material)
authority.close()
@pytest.fixture
def host_collector() -> HostCollector:
return HostCollector(rates=RateEstimator(span=10.0))
async def test_every_host_agent_answers_and_parses(settings: Settings,
host_collector: HostCollector) -> None:
host_collector: HostCollector,
agent_tls: dict[str, collectors.AgentTls]) -> None:
"""Each HOST_TARGETS entry must answer /snapshot and yield the metrics the page renders."""
if not settings.host_targets:
pytest.skip("HOST_TARGETS is unset")
required = {"cpu.usage_pct", "mem.used_pct", "disk.used_pct"}
async with httpx.AsyncClient(timeout=settings.host_timeout) as client:
for target in settings.host_targets:
tls = agent_tls.get(target.name)
# Poll twice: usage_pct is a Δ/jiffies rate, and a rate has no second point yet
# on the first scrape. That is the design (the agent never remembers who polled
# it), so a live test that scraped once would be asserting a false failure.
await collectors.fetch_host_snapshot(client, target, host_collector)
await collectors.fetch_host_snapshot(client, target, host_collector, tls=tls)
await asyncio.sleep(1.2)
result = await collectors.fetch_host_snapshot(client, target, host_collector)
result = await collectors.fetch_host_snapshot(client, target, host_collector, tls=tls)
metrics = {sample.metric for sample in result.samples}
assert result.payload["info"]["hostname"], f"{target.name}: no hostname"
missing = required - metrics
@@ -76,7 +141,8 @@ async def test_every_host_agent_answers_and_parses(settings: Settings,
async def test_every_host_serves_the_hero_card_contract(settings: Settings,
host_collector: HostCollector) -> None:
host_collector: HostCollector,
agent_tls: dict[str, collectors.AgentTls]) -> None:
"""The phase-03 `/system` card paths exist on the **real** agents, not just the captures.
Values may honestly be `n/a` on real hardware (gateway reports no CPU sensor and no GPU
@@ -87,16 +153,18 @@ async def test_every_host_serves_the_hero_card_contract(settings: Settings,
pytest.skip("HOST_TARGETS is unset")
async with httpx.AsyncClient(timeout=settings.host_timeout) as client:
for target in settings.host_targets:
await collectors.fetch_host_snapshot(client, target, host_collector)
tls = agent_tls.get(target.name)
await collectors.fetch_host_snapshot(client, target, host_collector, tls=tls)
await asyncio.sleep(1.2)
result = await collectors.fetch_host_snapshot(client, target, host_collector)
result = await collectors.fetch_host_snapshot(client, target, host_collector, tls=tls)
assert_card_contract(result.payload)
assert result.payload["cpu"]["temp_c"] == result.payload["temperatures"]["cpu_c"], (
f"{target.name}: the card and /api/summary would report different CPU temps"
)
async def test_agent_disk_rows_carry_the_smart_shape(settings: Settings) -> None:
async def test_agent_disk_rows_carry_the_smart_shape(
settings: Settings, agent_tls: dict[str, collectors.AgentTls]) -> None:
"""Phase-09: a redeployed agent answers `smart` per physical disk of `/snapshot`.
Agents are redeployed by hand, so an agent whose rows carry no `smart` key at all
@@ -111,10 +179,16 @@ async def test_agent_disk_rows_carry_the_smart_shape(settings: Settings) -> None
seen_smart = 0
async with httpx.AsyncClient(timeout=settings.host_timeout) as client:
for target in settings.host_targets:
response = await client.get(f"{target.url}/snapshot", headers=target.headers)
if response.status_code >= 400:
try:
# through `fetch_snapshot`, not a hand-rolled `get`, because that is the function
# that owns the certificate dial: an `https://` agent needs this app's own CA and
# client leaf, and a test dialling it with the platform trust store would fail on
# a healthy fleet (`CERTIFICATE_VERIFY_FAILED: self-signed certificate in chain`).
response = await collectors.fetch_snapshot(client, target,
tls=agent_tls.get(target.name))
except collectors.UpstreamError:
continue # reachability has its own live test
devices = (response.json().get("disk_io") or {}).get("devices") or []
devices = (response.get("disk_io") or {}).get("devices") or []
if devices and not all("smart" in device for device in devices):
continue # agent predates the smart pass — redeploy it to test this
for device in devices:
@@ -138,7 +212,7 @@ async def test_agent_disk_rows_carry_the_smart_shape(settings: Settings) -> None
async def test_disks_source_charts_the_physical_drives_the_agent_reports(
settings: Settings) -> None:
settings: Settings, agent_tls: dict[str, collectors.AgentTls]) -> None:
"""Phase 09 task 02: the `disks` source over a real agent snapshot.
This is the production code path — the fetch the poller runs, not a hand-built dict —
@@ -153,8 +227,9 @@ async def test_disks_source_charts_the_physical_drives_the_agent_reports(
async with httpx.AsyncClient(timeout=settings.disks_timeout) as client:
for target in settings.host_targets:
try:
result = await collectors.fetch_disk_snapshot(client, target, collector,
timeout=settings.disks_timeout)
result = await collectors.fetch_disk_snapshot(
client, target, collector, timeout=settings.disks_timeout,
tls=agent_tls.get(target.name))
except collectors.UpstreamError:
continue # reachability has its own live test
checked += 1