phase: 34_radiance_llm_engine
Everything is verified and green. Final report: **Phase 34 final verification pass — complete** - Verified all 6 completed tasks' work against the live fleet and hermetic suite: `llm:radiance` job registered, `llm_engines()` registry drives 404 gate / history defaults / `/api/config` (additive, byte-identical gufo/vllm keys), radiance field map + `/stats` sidecar + 7-position spec curve, `/llm` strip/curve/rows. - **Defect found & fixed (pre-existing, task 05):** the vLLM-only sentence "vLLM exposes no per-slot JSON…" rendered on the **radiance** panel (`slotSection`'s non-gufo branch). Fix: vLLM line gated to `engine === 'vllm'`; radiance's panel slot section renders nothing (its session home is the strip) — `page-llm.js` + the served-byte pin in `tests/test_api.py` (now asserts the radiance branch is empty and the sentence can't leak). Browser-verified; re-shot `data/logs/34-llm-radiance-rest.png` (the 4.5-min wait found no in-flight request, so the busy shot remains the earlier pass's — its strip rows still stand; only the panel text predates the fix). - Tests: `uv run pytest` → **2031 passed, 19 skipped** (100%); `uv run pytest --cov` → **98.73%** (floor 94%; config 100%, llm 99%, collectors 98%, api 99%); `uv run ruff check src tests agent && uv run pyright` → clean; `node --check` OK; `uv pip list --outdated` → empty. - `uv run pytest -m live --live` → **13 passed, 2 skipped** (vLLM unset — correct; live-write opt-in — correct); the read-only radiance test passed. - Completion criteria: (1) engines `gufo`+`radiance`, `fleet.engines_live` 2, `config.llm.radiance.enabled` true — **pass (live curl)**; (2) unset → 404 no-dial + no job — **pass (spy-pinned, green)**; (3) fixtures present, both families (79 `radiance:` / 198 `vllm:` lines), drift pin — **pass**; (4) panel/curve/rows — **pass (re-shot rest screenshot)**; (5) live radiance test — **pass**; (6) suite/coverage/lint/`bash .agents/validate.sh` (exit 0) — **pass**; (7) docs — below; (8) gufo/vLLM behaviour unchanged — **pass (existing pins untouched)**. - Documentation: **README.md** full review — env table, LLM-engines section, architecture diagram, capture-script and `/llm` section all name radiance correctly; no inconsistencies found, no changes needed. **AGENTS.md** — the three task-06 fixes (fleet line, route row, module bullets) are in place and all other statements match reality; no changes needed. - Dev server healthy (`./scripts/dev-server.sh status`: all sources ok, `llm:radiance` answering); all changes left in the working tree for the harness. - Next pending phase: none — 34 was the last phase in `todo/`.
This commit is contained in:
+226
-1
@@ -20,6 +20,7 @@ import httpx
|
||||
import pytest
|
||||
|
||||
from dashboards import certs
|
||||
from dashboards.app import build_hub
|
||||
from dashboards.collectors import (
|
||||
AgentTls,
|
||||
NutMonitor,
|
||||
@@ -33,10 +34,12 @@ from dashboards.collectors import (
|
||||
fetch_snapshot,
|
||||
gateway_job,
|
||||
host_jobs,
|
||||
llm_jobs,
|
||||
post_route,
|
||||
)
|
||||
from dashboards.config import Settings, Target
|
||||
from dashboards.poller import Result
|
||||
from dashboards.prom import Prom
|
||||
from dashboards.rates import RateEstimator
|
||||
from dashboards.sources import nut
|
||||
from dashboards.sources.disks import DisksCollector
|
||||
@@ -1121,7 +1124,10 @@ async def test_first_gufo_poll_has_no_measured_rate_yet() -> None:
|
||||
decode = result.payload["metrics"]["decode_tokens_per_second"]
|
||||
assert decode["value"] is None
|
||||
assert decode["source"] is None
|
||||
assert result.payload["sessions"]["active"] == 0.0
|
||||
# `active` is the /metrics gauge, verbatim the fixture's own number — the capture
|
||||
# may have been taken with a request in flight (volatile between re-captures)
|
||||
assert result.payload["sessions"]["active"] == \
|
||||
Prom(GUFO).scalar(["llamacpp:requests_processing"])
|
||||
assert "no /slots answer" in result.payload["sessions"]["detail"]
|
||||
|
||||
|
||||
@@ -1204,6 +1210,225 @@ async def test_engine_timeout_is_504() -> None:
|
||||
assert excinfo.value.status == 504
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# radiance: the /stats sidecar (phase 34, task 04)
|
||||
# --------------------------------------------------------------------------- #
|
||||
RADIANCE = (FIXTURES / "metrics_radiance.txt").read_text()
|
||||
|
||||
|
||||
def _radiance_stats() -> dict[str, Any]:
|
||||
"""The capture's /stats document (the `#` header lines are the script's, not JSON)."""
|
||||
body = "\n".join(line for line in
|
||||
(FIXTURES / "stats_radiance.json").read_text().splitlines()
|
||||
if not line.startswith("#"))
|
||||
return json.loads(body)
|
||||
|
||||
|
||||
def fake_engine(metrics: httpx.Response | BaseException,
|
||||
stats: httpx.Response | BaseException | None = None
|
||||
) -> tuple[httpx.AsyncClient, list[str]]:
|
||||
"""One fake engine: a scripted `/metrics` answer, and a scripted `/stats` sidecar answer
|
||||
if the job dials for it; every requested path is kept, so a retry or an alias shows up.
|
||||
"""
|
||||
seen: list[str] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
seen.append(request.url.path)
|
||||
if request.url.path == "/metrics":
|
||||
if isinstance(metrics, BaseException):
|
||||
raise metrics
|
||||
return metrics
|
||||
if request.url.path == "/stats":
|
||||
assert stats is not None, "the job dialled /stats without one scripted"
|
||||
if isinstance(stats, BaseException):
|
||||
raise stats
|
||||
return stats
|
||||
return httpx.Response(404)
|
||||
|
||||
return httpx.AsyncClient(transport=httpx.MockTransport(handler)), seen
|
||||
|
||||
|
||||
async def test_a_configured_radiance_alone_builds_exactly_one_llm_job() -> None:
|
||||
"""Task 02's registry shape, extended with the job's own fields: a configured
|
||||
`RADIANCE_BASE_URL` (gufo and vLLM unset) is exactly one job, with the LLM dashboard,
|
||||
the registry's display name and the LLM cadence — the same three facts `llm:gufo` and
|
||||
`llm:vllm` already carried."""
|
||||
settings = Settings(radiance_url="http://radiance.test:8000")
|
||||
async with httpx.AsyncClient() as client:
|
||||
jobs = llm_jobs(settings, client, RateEstimator())
|
||||
assert len(jobs) == 1
|
||||
job = jobs[0]
|
||||
assert job.key == "llm:radiance"
|
||||
assert job.dashboard == "llm"
|
||||
assert job.label == "radiance"
|
||||
assert job.interval == settings.llm_interval, "the LLM cadence, not the host's"
|
||||
|
||||
|
||||
async def test_the_hub_wires_the_radiance_job_with_the_llm_cadence_and_estimator(
|
||||
store: Any, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""`app.py` needed no change — this is the check, asserted by running the job through
|
||||
its own `fn` rather than by comment: the `llm:radiance` job flows into the hub from
|
||||
`llm_jobs` with the LLM interval and the LLM-span estimator (one estimator per
|
||||
measuring cadence, phase 23), and the sidecar flag reaches `fetch_engine` from the
|
||||
job's call alone."""
|
||||
seen: dict[str, Any] = {}
|
||||
|
||||
async def spy_fetch(client: Any, engine: str, base_url: str, api_key: str,
|
||||
rates: RateEstimator, **kwargs: Any) -> Result:
|
||||
seen["engine"] = engine
|
||||
seen["rates"] = rates
|
||||
seen["kwargs"] = kwargs
|
||||
return Result(payload={"engine": engine})
|
||||
|
||||
monkeypatch.setattr("dashboards.collectors.fetch_engine", spy_fetch)
|
||||
settings = Settings(radiance_url="http://radiance.test:8000")
|
||||
async with httpx.AsyncClient() as client:
|
||||
hub, _ = build_hub(settings, store, client)
|
||||
assert "llm:radiance" in hub.jobs
|
||||
job = hub.jobs["llm:radiance"]
|
||||
assert job.interval == settings.llm_interval
|
||||
assert await job.run_once(store) is True
|
||||
assert seen["engine"] == "radiance"
|
||||
assert seen["rates"].span == settings.llm_interval, "the LLM-span estimator"
|
||||
assert seen["kwargs"] == {"with_slots": False, "with_stats": True}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("settings, expected", [
|
||||
(Settings(gufo_url="http://gufo.test:8731"), ["/metrics", "/slots"]),
|
||||
(Settings(vllm_url="http://vllm.test:8000"), ["/metrics"]),
|
||||
(Settings(radiance_url="http://radiance.test:8000"), ["/metrics", "/stats"]),
|
||||
])
|
||||
async def test_each_engine_job_dials_only_its_own_sidecars(
|
||||
settings: Settings, expected: list[str]) -> None:
|
||||
"""The grep criterion, proven by behaviour: `llm_jobs`' flag is what makes
|
||||
`fetch_engine` dial `/stats` — the gufo job's `/slots` pass and the vLLM job's bare
|
||||
`/metrics` read behave exactly as before (gufo's alias aside, a 200 `/slots` answers
|
||||
before `/v1/slots` is tried), and radiance is the only engine that dials the sidecar,
|
||||
exactly once and on its unaliased path."""
|
||||
text_by_host = {"gufo.test": GUFO, "vllm.test": VLLM, "radiance.test": RADIANCE}
|
||||
paths: list[str] = []
|
||||
|
||||
def handler(request: httpx.Request) -> httpx.Response:
|
||||
paths.append(request.url.path)
|
||||
if request.url.path == "/metrics":
|
||||
return httpx.Response(200, text=text_by_host[request.url.host])
|
||||
if request.url.path == "/slots":
|
||||
return httpx.Response(200, json=[{"index": 0}])
|
||||
if request.url.path == "/stats":
|
||||
return httpx.Response(200, json=_radiance_stats())
|
||||
return httpx.Response(404)
|
||||
|
||||
async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client:
|
||||
jobs = llm_jobs(settings, client, RateEstimator())
|
||||
assert len(jobs) == 1
|
||||
result = await jobs[0].fn()
|
||||
assert isinstance(result, Result)
|
||||
assert paths == expected, f"{jobs[0].key}: the job dials only its own sidecars"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("stats_failure", ["http-500", "non-json", "json-list", "connection-error"])
|
||||
async def test_a_dead_or_partial_stats_sidecar_keeps_the_metrics_poll_healthy(
|
||||
stats_failure: str) -> None:
|
||||
"""Invariant 4, the sidecar's own arm: a `/stats` that is dead, 500, non-JSON, or a
|
||||
JSON that is not a dict degrades the session arm (task 03's missing-stats fallback —
|
||||
the counts from the /metrics gauges, the cap n/a, never 0) and never takes the
|
||||
`/metrics` poll down: one best-effort read, no retry, no second dial, no alias, no
|
||||
`UpstreamError`."""
|
||||
if stats_failure == "http-500":
|
||||
stats: Any = httpx.Response(500, text="sidecar down")
|
||||
elif stats_failure == "non-json":
|
||||
stats = httpx.Response(200, text="not json at all")
|
||||
elif stats_failure == "json-list":
|
||||
stats = httpx.Response(200, json=[1, 2])
|
||||
else:
|
||||
stats = httpx.ConnectError("sidecar down")
|
||||
client, seen = fake_engine(httpx.Response(200, text=RADIANCE), stats)
|
||||
async with client:
|
||||
result = await fetch_engine(client, "radiance", "http://radiance.test:8000",
|
||||
"", RateEstimator(), with_stats=True)
|
||||
assert result.payload["engine"] == "radiance"
|
||||
assert result.payload["metrics"]["num_requests_running"]["value"] == \
|
||||
Prom(RADIANCE).scalar(["vllm:num_requests_running"]), \
|
||||
"the /metrics answer is the poll's one truth, verbatim the fixture's gauge"
|
||||
assert "stats" not in result.payload, "an absent or non-dict sidecar carries nothing"
|
||||
detail = result.payload["sessions"]["detail"]
|
||||
assert "did not answer" in detail, "the session arm names the missing sidecar"
|
||||
assert "gauges" in detail, "…and the fallback counts' different provenance"
|
||||
assert result.payload["sessions"]["queue_cap"] is None, "n/a, never 0 (rule 2)"
|
||||
assert [path for path in seen if path == "/stats"] == ["/stats"], (
|
||||
"one best-effort read, no retry, no second dial")
|
||||
|
||||
|
||||
async def test_a_healthy_stats_sidecar_travels_into_the_result() -> None:
|
||||
client, seen = fake_engine(httpx.Response(200, text=RADIANCE),
|
||||
httpx.Response(200, json=_radiance_stats()))
|
||||
async with client:
|
||||
result = await fetch_engine(client, "radiance", "http://radiance.test:8000",
|
||||
"", RateEstimator(), with_stats=True)
|
||||
block = result.payload["stats"]
|
||||
assert block["model"] == "juggernaut"
|
||||
for key in ("version", "uptime_s", "startup_s", "total_requests", "cards"):
|
||||
assert key in block, "the whitelisted subset the page reads (task 03)"
|
||||
assert block["cards"][0]["name"] == "AMD Radeon AI PRO R9700"
|
||||
assert "vram_used_pct" in block["cards"][0], "the one computed number in the block"
|
||||
sessions = result.payload["sessions"]
|
||||
assert sessions["source"] == "radiance /stats"
|
||||
assert sessions["queue_cap"] == _radiance_stats()["queue_cap"]
|
||||
if not _radiance_stats()["requests"]:
|
||||
assert "counts, not sessions" in sessions["detail"], "the capture is at rest"
|
||||
assert seen == ["/metrics", "/stats"], "one unaliased read, nothing else"
|
||||
|
||||
|
||||
async def test_a_dead_metrics_is_the_upstream_error_and_the_sidecar_never_masks_it() -> None:
|
||||
"""A 500 `/metrics` beside a perfect `/stats`: the failure is the metrics one, and
|
||||
the sidecar is never even consulted — consulted first it could mask the real failure,
|
||||
consulted after it would only be wasted work on a poll that is already failed."""
|
||||
client, seen = fake_engine(httpx.Response(500, text="boom"),
|
||||
httpx.Response(200, json=_radiance_stats()))
|
||||
async with client:
|
||||
with pytest.raises(UpstreamError) as excinfo:
|
||||
await fetch_engine(client, "radiance", "http://radiance.test:8000",
|
||||
"", RateEstimator(), with_stats=True)
|
||||
assert "HTTP 500" in excinfo.value.message
|
||||
assert "/metrics" in excinfo.value.message
|
||||
assert "/stats" not in seen, "a failed metrics dial never reaches for the sidecar"
|
||||
|
||||
|
||||
async def test_a_failing_radiance_never_echoes_credentials_in_its_url() -> None:
|
||||
"""Invariant 6 for the radiance dial, the engine-fetch precedent extended: a
|
||||
`RADIANCE_BASE_URL` written with a userinfo token fails with the redacted URL, and
|
||||
the key-free form leaks nothing either — there is nothing to leak."""
|
||||
client, _ = fake_engine(httpx.ConnectError("connection refused"))
|
||||
async with client:
|
||||
with pytest.raises(UpstreamError, match="unreachable") as excinfo:
|
||||
await fetch_engine(client, "radiance",
|
||||
"http://SEKRIT-RADIANCE-TOKEN@radiance.test:8000", "",
|
||||
RateEstimator(), with_stats=True)
|
||||
assert "SEKRIT-RADIANCE-TOKEN" not in excinfo.value.message
|
||||
assert "http://radiance.test:8000/metrics" in excinfo.value.message
|
||||
|
||||
|
||||
async def test_the_job_list_never_dials_an_unconfigured_engine(
|
||||
monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The registry's gate, spy-pinned at the dial layer (the existing spy pattern): with
|
||||
every engine unset, `llm_jobs` builds nothing, and nothing that was built dials
|
||||
anything — an unconfigured engine has no job to fail and no dial to hang (invariant 3)."""
|
||||
calls: list[str] = []
|
||||
|
||||
async def spy_get(self: Any, url: Any, **kwargs: Any) -> Any:
|
||||
calls.append(str(url))
|
||||
raise AssertionError(f"an unconfigured engine was dialled at {url}")
|
||||
|
||||
monkeypatch.setattr(httpx.AsyncClient, "get", spy_get)
|
||||
settings = Settings()
|
||||
async with httpx.AsyncClient() as client:
|
||||
jobs = llm_jobs(settings, client, RateEstimator())
|
||||
for job in jobs:
|
||||
await job.fn()
|
||||
assert jobs == []
|
||||
assert calls == []
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# NUT monitor
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
Reference in New Issue
Block a user