phase: 34_radiance_llm_engine
Everything is verified and green. Final report: **Phase 34 final verification pass — complete** - Verified all 6 completed tasks' work against the live fleet and hermetic suite: `llm:radiance` job registered, `llm_engines()` registry drives 404 gate / history defaults / `/api/config` (additive, byte-identical gufo/vllm keys), radiance field map + `/stats` sidecar + 7-position spec curve, `/llm` strip/curve/rows. - **Defect found & fixed (pre-existing, task 05):** the vLLM-only sentence "vLLM exposes no per-slot JSON…" rendered on the **radiance** panel (`slotSection`'s non-gufo branch). Fix: vLLM line gated to `engine === 'vllm'`; radiance's panel slot section renders nothing (its session home is the strip) — `page-llm.js` + the served-byte pin in `tests/test_api.py` (now asserts the radiance branch is empty and the sentence can't leak). Browser-verified; re-shot `data/logs/34-llm-radiance-rest.png` (the 4.5-min wait found no in-flight request, so the busy shot remains the earlier pass's — its strip rows still stand; only the panel text predates the fix). - Tests: `uv run pytest` → **2031 passed, 19 skipped** (100%); `uv run pytest --cov` → **98.73%** (floor 94%; config 100%, llm 99%, collectors 98%, api 99%); `uv run ruff check src tests agent && uv run pyright` → clean; `node --check` OK; `uv pip list --outdated` → empty. - `uv run pytest -m live --live` → **13 passed, 2 skipped** (vLLM unset — correct; live-write opt-in — correct); the read-only radiance test passed. - Completion criteria: (1) engines `gufo`+`radiance`, `fleet.engines_live` 2, `config.llm.radiance.enabled` true — **pass (live curl)**; (2) unset → 404 no-dial + no job — **pass (spy-pinned, green)**; (3) fixtures present, both families (79 `radiance:` / 198 `vllm:` lines), drift pin — **pass**; (4) panel/curve/rows — **pass (re-shot rest screenshot)**; (5) live radiance test — **pass**; (6) suite/coverage/lint/`bash .agents/validate.sh` (exit 0) — **pass**; (7) docs — below; (8) gufo/vLLM behaviour unchanged — **pass (existing pins untouched)**. - Documentation: **README.md** full review — env table, LLM-engines section, architecture diagram, capture-script and `/llm` section all name radiance correctly; no inconsistencies found, no changes needed. **AGENTS.md** — the three task-06 fixes (fleet line, route row, module bullets) are in place and all other statements match reality; no changes needed. - Dev server healthy (`./scripts/dev-server.sh status`: all sources ok, `llm:radiance` answering); all changes left in the working tree for the harness. - Next pending phase: none — 34 was the last phase in `todo/`.
This commit is contained in:
@@ -25,6 +25,7 @@ from __future__ import annotations
|
||||
import asyncio
|
||||
import ipaddress
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sqlite3
|
||||
from collections.abc import Iterator
|
||||
@@ -479,6 +480,86 @@ async def test_gufo_slots_answer_the_pinned_shape(settings: Settings) -> None:
|
||||
assert_slot_shape(rows)
|
||||
|
||||
|
||||
async def test_radiance_answers_the_pinned_shape(settings: Settings) -> None:
|
||||
"""Phase 34: the radiance `/metrics` + `/stats` poll over the **running** engine.
|
||||
|
||||
The production path — `collectors.fetch_engine` with the LLM cadence's estimator,
|
||||
`with_stats` the way `llm_jobs` runs it — so what fails here is a rename between the
|
||||
engine and the map, not a typo in a fixture. Read-only: radiance exposes no write
|
||||
endpoint, so "read-only" is a list of GETs. Skips when `RADIANCE_BASE_URL` is unset
|
||||
(invariant 3's live twin: unset is not deployed here, not broken). The assertions
|
||||
are shape, never numbers: the `stats` block's whitelist, the ordered 7-position spec
|
||||
curve (when the build publishes it), one of the two specced session arms, and every
|
||||
cell the engine answered is a finite number or `None` — an absent gauge is `None`,
|
||||
never a faked `0` (rule 2's live twin). `unparsed` must be empty: a parse error on
|
||||
the live wire is drift, not a tolerance.
|
||||
"""
|
||||
if not settings.radiance_url:
|
||||
pytest.skip("RADIANCE_BASE_URL is unset — that engine is not deployed here")
|
||||
# The span the LLM jobs really get: phase 23 spans the LLM estimator by the poll
|
||||
# interval, not by the host's `RATE_WINDOW_SECONDS` (the other tests' idiom).
|
||||
rates = RateEstimator(span=settings.llm_interval)
|
||||
async with httpx.AsyncClient(timeout=settings.llm_timeout) as client:
|
||||
result = await collectors.fetch_engine(client, "radiance", settings.radiance_url,
|
||||
settings.radiance_api_key, rates,
|
||||
with_stats=True)
|
||||
payload = result.payload
|
||||
assert payload["engine"] == "radiance"
|
||||
assert payload["unparsed"] == 0, (
|
||||
f"a parse error on the live wire is drift, not a tolerance: {payload['unparsed']}")
|
||||
metrics = payload["metrics"]
|
||||
|
||||
# The /stats block, task 03's whitelist: present only when the sidecar answered, and
|
||||
# then only those keys (the engine internals stay out — decision 3).
|
||||
stats = payload.get("stats")
|
||||
if stats is not None:
|
||||
allowed = {"model", "version", "uptime_s", "startup_s", "total_requests", "cards"}
|
||||
extra = set(stats) - allowed
|
||||
assert not extra, f"the /stats subset carried keys the whitelist does not name: {sorted(extra)}"
|
||||
for card in stats.get("cards", []):
|
||||
assert set(card) <= {"index", "name", "vram_used", "vram_total", "vram_used_pct"}, card
|
||||
|
||||
# The spec-acceptance curve: the ordered seven, when the build publishes it.
|
||||
positions = metrics["spec_positions"]
|
||||
assert isinstance(positions, list)
|
||||
if positions:
|
||||
assert len(positions) == 7, (
|
||||
f"the spec curve has {len(positions)} positions — the pin is 0..6, "
|
||||
f"not {[p['position'] for p in positions]}")
|
||||
assert [p["position"] for p in positions] == list(range(7)), ("not 0..6 in order")
|
||||
for pos in positions:
|
||||
ratio = pos["ratio"]
|
||||
assert ratio is None or (isinstance(ratio, int | float)
|
||||
and not isinstance(ratio, bool)
|
||||
and math.isfinite(ratio) and 0.0 <= ratio <= 1.0), (
|
||||
f"position {pos['position']}'s ratio {ratio!r} is not a 0-1 gauge")
|
||||
|
||||
# Every cell the engine answered is a finite number or None — never a faked 0.
|
||||
for key, cell in metrics.items():
|
||||
if key == "spec_positions":
|
||||
continue # the ordered list, checked above
|
||||
if not isinstance(cell, dict) or "value" not in cell:
|
||||
continue
|
||||
value = cell["value"]
|
||||
assert value is None or (isinstance(value, int | float)
|
||||
and not isinstance(value, bool)
|
||||
and math.isfinite(value)), (
|
||||
f"{key}: {value!r} is neither a number nor None")
|
||||
|
||||
# The session arm is one of the two specced arms: /stats answered (the engine's own
|
||||
# summary is the source) or it did not (the /metrics gauges, and it says so).
|
||||
sessions = payload["sessions"]
|
||||
if stats is not None:
|
||||
assert sessions["source"] == "radiance /stats"
|
||||
assert "running" in sessions and "waiting" in sessions and "queue_cap" in sessions
|
||||
else:
|
||||
assert sessions["source"] is None
|
||||
assert sessions.get("queue_cap") is None, ("an absent /stats leaves the cap n/a, never 0")
|
||||
assert "/stats" in (sessions.get("detail") or ""), (
|
||||
f"the fallback arm must name the missing sidecar: {sessions.get('detail')!r}")
|
||||
assert sessions["active"] is None or isinstance(sessions["active"], int | float)
|
||||
|
||||
|
||||
def require_upsd(settings: Settings) -> None:
|
||||
"""Skip, not fail, when there is no real upsd to talk to."""
|
||||
if not settings.nut_enabled:
|
||||
|
||||
Reference in New Issue
Block a user