phase: 34_radiance_llm_engine
Everything is verified and green. Final report: **Phase 34 final verification pass — complete** - Verified all 6 completed tasks' work against the live fleet and hermetic suite: `llm:radiance` job registered, `llm_engines()` registry drives 404 gate / history defaults / `/api/config` (additive, byte-identical gufo/vllm keys), radiance field map + `/stats` sidecar + 7-position spec curve, `/llm` strip/curve/rows. - **Defect found & fixed (pre-existing, task 05):** the vLLM-only sentence "vLLM exposes no per-slot JSON…" rendered on the **radiance** panel (`slotSection`'s non-gufo branch). Fix: vLLM line gated to `engine === 'vllm'`; radiance's panel slot section renders nothing (its session home is the strip) — `page-llm.js` + the served-byte pin in `tests/test_api.py` (now asserts the radiance branch is empty and the sentence can't leak). Browser-verified; re-shot `data/logs/34-llm-radiance-rest.png` (the 4.5-min wait found no in-flight request, so the busy shot remains the earlier pass's — its strip rows still stand; only the panel text predates the fix). - Tests: `uv run pytest` → **2031 passed, 19 skipped** (100%); `uv run pytest --cov` → **98.73%** (floor 94%; config 100%, llm 99%, collectors 98%, api 99%); `uv run ruff check src tests agent && uv run pyright` → clean; `node --check` OK; `uv pip list --outdated` → empty. - `uv run pytest -m live --live` → **13 passed, 2 skipped** (vLLM unset — correct; live-write opt-in — correct); the read-only radiance test passed. - Completion criteria: (1) engines `gufo`+`radiance`, `fleet.engines_live` 2, `config.llm.radiance.enabled` true — **pass (live curl)**; (2) unset → 404 no-dial + no job — **pass (spy-pinned, green)**; (3) fixtures present, both families (79 `radiance:` / 198 `vllm:` lines), drift pin — **pass**; (4) panel/curve/rows — **pass (re-shot rest screenshot)**; (5) live radiance test — **pass**; (6) suite/coverage/lint/`bash .agents/validate.sh` (exit 0) — **pass**; (7) docs — below; (8) gufo/vLLM behaviour unchanged — **pass (existing pins untouched)**. - Documentation: **README.md** full review — env table, LLM-engines section, architecture diagram, capture-script and `/llm` section all name radiance correctly; no inconsistencies found, no changes needed. **AGENTS.md** — the three task-06 fixes (fleet line, route row, module bullets) are in place and all other statements match reality; no changes needed. - Dev server healthy (`./scripts/dev-server.sh status`: all sources ok, `llm:radiance` answering); all changes left in the working tree for the harness. - Next pending phase: none — 34 was the last phase in `todo/`.
This commit is contained in:
@@ -3362,3 +3362,213 @@ def test_the_install_checks_the_agent_can_reach_the_socket(tmp_path: Path) -> No
|
||||
assert result.returncode == 1, "an unreachable socket is a failed install, not a silent success"
|
||||
assert "cannot connect" in result.stderr
|
||||
assert "sudo -u dashboards-agent python3" in call_log.read_text(), "the check ran as the agent uid"
|
||||
|
||||
|
||||
# --- scripts/capture-fixtures.sh: the radiance pass (phase 34, task 01) --------------
|
||||
#
|
||||
# The script learns radiance through the same `--with-llm` gate as the other engines and
|
||||
# carries no deployment detail of its own: RADIANCE_BASE_URL is the dial, `/metrics`
|
||||
# joins the per-engine loop, and `/stats` + `/v1/models` are recorded in the same pass
|
||||
# (radiance is a vLLM derivative; the vLLM engine and env var stay for other deployments).
|
||||
# A stub `curl` answers the three endpoints from canned bodies, so the script's own
|
||||
# branches run — verbatim bodies, the script's header, the at-rest hint, the
|
||||
# never-overwrite-a-good-fixture rule — with no engine in sight.
|
||||
|
||||
RADIANCE_CAPTURE = SCRIPTS / "capture-fixtures.sh"
|
||||
|
||||
#: The stub `curl`: it drops the options the script passes (`-fsS`, `--max-time N`, `-o`),
|
||||
#: records every call, and answers the engine's three read-only endpoints from `$CANNED_*`.
|
||||
#: Nothing here can reach a network.
|
||||
RADIANCE_CURL_STUB = """#!/bin/sh
|
||||
printf '%s\\n' "$*" >> "$STUB_LOG"
|
||||
if [ "${FAIL_CURL:-0}" = "1" ]; then
|
||||
echo "curl: (7) Failed to connect" >&2
|
||||
exit 7
|
||||
fi
|
||||
out=""
|
||||
url=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--max-time) shift 2 ;;
|
||||
-o) out="$2"; shift 2 ;;
|
||||
-*) shift ;;
|
||||
*) url="$1"; shift ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$out" ] || { echo "stub curl: no -o target" >&2; exit 2; }
|
||||
case "$url" in
|
||||
*/metrics) printf '%s' "$CANNED_METRICS" > "$out" ;;
|
||||
*/stats) printf '%s' "$CANNED_STATS" > "$out" ;;
|
||||
*/v1/models) printf '%s' "$CANNED_MODELS" > "$out" ;;
|
||||
*) echo "stub curl: unexpected URL $url" >&2; exit 2 ;;
|
||||
esac
|
||||
"""
|
||||
|
||||
#: The seven per-position spec-acceptance lines: the phase pins position="0".."6".
|
||||
RADIANCE_CANNED_METRICS = (
|
||||
"# HELP vllm:num_requests_running Number of requests currently running on GPU.\n"
|
||||
"# TYPE vllm:num_requests_running gauge\n"
|
||||
'vllm:num_requests_running{model_name="juggernaut",engine="0"} 0\n'
|
||||
'radiance:decode_tokens_per_second{model_name="juggernaut",engine="0"} 110.9\n'
|
||||
+ "".join(
|
||||
f'radiance:draft_position_acceptance_ratio{{model_name="juggernaut",engine="0",'
|
||||
f'position="{i}"}} 0.7\n'
|
||||
for i in range(7)
|
||||
)
|
||||
)
|
||||
|
||||
RADIANCE_CANNED_STATS = (
|
||||
'{"model":"juggernaut","version":"1.15.1","uptime_s":67490,"queue_cap":24,'
|
||||
'"in_flight":0,"kv_util":0.7241322994232178,"prefix_hit_rate":0.9572382569313049,'
|
||||
'"draft_positions":[{"reached":701741,"accepted":516252,"rate":0.7356731329650114}],'
|
||||
'"requests":[],"cards":[{"index":0,"name":"AMD Radeon AI PRO R9700",'
|
||||
'"vram_used":32119517184,"vram_total":34208743424}]}'
|
||||
)
|
||||
|
||||
RADIANCE_CANNED_STATS_BUSY = (
|
||||
'{"model":"juggernaut","version":"1.15.1","uptime_s":67500,"queue_cap":24,'
|
||||
'"in_flight":1,"kv_util":0.72,"prefix_hit_rate":0.95,'
|
||||
'"draft_positions":[{"reached":701741,"accepted":516252,"rate":0.735}],'
|
||||
'"requests":[{"id":"req-1","queue_pos":1}],'
|
||||
'"cards":[{"index":0,"name":"AMD Radeon AI PRO R9700",'
|
||||
'"vram_used":32119517184,"vram_total":34208743424}]}'
|
||||
)
|
||||
|
||||
RADIANCE_CANNED_MODELS = (
|
||||
'{"object":"list","data":[{"id":"juggernaut","object":"model","created":1793484879,'
|
||||
'"owned_by":"radiance","max_model_len":131072,"capabilities":{"chat":true}}]}'
|
||||
)
|
||||
|
||||
RADIANCE_BASE = "http://radiance.test:8000"
|
||||
|
||||
|
||||
def _radiance_capture_sandbox(
|
||||
tmp_path: Path,
|
||||
*,
|
||||
recorded: dict[str, str] | None = None,
|
||||
fail_curl: bool = False,
|
||||
busy: bool = False,
|
||||
no_engine: bool = False,
|
||||
) -> tuple[Path, Path, Path, dict[str, str]]:
|
||||
"""A copy of the script, a canned radiance, and a PATH whose `curl` reaches only the canned.
|
||||
|
||||
Returns (script, fixtures dir, stub call log, env). The script is copied into
|
||||
`<tmp>/scripts/` because it derives `tests/fixtures/` from its own location, which is
|
||||
what keeps the repository's fixtures out of reach of a test.
|
||||
"""
|
||||
script = tmp_path / "scripts" / RADIANCE_CAPTURE.name
|
||||
script.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copyfile(RADIANCE_CAPTURE, script)
|
||||
fixtures = tmp_path / "tests" / "fixtures"
|
||||
fixtures.mkdir(parents=True, exist_ok=True)
|
||||
for name, body in (recorded or {}).items():
|
||||
(fixtures / name).write_text(body, encoding="utf-8")
|
||||
|
||||
stubs = tmp_path / "stubs"
|
||||
stubs.mkdir()
|
||||
stub = stubs / "curl"
|
||||
stub.write_text(RADIANCE_CURL_STUB, encoding="utf-8")
|
||||
stub.chmod(0o755)
|
||||
log = tmp_path / "curl.log"
|
||||
|
||||
env = {
|
||||
"PATH": f"{stubs}{os.pathsep}{os.environ['PATH']}",
|
||||
"STUB_LOG": str(log),
|
||||
"CANNED_METRICS": RADIANCE_CANNED_METRICS,
|
||||
"CANNED_STATS": RADIANCE_CANNED_STATS_BUSY if busy else RADIANCE_CANNED_STATS,
|
||||
"CANNED_MODELS": RADIANCE_CANNED_MODELS,
|
||||
}
|
||||
if not no_engine:
|
||||
env["RADIANCE_BASE_URL"] = RADIANCE_BASE
|
||||
if fail_curl:
|
||||
env["FAIL_CURL"] = "1"
|
||||
return script, fixtures, log, env
|
||||
|
||||
|
||||
def _radiance_run(tmp_path: Path, **kwargs: object) -> subprocess.CompletedProcess[str]:
|
||||
script, _fixtures, _log, env = _radiance_capture_sandbox(tmp_path, **kwargs) # type: ignore[arg-type]
|
||||
return subprocess.run([BASH, str(script), "--with-llm"], capture_output=True, text=True,
|
||||
env=env, check=False)
|
||||
|
||||
|
||||
def _dialled_urls(log: Path) -> list[str]:
|
||||
urls: list[str] = []
|
||||
for line in log.read_text().splitlines():
|
||||
for token in line.split():
|
||||
if token.startswith("http"):
|
||||
urls.append(token)
|
||||
return urls
|
||||
|
||||
|
||||
def _assert_verbatim(path: Path, canned: str) -> None:
|
||||
"""The engine's reply inside a fixture, claimed verbatim.
|
||||
|
||||
The script's header is comment text; the engine's reply is the file's complete tail —
|
||||
so the reply is verbatim iff the file ends with it byte for byte and everything before
|
||||
it is comment lines (the metrics body's own `# HELP` / `# TYPE` lines are part of the
|
||||
reply and stay in the fixture).
|
||||
"""
|
||||
text = path.read_text()
|
||||
assert text.endswith(canned), f"{path.name} lost the tail of the engine's reply"
|
||||
prefix = text[: -len(canned)]
|
||||
non_comment = [line for line in prefix.splitlines() if line and not line.startswith("#")]
|
||||
assert not non_comment, f"{path.name} has non-comment text before the reply: {non_comment}"
|
||||
assert prefix, f"{path.name} lost its verbatim header"
|
||||
|
||||
|
||||
def test_the_radiance_capture_records_all_three_endpoints_verbatim(tmp_path: Path) -> None:
|
||||
"""The happy path, run: three fixtures, each with the script's verbatim header, the body
|
||||
byte-for-byte what the engine answered, and a dial to exactly the three read-only
|
||||
endpoints — nothing else, in the documented order."""
|
||||
result = _radiance_run(tmp_path)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
fixtures = tmp_path / "tests" / "fixtures"
|
||||
assert {p.name for p in fixtures.iterdir()} == {
|
||||
"metrics_radiance.txt", "stats_radiance.json", "models_radiance.json"}
|
||||
for name in ("metrics_radiance.txt", "stats_radiance.json", "models_radiance.json"):
|
||||
text = (fixtures / name).read_text()
|
||||
assert text.startswith("# Verbatim"), f"{name} lost its verbatim header"
|
||||
assert "./scripts/capture-fixtures.sh" in text, f"{name} names its recorder"
|
||||
_assert_verbatim(fixtures / "metrics_radiance.txt", RADIANCE_CANNED_METRICS)
|
||||
_assert_verbatim(fixtures / "stats_radiance.json", RADIANCE_CANNED_STATS)
|
||||
_assert_verbatim(fixtures / "models_radiance.json", RADIANCE_CANNED_MODELS)
|
||||
assert _dialled_urls(fixtures.parent.parent / "curl.log") == [
|
||||
f"{RADIANCE_BASE}/metrics", f"{RADIANCE_BASE}/stats", f"{RADIANCE_BASE}/v1/models"]
|
||||
# the capture at rest records `requests: []` — the script says so, so the operator
|
||||
# re-runs with a request in flight to record its row shape
|
||||
assert "requests[] was empty — re-run with a request in flight to record its shape" in result.stdout
|
||||
|
||||
|
||||
def test_the_radiance_capture_says_nothing_to_re_run_when_a_request_is_in_flight(tmp_path: Path) -> None:
|
||||
"""With a request in flight the hint is gone (it would send the operator to re-run a
|
||||
capture that already has the row shape), and the header still records what it saw."""
|
||||
result = _radiance_run(tmp_path, busy=True)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
assert "re-run with a request in flight" not in result.stdout
|
||||
stats = (tmp_path / "tests" / "fixtures" / "stats_radiance.json").read_text()
|
||||
assert 'requests":[{"id":"req-1","queue_pos":1}]' in stats
|
||||
assert "in_flight was 1 at capture" in stats
|
||||
|
||||
|
||||
def test_the_radiance_capture_without_an_engine_says_which_env_var_it_wants(tmp_path: Path) -> None:
|
||||
"""No engine env var is not a silent no-op: the gate names the var, like the others."""
|
||||
script, _fixtures, _log, env = _radiance_capture_sandbox(tmp_path, no_engine=True)
|
||||
result = subprocess.run([BASH, str(script), "--with-llm"], capture_output=True, text=True,
|
||||
env=env, check=False)
|
||||
assert result.returncode == 1
|
||||
assert "no GUFO_BASE_URL / VLLM_BASE_URL / RADIANCE_BASE_URL set" in result.stdout
|
||||
|
||||
|
||||
def test_a_dead_radiance_engine_never_replaces_a_recorded_capture(tmp_path: Path) -> None:
|
||||
"""The `slots_gufo.json` rule: a capture script must never turn three good fixtures into
|
||||
three empty files, because then the shape tests pass against nothing (or fail for the
|
||||
wrong reason) and the ground truth is gone."""
|
||||
good = {"metrics_radiance.txt": "# old\nradiance:kept 1\n",
|
||||
"stats_radiance.json": "# old\n{\"cards\":[]}\n",
|
||||
"models_radiance.json": "# old\n{\"object\":\"list\"}\n"}
|
||||
script, fixtures, _log, env = _radiance_capture_sandbox(tmp_path, recorded=good, fail_curl=True)
|
||||
result = subprocess.run([BASH, str(script), "--with-llm"], capture_output=True, text=True,
|
||||
env=env, check=False)
|
||||
assert result.returncode == 1
|
||||
for name, body in good.items():
|
||||
assert (fixtures / name).read_text() == body, f"{name} was replaced by a dead engine"
|
||||
|
||||
Reference in New Issue
Block a user