phase: 01_severity_language

**Gate defect found & fixed (the "does not lint" bug).** `.agents/validate.sh` — the phase gate — ran `python3 -m pytest` on `/usr/bin/python3` + `~/.local` site-packages (not the uv venv), ran `ruff check .` *only if* a ruff happened to be on PATH, and **never ran pyright or the coverage floor**. Rewrote it to the project's real gate: cd to repo root → `uv run pytest --cov --cov-fail-under=94` → `uv run ruff check src tests agent` → `uv run pyright` → `node --check web/static/*.js` → hygiene gates; a missing toolchain now fails loudly. Provoked it to confirm it bites (floor + a seeded F401 both fail).

**Verification (all green)**
- `bash .agents/validate.sh` → **exit 0**: 356 passed, 6 skipped · coverage **94.75%** (floor 94) · ruff `All checks passed!` · pyright `0 errors` · node syntax clean
- `uv run pytest --cov` → `severity.py` **100%**; `uv run pytest -m live --live` → **6 passed**; `uv pip list --outdated` → empty
- Criteria: `classify('cpu.usage_pct',…)` → `fail`/`ok`/`None` ✓ · `/api/ups/snapshot` carries `severity` beside `charge_pct`/`load_pct`/`battery_runtime` (apc now live-`warn` at 498 s — threshold working) ✓ · no `.sev-*`/synonym vocabulary (only a comment saying so) ✓ · pytest/coverage/lint ✓ · dev server restarted, `/api/health` ok 6/6 sources, all pages+assets 200 ✓
- Chromium against live fleet: `severityFrom` cell→`fail`, bare→`warn`, null→`''`, unrated→`''`; `hero` fail → `rgb(207,34,46)`, null → `n/a` with **no** verdict class, warn+bar 42%; all four pages render unchanged → `data/logs/phase01-final-{index,system,llm,ups}.png`

**Docs (full review, incl. pre-existing drift).** README: `/ups` row claimed "upsd on 3493" (fleet is 3051, stated elsewhere in the same file); config table omitted `DASH_HOST/PORT`, llm/ups `*_INTERVAL_SECONDS`, `*_TIMEOUT_SECONDS`, `GUFO/VLLM_API_KEY`, `POLL_MS_*`, `DEBUGPY` (+ dedup); severity bullet omitted `disks`/`gpus[]`/rated LLM cells and the ratio rescale; agent section listed only `/snapshot` though `/info`+`/health` exist. AGENTS.md: same 3493 claim; `install-agent.sh --token-file` (its flags are `--bind/--port/--token/--rotate-token/--status/--uninstall` — it hands *the agent* `--token-file`); gap 3 "Two commits" (now 6, still no `origin`); gap 1 given the sudoers install command + verified live gating reason; coverage line now 80% repo floor vs 94% gate floor; cyberpower `identity.model: GX1500U` vs `ups.description` quirk. `.env.dev.example`: `DEV_FORWARDS` remote port 3493→3051 and engine URLs that pointed at `127.0.0.1` with nothing forwarding them.

**Deviations.** No `ruff format` (30 files would reformat; the project gate is `ruff check` + pyright — reformatting completed phases is out of scope). **No live-infra changes**: enabling the battery-test buttons needs `deploy/nut-sudoers` installed on driveripper + a nut-server restart and a scoped `[dashboards]` `instcmds` user — prod write with a documented shutdown risk (AGENTS gap 1), so repo-side only. 36 pre-existing `ResourceWarning: unclosed database` from test fixtures are noise, not failures; left rather than churn ~15 fixtures. `severity.attach()` is unit-tested but has no production call site until phases 02–05.

**Next pending phase:** `02_overview_fleet_and_alerts` (`01_alerts_builder.md`).
This commit is contained in:
2026-10-16 22:22:26 -04:00
parent 4b4962d7fb
commit ad993c7b03
40 changed files with 1384 additions and 25 deletions
+46
View File
@@ -100,6 +100,35 @@ def test_absent_gufo_metrics_are_none_with_no_source() -> None:
assert metrics.get("spec_accept_rate", {}).get("value") is None
# ---- severity: KV pressure arrives with a verdict ------------------------- #
def test_gufo_kv_cache_cell_keeps_provenance_and_gains_severity(metrics_text) -> None:
metrics = parse_gufo_metrics(metrics_text("gufo"))
cell = metrics["kv_cache_usage_ratio"]
assert cell["value"] == 0.0
assert cell["source"] == "llamacpp:kv_cache_usage_ratio"
assert cell["derived"] is False, "the verdict is additive; the cell still says what it was"
assert cell["severity"] == "ok"
@pytest.mark.parametrize("ratio,expected", [
("0.74", "ok"), ("0.75", "warn"), ("0.89", "warn"), ("0.9", "fail"), ("1.0", "fail"),
])
def test_kv_ratio_is_classified_on_a_percent_scale(metrics_text, ratio, expected) -> None:
"""0.9 of a KV cache is 90 %, not 0.9 % — classifying the raw ratio is a permanent green."""
text = re.sub(r"^(llamacpp:kv_cache_usage_ratio) \S+$", rf"\g<1> {ratio}",
metrics_text("gufo"), count=1, flags=re.M)
metrics = parse_gufo_metrics(text)
assert metrics["kv_cache_usage_ratio"]["severity"] == expected
assert metrics["kv_cache_usage_ratio"]["value"] == pytest.approx(float(ratio)), \
"the scale is for the verdict only; the published value is untouched"
def test_absent_kv_cell_gets_no_verdict() -> None:
metrics = parse_gufo_metrics("# nothing here\n")
assert metrics["kv_cache_usage_ratio"]["severity"] is None
# ---- vLLM ----------------------------------------------------------------- #
def test_vllm_counters_and_gauges(metrics_text) -> None:
@@ -112,6 +141,23 @@ def test_vllm_counters_and_gauges(metrics_text) -> None:
assert metrics["num_preemptions_total"]["value"] is not None
def test_vllm_kv_cache_cell_carries_a_severity(metrics_text) -> None:
assert parse_vllm_metrics(metrics_text("vllm"))["gpu_cache_usage"]["severity"] == "ok"
pressed = bump_metric(metrics_text("vllm"), "vllm:kv_cache_usage_perc", 0.95)
cell = parse_vllm_metrics(pressed)["gpu_cache_usage"]
assert cell["value"] == pytest.approx(0.95)
assert cell["severity"] == "fail"
assert cell["source"] == "vllm:kv_cache_usage_perc"
def test_vllm_absent_kv_cell_gets_no_verdict(metrics_text) -> None:
stripped = "\n".join(line for line in metrics_text("vllm").splitlines()
if "kv_cache_usage" not in line and "gpu_cache_usage" not in line)
metrics = parse_vllm_metrics(stripped)
assert metrics["gpu_cache_usage"]["value"] is None
assert metrics["gpu_cache_usage"]["severity"] is None
def test_vllm_latency_histograms(metrics_text) -> None:
metrics = parse_vllm_metrics(metrics_text("vllm"))
ttft = metrics["time_to_first_token_seconds"]