ducoterra
fefbeb8227
**Phase 09 — final verification pass (all 4 task files already in `complete/`; nothing re-implemented)**
- Verified `GET /api/disks/snapshot` live: 3 hosts / 18 whole-`/sys/block` rows, `smartctl_available` three-way, per-row `severity`, API-summed `fleet`; 404 `{"configured": false}` proven *without dialling* (spies on `httpx.AsyncClient.get` + `collectors.fetch_snapshot`).
- Verified `/api/disks/history` live (`disk.temp_c@dev`, unknown names → empty series) + `test_store.py`'s 300 s rollup/prune buckets; `/api/metrics?source=disks` answers the `disks` family.
- Browser against live data: 18 drive cards (never a partition/mount), collapsed per-host `Charts` with `aria-label`, 3 `.stale` grids + one `.banner.info` why-line and **0 `.fail` classes / green `disks` dot** → criteria 3 and 4 hold; new shots `data/logs/09-verify-disks-live-dimmed.png`, `09-verify-overview-live.png` (13 total for the phase).
- Real-hardware criterion: read 4 driveripper drives through `ops`' `sudo -n smartctl -i -A -H` and ran the live output through `host_agent.parse_smart_output` → `sources/disks.parse_disks_snapshot`: `sda` 35 °C / 30 423 h, `nvme1n1` 58 % wear, `nvme5n1` 13 uncorrectable → `alarm_disks: 1`, temps graded `warn`; consistent with the 16 committed fixtures.
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov` → **769 passed, 11 skipped, coverage 98.33%** (floor 94); `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene + dependency advisories clean. `uv run pytest -m live --live` → 10 passed, 1 skipped (agents still predate the SMART pass). Dev server healthy: `/api/health` `ok`, 9/9 sources.
- Docs fixed: AGENTS.md test count 698→769 and coverage 98.2→98.33 %; `disks` added to the event dashboard-name list and to the "every feed renders `Dash.eventText`" sentence; README `POLL_MS_*` row now states `/disks`' deliberate fixed 30 s; new Known gap 4 records the dev-DB `database is locked` writes (reported/recovered per invariant 4, ~590 MB DB / ~70 MB WAL, not reproduced by a controlled 25 s external read txn).
- Deviations: **no live-infra change** — granting the agent user the `disk` group and redeploying `host_agent.py` on driveripper/deskwork stay the documented paired operator action (redeploying without the grant yields 16 permission-denied lines; a sudo-spawning agent would break the confirmed "direct read, no sudo" assumption). Store-lock issue left to its own phase rather than patched speculatively.
- Next pending phase: `10_dev_direct_dial`.
151 lines
7.4 KiB
HTML
151 lines
7.4 KiB
HTML
<!doctype html>
|
|
<html lang="en" data-theme="dark">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
<title>Homelab · LLM</title>
|
|
<link rel="stylesheet" href="/static/dash.css">
|
|
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100'><rect width='100' height='100' rx='22' fill='%23ff8a3d'/></svg>">
|
|
</head>
|
|
<body>
|
|
<header class="masthead">
|
|
<a class="brand" href="/"><span class="brand-mark">◆</span> Homelab</a>
|
|
<nav class="tabs" aria-label="Dashboards">
|
|
<a href="/">Overview</a>
|
|
<a href="/system">System</a>
|
|
<a href="/disks">Disks</a>
|
|
<a href="/llm" aria-current="page">LLM</a>
|
|
<a href="/ups">UPS</a>
|
|
</nav>
|
|
<div class="masthead-right">
|
|
<div id="nav-status" class="gap" aria-live="polite"></div>
|
|
<span class="clock" id="clock">--:--:--</span>
|
|
<button class="ghost" id="pause" type="button" aria-pressed="false">⏸ Pause</button>
|
|
<button class="ghost" id="theme-toggle" type="button" aria-pressed="false">☾ Dark</button>
|
|
</div>
|
|
</header>
|
|
|
|
<main>
|
|
<div class="page-head">
|
|
<div>
|
|
<h1>LLM engines</h1>
|
|
<div class="sub" id="sub">reading engines…</div>
|
|
</div>
|
|
<div class="spacer"></div>
|
|
<div class="gap"><span class="faint">window</span><div id="range"></div></div>
|
|
</div>
|
|
|
|
<div id="banners" class="stack" style="margin-bottom:.85rem"></div>
|
|
|
|
<!-- The aggregate block, in llm_monitor's shape: the sums no single engine panel can
|
|
say, above the panels. Hidden until a second engine exists — a cross-engine sum of
|
|
one engine only duplicates that engine's own panel.
|
|
|
|
Every number here is `fleet_totals()` over the engines that answered *this poll*: a
|
|
dead engine keeps its last-good values inside its own card (correct there, because
|
|
that card is labelled stale) but is dropped from a total presented as current, and
|
|
the tag above says how many engines the sum covers. No number on this block carries
|
|
a verdict — `severity.py` has no throughput or queue-depth row, because 0 tok/s on
|
|
an idle fleet is healthy — so nothing here is coloured, and the KV bar that used to
|
|
sit beside these sums left with phase 04 for exactly that reason. -->
|
|
<section class="card flat fleet" id="fleet" aria-label="Fleet totals" hidden>
|
|
<div class="body">
|
|
<div class="fleet-head">
|
|
<span class="dot idle" id="fleet-dot"></span>
|
|
<span class="fleet-name">all engines</span>
|
|
<span class="chip" id="fleet-count">no engines yet</span>
|
|
<span class="fleet-note">sums only engines answering now</span>
|
|
</div>
|
|
<div class="metrics">
|
|
<div class="metric" title="Requests generating tokens across every answering engine.">
|
|
<div class="label">running</div>
|
|
<div class="value"><span id="f-running">n/a</span><span class="unit"></span></div>
|
|
</div>
|
|
<div class="metric" title="Requests queued behind them across every answering engine.">
|
|
<div class="label">waiting</div>
|
|
<div class="value"><span id="f-waiting">n/a</span><span class="unit"></span></div>
|
|
</div>
|
|
<div class="metric">
|
|
<div class="label">prompt tok/s</div>
|
|
<div class="value"><span id="f-prompt">n/a</span><span class="unit">tok/s</span></div>
|
|
</div>
|
|
<div class="metric" title="Every busy slot on every answering engine: tokens generated per second.">
|
|
<div class="label">decode tok/s</div>
|
|
<div class="value"><span id="f-decode">n/a</span><span class="unit">tok/s</span></div>
|
|
</div>
|
|
<div class="metric" title="Tokens counted since each answering engine started.">
|
|
<div class="label">total tokens</div>
|
|
<div class="value"><span id="f-tokens">n/a</span><span class="unit"></span></div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</section>
|
|
|
|
<!-- The per-engine stats strip (phase 07 task 03, `llm_monitor`'s shared strip): one card
|
|
per **configured** engine, filled by `page-llm.js` from `/api/llm/snapshot` — the
|
|
KV-cache bar (the one cell on this page with a verdict, coloured from the payload), the
|
|
two throughputs with their provenance tags, lifetime tokens, the sessions this engine
|
|
reports, its own extra rows, and alarm chips that exist only while non-zero.
|
|
|
|
It ships empty and `hidden`: an engine that is not configured gets no card, and a
|
|
single-engine deployment gets one card across the page instead of a red half (the
|
|
fleet block above simply stays hidden below two engines). No value is printed here and
|
|
in the panels below — the rows that moved in were deleted from the panel cell list in
|
|
the same change. -->
|
|
<section class="strip" id="strip" aria-label="Engine summary" hidden></section>
|
|
|
|
<!-- Then the engine panels: two throughput heroes per engine and the detail the strip
|
|
does not print (the latency means with their p95 feet and window sparklines). The four
|
|
trend charts follow, half-height — supporting evidence, not the main event. The gufo
|
|
slot table that used to sit beside the events panel was deleted in phase 07: it read
|
|
field names (`slot.state`, `slot.n_prompt_tokens`, `slot.decode_per_second`) that
|
|
gufo's live `/slots` never publishes, so it could only ever render dashes. Its
|
|
replacement is built from the shape pinned in `tests/fixtures/slots_gufo.json`
|
|
(`tests/slots_contract.py`). -->
|
|
<section style="margin-top:.85rem">
|
|
<div class="grid cols-2 items-start" id="engines"></div>
|
|
<div class="empty-state" id="engines-empty" hidden>
|
|
<h3>No engines configured</h3>
|
|
<p>Set <code>GUFO_BASE_URL</code> (llama.cpp / gufo) and/or <code>VLLM_BASE_URL</code>.</p>
|
|
</div>
|
|
</section>
|
|
|
|
<section class="grid cols-2" style="margin-top:.85rem" aria-label="Trend charts">
|
|
<article class="card">
|
|
<header><h3>Decode throughput</h3><span class="spacer"></span>
|
|
<span class="faint">tok/s</span></header>
|
|
<div class="body"><div class="chart" id="chart-decode"></div><div class="legend" id="legend-decode"></div></div>
|
|
</article>
|
|
<article class="card">
|
|
<header><h3>Prompt / prefill throughput</h3><span class="spacer"></span>
|
|
<span class="faint">tok/s</span></header>
|
|
<div class="body"><div class="chart" id="chart-prompt"></div><div class="legend" id="legend-prompt"></div></div>
|
|
</article>
|
|
<article class="card">
|
|
<header><h3>KV / prefix cache pressure</h3><span class="spacer"></span><span class="faint">% in use</span></header>
|
|
<div class="body"><div class="chart" id="chart-kv"></div><div class="legend" id="legend-kv"></div></div>
|
|
</article>
|
|
<article class="card">
|
|
<header><h3>Sessions & queue</h3><span class="spacer"></span><span class="faint">running vs waiting</span></header>
|
|
<div class="body"><div class="chart" id="chart-queue"></div><div class="legend" id="legend-queue"></div></div>
|
|
</article>
|
|
</section>
|
|
|
|
<section style="margin-top:.85rem">
|
|
<article class="card">
|
|
<header><h3>Events</h3></header>
|
|
<div class="body"><ul class="events" id="events"><li class="muted">loading…</li></ul></div>
|
|
</article>
|
|
</section>
|
|
|
|
<footer class="pagefoot">
|
|
<span class="mono" id="foot-note"></span>
|
|
</footer>
|
|
</main>
|
|
|
|
<script src="/static/charts.js"></script>
|
|
<script src="/static/app.js"></script>
|
|
<script src="/static/page-llm.js"></script>
|
|
</body>
|
|
</html>
|