llm-monitor/AGENTS.md

AGENTS.md — llm_monitor

Read-only real-time dashboard for two LLM engines side by side: a gufo server (llama.cpp-family, llamacpp:* metrics + /slots) and a vLLM server (vllm:* metrics). Either engine is optional — it appears only if its *_BASE_URL env var is set, so a vLLM-only user gets a one-panel dashboard with no dead gufo half. FastAPI backend, one HTML page, vanilla JS, inline-SVG sparklines. No build step.

Commands

uv sync                                          # install (incl. dev group) -> .venv
uv run uvicorn app:app --host 127.0.0.1 --port 8090   # run — ALWAYS pass --port 8090
uv run python app.py                             # same, via DASHBOARD_HOST/DASHBOARD_PORT
uv run pytest                                    # 91 tests, no network, no engines needed
uv run pytest --cov                              # coverage gate: app.py should stay >=95%
uv run -p 3.10 pytest                            # requires-python is >=3.10; keep the floor
uv lock                                          # after editing pyproject.toml deps
uv export --no-dev --no-hashes > requirements.txt  # regenerate the pip shim

uv run auto-syncs, so you rarely need a bare uv sync. Use uv run --no-dev sync for a production install. Never pip install into the system interpreter here.

Non-negotiables

1. Read-only. GET only. The backend must only ever call client.get(...). Do not add a POST/PUT/PATCH/DELETE route, do not add "clear cache"/"unload model" buttons, do not POST a test prompt to the engines. Tests enforce 405 on every non-GET verb for every route.

2. Engines are opt-in, and ports come from the deployment repo, not from memory. An engine exists only when its GUFO_BASE_URL / VLLM_BASE_URL is set: there is no in-code default host on purpose, because a built-in 127.0.0.1:8731 gives a vLLM-only user a permanently red panel for a server they do not run. Unset → /api/gufo/* answers 404 {"configured": false} without dialling, and the page drops that section (panel, strip card, polling, fleet share); neither set → the setup card. The enabled state is derived from the base URL (bool(GUFO_BASE_URL)) and published as gufo_enabled / vllm_enabled — never add a separate *_ENABLED flag that could disagree with it. Source of truth for real endpoints: ~/Deployments/reeseapps/ai/deployments/*/quadlets/*.container — today gufo is 8731 (deskwork-ai, deskwork.reeselink.com), vLLM radiance is 8000 (driveripper-ai, driveripper.reeselink.com). Re-check those files before documenting a port, and keep examples tunnel-friendly (127.0.0.1). Never make the dashboard bind :8000 — that is an engine's port, and the dashboard would either fail to start or scrape itself.

3. No CDN, no third-party assets, no bundler. These engines sit on an internet-disconnected podman network; a CDN <script>/@import is a dead request and an offline blank chart. Sparklines stay hand-rolled inline SVG. A test asserts the page references no external URL.

4. Per-engine independence. Every engine is polled and health-tracked separately. One dead engine must never blank, block, or error the other, and must keep its last good values with a red dot + stale Ns. Keep the four polls fire-and-forget with the in-flight guard.

5. Absent metric = n/a, never an exception. A missing/renamed series yields {"value": null, "source": null} and the UI renders n/a. Never raise on a missing metric, never coerce absent to 0 (0 and unknown are different statements).

6. One Prometheus parser. class Prom in app.py is shared by both engines. Do not fork it, do not add a prometheus_client dependency. Engine-specific knowledge goes in the GUFO_SERIES / VLLM_SERIES / VLLM_HISTOGRAMS field maps only.

7. Trust the engine, not the docs — including the docs in this repo. The deployed servers do not export the names anyone expects: gufo uses tokens_predicted_total (not tokens_generated_total) and names its speed gauges *_tokens_seconds while the unit is tokens/second; vLLM uses kv_cache_usage_perc and generation_tokens_total, and reports TPOT as inter_token_latency_seconds. Before adding or renaming a series, scrape the real thing:

   curl -s http://deskwork.reeselink.com:8731/metrics | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
   curl -s http://driveripper.reeselink.com:8000/metrics | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
   

   tests/fixtures/*_real.txt are verbatim captures of both, and tests/test_real_contracts.py asserts every mapped field resolves against them. Re-capture those files when an engine is upgraded (and drop the old capture into the commit message so the rename is traceable).

Where to change things

Task	File / spot
New/renamed upstream metric	GUFO_SERIES / VLLM_SERIES / VLLM_HISTOGRAMS (deployed name first, aliases after, right agg) + a case in tests/test_real_contracts.py
Parser behaviour (labels, aggregation, histogram)	class Prom + _quantile, with a test in tests/test_parsing.py
A new endpoint	routes block at the bottom of app.py; add it to API_PATHS in tests/test_proxy.py so the 405 matrix covers it
Anything visual	static/index.html only (CSS + JS live there)
A new number in the UI	add the id to the cacheEls() list, then setTxt/setBar it
Fleet totals (cross-engine sum)	renderFleet() + FLEET_FIELDS in static/index.html — never a new backend endpoint
gufo per-request speed (not the fleet/aggregate)	read *_tokens_per_second_native, don't re-point the aggregate cell
Config	module-level os.environ.get(...) block in app.py
"Does this engine exist" / hiding a section	bool(GUFO_BASE_URL) + not_configured() in app.py; applyEngines() / ENGINES / isEnabled() and the #no-engines card in static/index.html

Aggregation rules in the field maps are deliberate: sum for counters and queue gauges, max for usage ratios (a summed ratio can exceed 1.0 and lie), avg for rate gauges, and p95 suppressed when a histogram has more than one label set (blending label sets invents a percentile). Preserve those, or change them consciously and say why in the commit.

Rates: the precedence rule

When an engine exposes no throughput gauge (stock vLLM has zero rate series — verify with curl -s $VLLM_BASE_URL/metrics | grep -iE 'throughput|_per_second|rate'), never leave the cell n/a if the counters can answer. Precedence, in parse_vllm_metrics:

native gauge → measured Δcounter/Δt → estimate → n/a

One deliberate exception, and do not "restore" it: both of gufo's throughput cells (decode AND prompt) are measured Δ first, gauge second (parse_gufo_metrics, via its local measured() helper). Each speed is one unlabeled gauge holding one request's last decode/prefill pass, so it under-reports whenever two slots are busy at once and goes stale-at-a-number when the box idles — it cannot answer the aggregate question the cell asks. The token counters sum every slot, so Δ/Δt wins and each gauge survives as decode_tokens_per_second_native / prompt_tokens_per_second_native + the first-scrape fallback. The precedence above is right for vLLM precisely because its native gauge is an aggregate.

Gufo's prompt counter excludes cache hits, so that cell measures newly-computed prefill work (matching the gauge it replaced) while vLLM's prompt counter includes cached prefix tokens — the fleet prompt sum leans slightly low for gufo. Keep counting per engine honestly and disclose the seam; do not inflate gufo with cache hits just to make the addition neat.

• Measured Δ comes from RateEstimator, a rolling window over recent scrapes (RATE_WINDOW_SECONDS, default 10). It is the only mutable state in the whole app — keep it that way, and keep it injected into the parser (parse_vllm_metrics(text, rates, now) / parse_gufo_metrics(text, rates, now)) so the parser stays pure: parse_gufo_metrics(text) and parse_vllm_metrics(text) must be deterministic and stateless.
• One module-level RATES instance serves both engines. It is keyed by the resolved upstream series name (llamacpp:tokens_predicted_total vs vllm:generation_tokens_total), so the engines cannot cross-read samples and a renamed counter starts a fresh window instead of diffing two different series into a fabricated rate. Keep keying by resolved source, never by a hardcoded primary name, when adding a derived rate.
• A bare 1s delta jitters between 0 and spikes (prompts arrive in bursts); that is why the window exists. Don't "simplify" it back to a last-scrape diff.
• Counter reset (engine restart) must yield None for one scrape, never a negative rate.
• window is part of the cell and the UI renders it (Δ10.0s); any new derived rate must say so the same way. Lifetime average (prefill_tokens_per_second_lifetime) is a different claim from current throughput — label them distinctly, never swap one for the other.

UI conventions

• Update the DOM in place. Slot cards are keyed by slot id in the slotCards Map and reused (appendChild reorders); no innerHTML rebuild per poll — it kills transitions and throws away focus.
• Write only through setTxt / setBar / drawSpark / setThroughput; they diff before writing. setThroughput(el, cell) is the only place that renders the provenance tag (Δ10.0s, from TPOT, lifetime avg) and puts the computed: source in the tooltip.
• Anything engine-shaped starts hidden in the markup and is revealed by applyEngines() from /api/config, so a single-engine deployment never flashes a panel it does not own. With fewer than two configured engines, .panels/.strip get .single (one column, no phantom gap) and #fleet is hidden.
• Both panels deliberately chart the same pair: decode tok/s · all slots + cache usage (gufo-decode/gufo-kv, vllm-decode/vllm-kv), each fed from that engine's decode_tokens_per_second so the two engines are comparable at a glance. Keep the pairing if you add a metric — vLLM's running count is intentionally not charted (banner + fleet total).
• Fleet totals are summed in the browser (renderFleet, called from runJob's finally so failures re-sum too), over the configured engines only (ENGINES — an absent engine is not "down", it is not in the fleet), and include only engines whose /metrics job is answering — a stale engine's last good sample belongs in its own labelled panel, not in a total presented as current; the header then reads 1 of 2 engines and the block goes amber. Liveness keys on the /metrics job alone, so gufo /slots failing must not drop gufo from the sums. Raw values are summed and rounded once, and each cell's tooltip itemises the parts.
• Sparkline y-axis labels live in a .spark-axis gutter outside the <svg>, as HTML. Never move them into the SVG: viewBox="0 0 100 30" + preserveAspectRatio="none" stretches the plot to the card width and would smear any text drawn in it. drawSpark labels exactly the lo/hi it plotted against (so a label can never drift from the curve), keeps the three levels distinguishable by adding decimals only when needed, and reports the value itself for all three when the window is flat — the zero-range pad is what keeps a flat line mid-plot, it is not a scale, and labelling it invents ±0.001 on an idle engine. Cache charts pass fixedMax, so their axis is pinned 0–1 and must read 100% / 50% / 0%.
• Everything user-visible has an n/a path. The pause button and Space freeze polling for reading a snapshot — keep that working.

Verifying a change

uv run pytest                                     # green
uv run uvicorn app:app --host 127.0.0.1 --port 8090 &
curl -s localhost:8090/api/config | python3 -m json.tool   # includes gufo_enabled / vllm_enabled
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8090/api/gufo/slots   # must be 405

Single-engine check (no real engines needed) — the unconfigured half must 404 without making an upstream call, and the page must render only the configured half:

env -u GUFO_BASE_URL VLLM_BASE_URL=http://127.0.0.1:1 \
  uv run uvicorn app:app --host 127.0.0.1 --port 8090
# /api/config -> gufo_enabled:false, vllm_enabled:true; /api/gufo/* -> 404 configured:false;
# browser: #panel-gufo/#strip-gufo hidden, .panels has .single, #fleet hidden,
# tab title "LLM Monitor · vLLM"

Degradation check (works with no real engines): point one env var at a dead port and leave the other at a live one —

GUFO_BASE_URL=http://127.0.0.1:8731 VLLM_BASE_URL=http://127.0.0.1:1 \
  uv run uvicorn app:app --host 127.0.0.1 --port 8090

Expected: gufo green and ticking, vLLM dot red, vLLM banner showing the upstream error, and recovery on the next successful poll with no page reload.

Gotchas

• Bare uvicorn app:app binds 8000 (an engine's port). Always --port 8090.
• Config is read once at import. --reload watches code, not env; restart for env changes. Same for which engines exist — the env var cannot appear or disappear mid-run.
• "Not configured" is a 404, never a 502: a server nobody installed is not an outage, and the page must not light a red dot for it. The 404 matrix in tests/test_proxy.py asserts no upstream call happens.
• httpx.Response(...) in tests needs a request= before you read .text; it does not accept reason_phrase (it derives the phrase from the status code).
• Histograms never emit a bare <name> series — resolve them via _bucket/_sum/_count (Prom.resolve_hist), which is why histogram() does not use resolve().
• gufo's /slots returns a bare array, not {"slots": [...]}; it also answers on the /v1/slots alias (the proxy tries both). Deferred requests are not in /slots — they only show up in llamacpp:requests_deferred.
• In a slot payload, -1 means "none" (id_task, n_remain) — render n/a, never -1. is_processing is authoritative; numeric state is the fallback.
• One gufo slot can host several requests, so slot count ≠ requests_processing; don't "reconcile" the two numbers.
• gufo's prompt_tokens_total excludes cache hits (prompt_tokens_cached_total is separate, and on a warm box it dwarfs it — 33M vs 0.38M). Anything named "total" has to add both or it under-reports by orders of magnitude.
• Payloads carry series and unparsed counts. unparsed > 0 or series == 0 means the exposition format drifted — that is the drift signal, not an outage.
• pytest prints a StarletteDeprecationWarning about httpx/TestClient. It is upstream and harmless; do not "fix" it by swapping test clients.
• requirements.txt is generated. Edit pyproject.toml, then uv lock + re-export. uv.lock and requirements.txt are both committed on purpose.

Definition of done

Tests green; both panels update live without a manual refresh; killing one engine flips only that engine's dot red while the other keeps ticking; an unconfigured engine shows nothing at all (no panel, no poll, no fleet share, 404 configured:false) while the configured one keeps ticking; every new number has an n/a path; no new dependency without a strong reason (this app is 3 deps on purpose); README updated when behaviour, env vars, or ports change.