llm-monitor/AGENTS.md

# AGENTS.md — llm_monitor

Read-only real-time dashboard for **two LLM engines side by side**: a gufo server
(llama.cpp-family, `llamacpp:*` metrics + `/slots`) and a vLLM server (`vllm:*` metrics).
Either engine is optional — it appears only if its `*_BASE_URL` env var is set, so a vLLM-only
user gets a one-panel dashboard with no dead gufo half. FastAPI backend, one HTML page, vanilla
JS, inline-SVG sparklines. No build step.

## Commands

```bash
uv sync                                          # install (incl. dev group) -> .venv
uv run uvicorn app:app --host 127.0.0.1 --port 8090   # run — ALWAYS pass --port 8090
uv run python app.py                             # same, via DASHBOARD_HOST/DASHBOARD_PORT
uv run pytest                                    # 91 tests, no network, no engines needed
uv run pytest --cov                              # coverage gate: app.py should stay >=95%
uv run -p 3.10 pytest                            # requires-python is >=3.10; keep the floor
uv lock                                          # after editing pyproject.toml deps
uv export --no-dev --no-hashes > requirements.txt  # regenerate the pip shim
```

`uv run` auto-syncs, so you rarely need a bare `uv sync`. Use `uv run --no-dev sync` for a
production install. Never `pip install` into the system interpreter here.

## Non-negotiables

1. **Read-only. GET only.** The backend must only ever call `client.get(...)`. Do not add a
   POST/PUT/PATCH/DELETE route, do not add "clear cache"/"unload model" buttons, do not POST
   a test prompt to the engines. Tests enforce 405 on every non-GET verb for every route.
2. **Engines are opt-in, and ports come from the deployment repo, not from memory.** An
   engine exists **only** when its `GUFO_BASE_URL` / `VLLM_BASE_URL` is set: there is no
   in-code default host on purpose, because a built-in `127.0.0.1:8731` gives a vLLM-only user
   a permanently red panel for a server they do not run. Unset → `/api/gufo/*` answers
   `404 {"configured": false}` **without dialling**, and the page drops that section
   (panel, strip card, polling, fleet share); neither set → the setup card. The enabled state
   is derived from the base URL (`bool(GUFO_BASE_URL)`) and published as `gufo_enabled` /
   `vllm_enabled` — never add a separate `*_ENABLED` flag that could disagree with it.
   Source of truth for real endpoints: `~/Deployments/reeseapps/ai/deployments/*/quadlets/*.container`
   — today gufo is `8731` (`deskwork-ai`, `deskwork.reeselink.com`), vLLM radiance is `8000`
   (`driveripper-ai`, `driveripper.reeselink.com`). Re-check those files before documenting a
   port, and keep examples tunnel-friendly (`127.0.0.1`). Never make the dashboard bind
   `:8000` — that is an engine's port, and the dashboard would either fail to start or scrape itself.
3. **No CDN, no third-party assets, no bundler.** These engines sit on an internet-disconnected
   podman network; a CDN `<script>`/`@import` is a dead request and an offline blank chart.
   Sparklines stay hand-rolled inline SVG. A test asserts the page references no external URL.
4. **Per-engine independence.** Every engine is polled and health-tracked separately. One dead
   engine must never blank, block, or error the other, and must keep its last good values with
   a red dot + `stale Ns`. Keep the four polls fire-and-forget with the in-flight guard.
5. **Absent metric = `n/a`, never an exception.** A missing/renamed series yields
   `{"value": null, "source": null}` and the UI renders `n/a`. Never `raise` on a missing
   metric, never coerce absent to `0` (0 and unknown are different statements).
6. **One Prometheus parser.** `class Prom` in `app.py` is shared by both engines. Do not fork
   it, do not add a `prometheus_client` dependency. Engine-specific knowledge goes in the
   `GUFO_SERIES` / `VLLM_SERIES` / `VLLM_HISTOGRAMS` field maps only.
7. **Trust the engine, not the docs — including the docs in this repo.** The deployed servers
   do not export the names anyone expects: gufo uses `tokens_predicted_total` (not
   `tokens_generated_total`) and names its speed gauges `*_tokens_seconds` while the unit is
   tokens/second; vLLM uses `kv_cache_usage_perc` and `generation_tokens_total`, and reports
   TPOT as `inter_token_latency_seconds`. Before adding or renaming a series, scrape the real
   thing:

   ```bash
   curl -s http://deskwork.reeselink.com:8731/metrics | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
   curl -s http://driveripper.reeselink.com:8000/metrics | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
   ```

   `tests/fixtures/*_real.txt` are verbatim captures of both, and `tests/test_real_contracts.py`
   asserts every mapped field resolves against them. Re-capture those files when an engine is
   upgraded (and drop the old capture into the commit message so the rename is traceable).

## Where to change things

| Task                                              | File / spot                                              |
| ------------------------------------------------- | -------------------------------------------------------- |
| New/renamed upstream metric                       | `GUFO_SERIES` / `VLLM_SERIES` / `VLLM_HISTOGRAMS` (deployed name first, aliases after, right `agg`) + a case in `tests/test_real_contracts.py` |
| Parser behaviour (labels, aggregation, histogram) | `class Prom` + `_quantile`, with a test in `tests/test_parsing.py` |
| A new endpoint                                    | routes block at the bottom of `app.py`; add it to `API_PATHS` in `tests/test_proxy.py` so the 405 matrix covers it |
| Anything visual                                   | `static/index.html` only (CSS + JS live there)           |
| A new number in the UI                            | add the `id` to the `cacheEls()` list, then `setTxt`/`setBar` it |
| Fleet totals (cross-engine sum)                    | `renderFleet()` + `FLEET_FIELDS` in `static/index.html` — **never a new backend endpoint** |
| gufo per-request speed (not the fleet/aggregate)   | read `*_tokens_per_second_native`, don't re-point the aggregate cell |
| Config                                            | module-level `os.environ.get(...)` block in `app.py`     |
| "Does this engine exist" / hiding a section        | `bool(GUFO_BASE_URL)` + `not_configured()` in `app.py`; `applyEngines()` / `ENGINES` / `isEnabled()` and the `#no-engines` card in `static/index.html` |

Aggregation rules in the field maps are deliberate: `sum` for counters and queue gauges,
`max` for usage ratios (a summed ratio can exceed 1.0 and lie), `avg` for rate gauges, and
p95 suppressed when a histogram has more than one label set (blending label sets invents a
percentile). Preserve those, or change them consciously and say why in the commit.

## Rates: the precedence rule

When an engine exposes no throughput gauge (stock vLLM has **zero** rate series — verify with
`curl -s $VLLM_BASE_URL/metrics | grep -iE 'throughput|_per_second|rate'`), never leave the
cell `n/a` if the counters can answer. Precedence, in `parse_vllm_metrics`:

**native gauge → measured Δcounter/Δt → estimate → `n/a`**

One deliberate exception, and do not "restore" it: **both of gufo's throughput cells (decode AND
prompt) are measured Δ first, gauge second** (`parse_gufo_metrics`, via its local `measured()`
helper). Each speed is one unlabeled gauge holding one request's last decode/prefill pass, so it
under-reports whenever two slots are busy at once and goes stale-at-a-number when the box idles —
it cannot answer the aggregate question the cell asks. The token counters sum every slot, so Δ/Δt
wins and each gauge survives as `decode_tokens_per_second_native` /
`prompt_tokens_per_second_native` + the first-scrape fallback. The precedence above is right for
vLLM precisely because its native gauge *is* an aggregate.

Gufo's prompt counter **excludes** cache hits, so that cell measures newly-computed prefill work
(matching the gauge it replaced) while vLLM's prompt counter includes cached prefix tokens — the
fleet prompt sum leans slightly low for gufo. Keep counting per engine honestly and disclose the
seam; do not inflate gufo with cache hits just to make the addition neat.

- Measured Δ comes from `RateEstimator`, a rolling window over recent scrapes
  (`RATE_WINDOW_SECONDS`, default 10). It is the **only mutable state in the whole app** —
  keep it that way, and keep it injected into the parser
  (`parse_vllm_metrics(text, rates, now)` / `parse_gufo_metrics(text, rates, now)`)
  so the parser stays pure: `parse_gufo_metrics(text)` and `parse_vllm_metrics(text)` must be
  deterministic and stateless.
- One module-level `RATES` instance serves **both** engines. It is keyed by the *resolved*
  upstream series name (`llamacpp:tokens_predicted_total` vs `vllm:generation_tokens_total`),
  so the engines cannot cross-read samples and a renamed counter starts a fresh window instead
  of diffing two different series into a fabricated rate. Keep keying by resolved source, never
  by a hardcoded primary name, when adding a derived rate.
- A bare 1s delta jitters between 0 and spikes (prompts arrive in bursts); that is why the
  window exists. Don't "simplify" it back to a last-scrape diff.
- Counter reset (engine restart) must yield `None` for one scrape, never a negative rate.
- `window` is part of the cell and the UI renders it (`Δ10.0s`); any new derived rate must
  say so the same way. Lifetime average (`prefill_tokens_per_second_lifetime`) is a different
  claim from current throughput — label them distinctly, never swap one for the other.

## UI conventions

- Update the DOM **in place**. Slot cards are keyed by slot id in the `slotCards` Map and
  reused (`appendChild` reorders); no `innerHTML` rebuild per poll — it kills transitions
  and throws away focus.
- Write only through `setTxt` / `setBar` / `drawSpark` / `setThroughput`; they diff before
  writing. `setThroughput(el, cell)` is the only place that renders the provenance tag
  (`Δ10.0s`, `from TPOT`, `lifetime avg`) and puts the `computed:` source in the tooltip.
- Anything engine-shaped starts **hidden** in the markup and is revealed by `applyEngines()`
  from `/api/config`, so a single-engine deployment never flashes a panel it does not own.
  With fewer than two configured engines, `.panels`/`.strip` get `.single` (one column, no
  phantom gap) and `#fleet` is hidden.
- Both panels deliberately chart the **same pair**: `decode tok/s · all slots` + cache usage
  (`gufo-decode`/`gufo-kv`, `vllm-decode`/`vllm-kv`), each fed from that engine's
  `decode_tokens_per_second` so the two engines are comparable at a glance. Keep the pairing if
  you add a metric — vLLM's running count is intentionally not charted (banner + fleet total).
- **Fleet totals are summed in the browser** (`renderFleet`, called from `runJob`'s `finally` so
  failures re-sum too), over **the configured engines only** (`ENGINES` — an absent engine is
  not "down", it is not in the fleet), and include **only engines whose `/metrics` job is
  answering** — a stale
  engine's last good sample belongs in its own labelled panel, not in a total presented as
  current; the header then reads `1 of 2 engines` and the block goes amber. Liveness keys on the
  `/metrics` job alone, so gufo `/slots` failing must not drop gufo from the sums. Raw values are
  summed and rounded once, and each cell's tooltip itemises the parts.
- Sparkline y-axis labels live in a `.spark-axis` gutter **outside** the `<svg>`, as HTML.
  Never move them into the SVG: `viewBox="0 0 100 30"` + `preserveAspectRatio="none"` stretches
  the plot to the card width and would smear any text drawn in it. `drawSpark` labels exactly
  the `lo`/`hi` it plotted against (so a label can never drift from the curve), keeps the three
  levels distinguishable by adding decimals only when needed, and reports the value itself for
  all three when the window is **flat** — the zero-range `pad` is what keeps a flat line
  mid-plot, it is not a scale, and labelling it invents `±0.001` on an idle engine. Cache charts
  pass `fixedMax`, so their axis is pinned 0–1 and must read `100% / 50% / 0%`.
- Everything user-visible has an `n/a` path. The pause button and `Space` freeze polling for
  reading a snapshot — keep that working.

## Verifying a change

```bash
uv run pytest                                     # green
uv run uvicorn app:app --host 127.0.0.1 --port 8090 &
curl -s localhost:8090/api/config | python3 -m json.tool   # includes gufo_enabled / vllm_enabled
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8090/api/gufo/slots   # must be 405
```

Single-engine check (no real engines needed) — the unconfigured half must 404 **without**
making an upstream call, and the page must render only the configured half:

```bash
env -u GUFO_BASE_URL VLLM_BASE_URL=http://127.0.0.1:1 \
  uv run uvicorn app:app --host 127.0.0.1 --port 8090
# /api/config -> gufo_enabled:false, vllm_enabled:true; /api/gufo/* -> 404 configured:false;
# browser: #panel-gufo/#strip-gufo hidden, .panels has .single, #fleet hidden,
# tab title "LLM Monitor · vLLM"
```

Degradation check (works with no real engines): point one env var at a dead port and leave the
other at a live one —

```bash
GUFO_BASE_URL=http://127.0.0.1:8731 VLLM_BASE_URL=http://127.0.0.1:1 \
  uv run uvicorn app:app --host 127.0.0.1 --port 8090
```

Expected: gufo green and ticking, vLLM dot red, vLLM banner showing the upstream error, and
recovery on the next successful poll with no page reload.

## Gotchas

- **Bare `uvicorn app:app` binds 8000** (an engine's port). Always `--port 8090`.
- Config is read **once at import**. `--reload` watches code, not env; restart for env changes.
  Same for which engines exist — the env var cannot appear or disappear mid-run.
- "Not configured" is a 404, never a 502: a server nobody installed is not an outage, and the
  page must not light a red dot for it. The 404 matrix in `tests/test_proxy.py` asserts no
  upstream call happens.
- `httpx.Response(...)` in tests needs a `request=` before you read `.text`; it does **not**
  accept `reason_phrase` (it derives the phrase from the status code).
- Histograms never emit a bare `<name>` series — resolve them via `_bucket`/`_sum`/`_count`
  (`Prom.resolve_hist`), which is why `histogram()` does not use `resolve()`.
- gufo's `/slots` returns a **bare array**, not `{"slots": [...]}`; it also answers on the
  `/v1/slots` alias (the proxy tries both). Deferred requests are **not** in `/slots` — they
  only show up in `llamacpp:requests_deferred`.
- In a slot payload, `-1` means "none" (`id_task`, `n_remain`) — render `n/a`, never `-1`.
  `is_processing` is authoritative; numeric `state` is the fallback.
- One gufo slot can host several requests, so slot count ≠ `requests_processing`; don't
  "reconcile" the two numbers.
- gufo's `prompt_tokens_total` **excludes** cache hits (`prompt_tokens_cached_total` is
  separate, and on a warm box it dwarfs it — 33M vs 0.38M). Anything named "total" has to add
  both or it under-reports by orders of magnitude.
- Payloads carry `series` and `unparsed` counts. `unparsed > 0` or `series == 0` means the
  exposition format drifted — that is the drift signal, not an outage.
- `pytest` prints a `StarletteDeprecationWarning` about httpx/TestClient. It is upstream and
  harmless; do not "fix" it by swapping test clients.
- `requirements.txt` is **generated**. Edit `pyproject.toml`, then `uv lock` + re-export.
  `uv.lock` and `requirements.txt` are both committed on purpose.

## Definition of done

Tests green; both panels update live without a manual refresh; killing one engine flips only
that engine's dot red while the other keeps ticking; an **unconfigured** engine shows nothing at
all (no panel, no poll, no fleet share, `404 configured:false`) while the configured one keeps
ticking; every new number has an `n/a` path; no new dependency without a strong reason (this app
is 3 deps on purpose); README updated when behaviour, env vars, or ports change.