initial import: read-only gufo + vLLM dashboard
Build and Push Container / build-and-push (push) Successful in 38s
Build and Push Container / build-and-push (push) Successful in 38s
Single-file FastAPI proxy plus a no-CDN vanilla-JS page for watching a gufo
server and a vLLM server side by side: one Prometheus parser (class Prom) with
per-engine field maps, measured Δcounter/Δt rates for engines that export no
rate gauge, and strict read-only GET-only routing. Engines are opt-in via
*_BASE_URL — unset means no panel, no poll, and a 404 {"configured": false}
without dialling.
Includes the container image and CI so the dashboard can be deployed as an
image rather than a checkout:
- Containerfile: uv-locked two-stage build on python:3.14-slim, non-root
(uid 10001), binds 8090 (8000 is vLLM's port in this fleet), healthcheck on
/api/config so a dead engine never marks the dashboard unhealthy. No frontend
build stage — static/index.html is bundler-free by design and is copied as-is.
- .gitea/workflows/build-and-push.yml: publishes
gitea.reeseapps.com/Experimental/llm-monitor on main + release tags.
114 tests, no network and no real engines required.
LLM-Generated: true
This commit is contained in:
@@ -0,0 +1,41 @@
|
||||
name: Build and Push Container
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
release:
|
||||
types:
|
||||
- published
|
||||
|
||||
jobs:
|
||||
build-and-push:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Login to Gitea Container Registry
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
registry: gitea.reeseapps.com
|
||||
username: ${{ secrets.REGISTRY_USERNAME }}
|
||||
password: ${{ secrets.REGISTRY_PASSWORD }}
|
||||
|
||||
- name: Extract metadata
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: gitea.reeseapps.com/Experimental/llm-monitor
|
||||
tags: |
|
||||
type=raw,value=main,enable=${{ github.ref == 'refs/heads/main' }}
|
||||
type=ref,event=tag
|
||||
|
||||
- name: Build and push
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: Containerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
# uv / python
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
.pytest_cache/
|
||||
.ruff_cache/
|
||||
.mypy_cache/
|
||||
.coverage
|
||||
coverage.xml
|
||||
htmlcov/
|
||||
# commit uv.lock and requirements.txt — this is a deployable app, not a library
|
||||
.env
|
||||
.env.*
|
||||
!.env.example
|
||||
|
||||
# logs & misc
|
||||
*.log
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
@@ -0,0 +1,231 @@
|
||||
# AGENTS.md — llm_monitor
|
||||
|
||||
Read-only real-time dashboard for **two LLM engines side by side**: a gufo server
|
||||
(llama.cpp-family, `llamacpp:*` metrics + `/slots`) and a vLLM server (`vllm:*` metrics).
|
||||
Either engine is optional — it appears only if its `*_BASE_URL` env var is set, so a vLLM-only
|
||||
user gets a one-panel dashboard with no dead gufo half. FastAPI backend, one HTML page, vanilla
|
||||
JS, inline-SVG sparklines. No build step.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
uv sync # install (incl. dev group) -> .venv
|
||||
uv run uvicorn app:app --host 127.0.0.1 --port 8090 # run — ALWAYS pass --port 8090
|
||||
uv run python app.py # same, via DASHBOARD_HOST/DASHBOARD_PORT
|
||||
uv run pytest # 91 tests, no network, no engines needed
|
||||
uv run pytest --cov # coverage gate: app.py should stay >=95%
|
||||
uv run -p 3.10 pytest # requires-python is >=3.10; keep the floor
|
||||
uv lock # after editing pyproject.toml deps
|
||||
uv export --no-dev --no-hashes > requirements.txt # regenerate the pip shim
|
||||
```
|
||||
|
||||
`uv run` auto-syncs, so you rarely need a bare `uv sync`. Use `uv run --no-dev sync` for a
|
||||
production install. Never `pip install` into the system interpreter here.
|
||||
|
||||
## Non-negotiables
|
||||
|
||||
1. **Read-only. GET only.** The backend must only ever call `client.get(...)`. Do not add a
|
||||
POST/PUT/PATCH/DELETE route, do not add "clear cache"/"unload model" buttons, do not POST
|
||||
a test prompt to the engines. Tests enforce 405 on every non-GET verb for every route.
|
||||
2. **Engines are opt-in, and ports come from the deployment repo, not from memory.** An
|
||||
engine exists **only** when its `GUFO_BASE_URL` / `VLLM_BASE_URL` is set: there is no
|
||||
in-code default host on purpose, because a built-in `127.0.0.1:8731` gives a vLLM-only user
|
||||
a permanently red panel for a server they do not run. Unset → `/api/gufo/*` answers
|
||||
`404 {"configured": false}` **without dialling**, and the page drops that section
|
||||
(panel, strip card, polling, fleet share); neither set → the setup card. The enabled state
|
||||
is derived from the base URL (`bool(GUFO_BASE_URL)`) and published as `gufo_enabled` /
|
||||
`vllm_enabled` — never add a separate `*_ENABLED` flag that could disagree with it.
|
||||
Source of truth for real endpoints: `~/Deployments/reeseapps/ai/deployments/*/quadlets/*.container`
|
||||
— today gufo is `8731` (`deskwork-ai`, `deskwork.reeselink.com`), vLLM radiance is `8000`
|
||||
(`driveripper-ai`, `driveripper.reeselink.com`). Re-check those files before documenting a
|
||||
port, and keep examples tunnel-friendly (`127.0.0.1`). Never make the dashboard bind
|
||||
`:8000` — that is an engine's port, and the dashboard would either fail to start or scrape itself.
|
||||
3. **No CDN, no third-party assets, no bundler.** These engines sit on an internet-disconnected
|
||||
podman network; a CDN `<script>`/`@import` is a dead request and an offline blank chart.
|
||||
Sparklines stay hand-rolled inline SVG. A test asserts the page references no external URL.
|
||||
4. **Per-engine independence.** Every engine is polled and health-tracked separately. One dead
|
||||
engine must never blank, block, or error the other, and must keep its last good values with
|
||||
a red dot + `stale Ns`. Keep the four polls fire-and-forget with the in-flight guard.
|
||||
5. **Absent metric = `n/a`, never an exception.** A missing/renamed series yields
|
||||
`{"value": null, "source": null}` and the UI renders `n/a`. Never `raise` on a missing
|
||||
metric, never coerce absent to `0` (0 and unknown are different statements).
|
||||
6. **One Prometheus parser.** `class Prom` in `app.py` is shared by both engines. Do not fork
|
||||
it, do not add a `prometheus_client` dependency. Engine-specific knowledge goes in the
|
||||
`GUFO_SERIES` / `VLLM_SERIES` / `VLLM_HISTOGRAMS` field maps only.
|
||||
7. **Trust the engine, not the docs — including the docs in this repo.** The deployed servers
|
||||
do not export the names anyone expects: gufo uses `tokens_predicted_total` (not
|
||||
`tokens_generated_total`) and names its speed gauges `*_tokens_seconds` while the unit is
|
||||
tokens/second; vLLM uses `kv_cache_usage_perc` and `generation_tokens_total`, and reports
|
||||
TPOT as `inter_token_latency_seconds`. Before adding or renaming a series, scrape the real
|
||||
thing:
|
||||
|
||||
```bash
|
||||
curl -s http://deskwork.reeselink.com:8731/metrics | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
|
||||
curl -s http://driveripper.reeselink.com:8000/metrics | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
|
||||
```
|
||||
|
||||
`tests/fixtures/*_real.txt` are verbatim captures of both, and `tests/test_real_contracts.py`
|
||||
asserts every mapped field resolves against them. Re-capture those files when an engine is
|
||||
upgraded (and drop the old capture into the commit message so the rename is traceable).
|
||||
|
||||
## Where to change things
|
||||
|
||||
| Task | File / spot |
|
||||
| ------------------------------------------------- | -------------------------------------------------------- |
|
||||
| New/renamed upstream metric | `GUFO_SERIES` / `VLLM_SERIES` / `VLLM_HISTOGRAMS` (deployed name first, aliases after, right `agg`) + a case in `tests/test_real_contracts.py` |
|
||||
| Parser behaviour (labels, aggregation, histogram) | `class Prom` + `_quantile`, with a test in `tests/test_parsing.py` |
|
||||
| A new endpoint | routes block at the bottom of `app.py`; add it to `API_PATHS` in `tests/test_proxy.py` so the 405 matrix covers it |
|
||||
| Anything visual | `static/index.html` only (CSS + JS live there) |
|
||||
| A new number in the UI | add the `id` to the `cacheEls()` list, then `setTxt`/`setBar` it |
|
||||
| Fleet totals (cross-engine sum) | `renderFleet()` + `FLEET_FIELDS` in `static/index.html` — **never a new backend endpoint** |
|
||||
| gufo per-request speed (not the fleet/aggregate) | read `*_tokens_per_second_native`, don't re-point the aggregate cell |
|
||||
| Config | module-level `os.environ.get(...)` block in `app.py` |
|
||||
| "Does this engine exist" / hiding a section | `bool(GUFO_BASE_URL)` + `not_configured()` in `app.py`; `applyEngines()` / `ENGINES` / `isEnabled()` and the `#no-engines` card in `static/index.html` |
|
||||
|
||||
Aggregation rules in the field maps are deliberate: `sum` for counters and queue gauges,
|
||||
`max` for usage ratios (a summed ratio can exceed 1.0 and lie), `avg` for rate gauges, and
|
||||
p95 suppressed when a histogram has more than one label set (blending label sets invents a
|
||||
percentile). Preserve those, or change them consciously and say why in the commit.
|
||||
|
||||
## Rates: the precedence rule
|
||||
|
||||
When an engine exposes no throughput gauge (stock vLLM has **zero** rate series — verify with
|
||||
`curl -s $VLLM_BASE_URL/metrics | grep -iE 'throughput|_per_second|rate'`), never leave the
|
||||
cell `n/a` if the counters can answer. Precedence, in `parse_vllm_metrics`:
|
||||
|
||||
**native gauge → measured Δcounter/Δt → estimate → `n/a`**
|
||||
|
||||
One deliberate exception, and do not "restore" it: **both of gufo's throughput cells (decode AND
|
||||
prompt) are measured Δ first, gauge second** (`parse_gufo_metrics`, via its local `measured()`
|
||||
helper). Each speed is one unlabeled gauge holding one request's last decode/prefill pass, so it
|
||||
under-reports whenever two slots are busy at once and goes stale-at-a-number when the box idles —
|
||||
it cannot answer the aggregate question the cell asks. The token counters sum every slot, so Δ/Δt
|
||||
wins and each gauge survives as `decode_tokens_per_second_native` /
|
||||
`prompt_tokens_per_second_native` + the first-scrape fallback. The precedence above is right for
|
||||
vLLM precisely because its native gauge *is* an aggregate.
|
||||
|
||||
Gufo's prompt counter **excludes** cache hits, so that cell measures newly-computed prefill work
|
||||
(matching the gauge it replaced) while vLLM's prompt counter includes cached prefix tokens — the
|
||||
fleet prompt sum leans slightly low for gufo. Keep counting per engine honestly and disclose the
|
||||
seam; do not inflate gufo with cache hits just to make the addition neat.
|
||||
|
||||
- Measured Δ comes from `RateEstimator`, a rolling window over recent scrapes
|
||||
(`RATE_WINDOW_SECONDS`, default 10). It is the **only mutable state in the whole app** —
|
||||
keep it that way, and keep it injected into the parser
|
||||
(`parse_vllm_metrics(text, rates, now)` / `parse_gufo_metrics(text, rates, now)`)
|
||||
so the parser stays pure: `parse_gufo_metrics(text)` and `parse_vllm_metrics(text)` must be
|
||||
deterministic and stateless.
|
||||
- One module-level `RATES` instance serves **both** engines. It is keyed by the *resolved*
|
||||
upstream series name (`llamacpp:tokens_predicted_total` vs `vllm:generation_tokens_total`),
|
||||
so the engines cannot cross-read samples and a renamed counter starts a fresh window instead
|
||||
of diffing two different series into a fabricated rate. Keep keying by resolved source, never
|
||||
by a hardcoded primary name, when adding a derived rate.
|
||||
- A bare 1s delta jitters between 0 and spikes (prompts arrive in bursts); that is why the
|
||||
window exists. Don't "simplify" it back to a last-scrape diff.
|
||||
- Counter reset (engine restart) must yield `None` for one scrape, never a negative rate.
|
||||
- `window` is part of the cell and the UI renders it (`Δ10.0s`); any new derived rate must
|
||||
say so the same way. Lifetime average (`prefill_tokens_per_second_lifetime`) is a different
|
||||
claim from current throughput — label them distinctly, never swap one for the other.
|
||||
|
||||
## UI conventions
|
||||
|
||||
- Update the DOM **in place**. Slot cards are keyed by slot id in the `slotCards` Map and
|
||||
reused (`appendChild` reorders); no `innerHTML` rebuild per poll — it kills transitions
|
||||
and throws away focus.
|
||||
- Write only through `setTxt` / `setBar` / `drawSpark` / `setThroughput`; they diff before
|
||||
writing. `setThroughput(el, cell)` is the only place that renders the provenance tag
|
||||
(`Δ10.0s`, `from TPOT`, `lifetime avg`) and puts the `computed:` source in the tooltip.
|
||||
- Anything engine-shaped starts **hidden** in the markup and is revealed by `applyEngines()`
|
||||
from `/api/config`, so a single-engine deployment never flashes a panel it does not own.
|
||||
With fewer than two configured engines, `.panels`/`.strip` get `.single` (one column, no
|
||||
phantom gap) and `#fleet` is hidden.
|
||||
- Both panels deliberately chart the **same pair**: `decode tok/s · all slots` + cache usage
|
||||
(`gufo-decode`/`gufo-kv`, `vllm-decode`/`vllm-kv`), each fed from that engine's
|
||||
`decode_tokens_per_second` so the two engines are comparable at a glance. Keep the pairing if
|
||||
you add a metric — vLLM's running count is intentionally not charted (banner + fleet total).
|
||||
- **Fleet totals are summed in the browser** (`renderFleet`, called from `runJob`'s `finally` so
|
||||
failures re-sum too), over **the configured engines only** (`ENGINES` — an absent engine is
|
||||
not "down", it is not in the fleet), and include **only engines whose `/metrics` job is
|
||||
answering** — a stale
|
||||
engine's last good sample belongs in its own labelled panel, not in a total presented as
|
||||
current; the header then reads `1 of 2 engines` and the block goes amber. Liveness keys on the
|
||||
`/metrics` job alone, so gufo `/slots` failing must not drop gufo from the sums. Raw values are
|
||||
summed and rounded once, and each cell's tooltip itemises the parts.
|
||||
- Sparkline y-axis labels live in a `.spark-axis` gutter **outside** the `<svg>`, as HTML.
|
||||
Never move them into the SVG: `viewBox="0 0 100 30"` + `preserveAspectRatio="none"` stretches
|
||||
the plot to the card width and would smear any text drawn in it. `drawSpark` labels exactly
|
||||
the `lo`/`hi` it plotted against (so a label can never drift from the curve), keeps the three
|
||||
levels distinguishable by adding decimals only when needed, and reports the value itself for
|
||||
all three when the window is **flat** — the zero-range `pad` is what keeps a flat line
|
||||
mid-plot, it is not a scale, and labelling it invents `±0.001` on an idle engine. Cache charts
|
||||
pass `fixedMax`, so their axis is pinned 0–1 and must read `100% / 50% / 0%`.
|
||||
- Everything user-visible has an `n/a` path. The pause button and `Space` freeze polling for
|
||||
reading a snapshot — keep that working.
|
||||
|
||||
## Verifying a change
|
||||
|
||||
```bash
|
||||
uv run pytest # green
|
||||
uv run uvicorn app:app --host 127.0.0.1 --port 8090 &
|
||||
curl -s localhost:8090/api/config | python3 -m json.tool # includes gufo_enabled / vllm_enabled
|
||||
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8090/api/gufo/slots # must be 405
|
||||
```
|
||||
|
||||
Single-engine check (no real engines needed) — the unconfigured half must 404 **without**
|
||||
making an upstream call, and the page must render only the configured half:
|
||||
|
||||
```bash
|
||||
env -u GUFO_BASE_URL VLLM_BASE_URL=http://127.0.0.1:1 \
|
||||
uv run uvicorn app:app --host 127.0.0.1 --port 8090
|
||||
# /api/config -> gufo_enabled:false, vllm_enabled:true; /api/gufo/* -> 404 configured:false;
|
||||
# browser: #panel-gufo/#strip-gufo hidden, .panels has .single, #fleet hidden,
|
||||
# tab title "LLM Monitor · vLLM"
|
||||
```
|
||||
|
||||
Degradation check (works with no real engines): point one env var at a dead port and leave the
|
||||
other at a live one —
|
||||
|
||||
```bash
|
||||
GUFO_BASE_URL=http://127.0.0.1:8731 VLLM_BASE_URL=http://127.0.0.1:1 \
|
||||
uv run uvicorn app:app --host 127.0.0.1 --port 8090
|
||||
```
|
||||
|
||||
Expected: gufo green and ticking, vLLM dot red, vLLM banner showing the upstream error, and
|
||||
recovery on the next successful poll with no page reload.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Bare `uvicorn app:app` binds 8000** (an engine's port). Always `--port 8090`.
|
||||
- Config is read **once at import**. `--reload` watches code, not env; restart for env changes.
|
||||
Same for which engines exist — the env var cannot appear or disappear mid-run.
|
||||
- "Not configured" is a 404, never a 502: a server nobody installed is not an outage, and the
|
||||
page must not light a red dot for it. The 404 matrix in `tests/test_proxy.py` asserts no
|
||||
upstream call happens.
|
||||
- `httpx.Response(...)` in tests needs a `request=` before you read `.text`; it does **not**
|
||||
accept `reason_phrase` (it derives the phrase from the status code).
|
||||
- Histograms never emit a bare `<name>` series — resolve them via `_bucket`/`_sum`/`_count`
|
||||
(`Prom.resolve_hist`), which is why `histogram()` does not use `resolve()`.
|
||||
- gufo's `/slots` returns a **bare array**, not `{"slots": [...]}`; it also answers on the
|
||||
`/v1/slots` alias (the proxy tries both). Deferred requests are **not** in `/slots` — they
|
||||
only show up in `llamacpp:requests_deferred`.
|
||||
- In a slot payload, `-1` means "none" (`id_task`, `n_remain`) — render `n/a`, never `-1`.
|
||||
`is_processing` is authoritative; numeric `state` is the fallback.
|
||||
- One gufo slot can host several requests, so slot count ≠ `requests_processing`; don't
|
||||
"reconcile" the two numbers.
|
||||
- gufo's `prompt_tokens_total` **excludes** cache hits (`prompt_tokens_cached_total` is
|
||||
separate, and on a warm box it dwarfs it — 33M vs 0.38M). Anything named "total" has to add
|
||||
both or it under-reports by orders of magnitude.
|
||||
- Payloads carry `series` and `unparsed` counts. `unparsed > 0` or `series == 0` means the
|
||||
exposition format drifted — that is the drift signal, not an outage.
|
||||
- `pytest` prints a `StarletteDeprecationWarning` about httpx/TestClient. It is upstream and
|
||||
harmless; do not "fix" it by swapping test clients.
|
||||
- `requirements.txt` is **generated**. Edit `pyproject.toml`, then `uv lock` + re-export.
|
||||
`uv.lock` and `requirements.txt` are both committed on purpose.
|
||||
|
||||
## Definition of done
|
||||
|
||||
Tests green; both panels update live without a manual refresh; killing one engine flips only
|
||||
that engine's dot red while the other keeps ticking; an **unconfigured** engine shows nothing at
|
||||
all (no panel, no poll, no fleet share, `404 configured:false`) while the configured one keeps
|
||||
ticking; every new number has an `n/a` path; no new dependency without a strong reason (this app
|
||||
is 3 deps on purpose); README updated when behaviour, env vars, or ports change.
|
||||
@@ -0,0 +1,59 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
# llm_monitor — production image (Podman/Docker compatible).
|
||||
#
|
||||
# Stage 1 (python): install the locked dependencies with uv (deps only — the app is
|
||||
# a single module, there is nothing to build or install).
|
||||
# Stage 2 (runtime): slim, non-root, no build tools.
|
||||
#
|
||||
# There is deliberately NO frontend build stage: static/index.html is hand-rolled vanilla
|
||||
# JS + inline SVG with no bundler and no CDN (the engines live on an internet-disconnected
|
||||
# podman network, so a CDN asset would be a dead request and a blank chart). The file is
|
||||
# copied verbatim and served as-is.
|
||||
#
|
||||
# podman build -t localhost/llm-monitor:main -f Containerfile .
|
||||
# podman run -d --name llm-monitor -p 8090:8090 \
|
||||
# -e GUFO_BASE_URL=http://deskwork-ai:8731 \
|
||||
# -e VLLM_BASE_URL=http://driveripper-ai:8000 \
|
||||
# localhost/llm-monitor:main
|
||||
#
|
||||
# Both env vars are optional — unset means this deployment does not run that engine, the
|
||||
# proxy answers 404 {"configured": false} without dialling, and the page drops the panel.
|
||||
# Nothing is baked into the image: no host guesses, no keys.
|
||||
#
|
||||
# Both stages must stay on the SAME Python (3.14): stage 1 builds the venv that stage 2
|
||||
# copies verbatim, so a mismatch would ship a venv the runtime interpreter cannot import.
|
||||
# requires-python is >=3.10; 3.14 is what the box that runs this is on.
|
||||
|
||||
# ---------- Stage 1: python dependencies ----------
|
||||
FROM docker.io/python:3.14-slim AS python
|
||||
WORKDIR /app
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
|
||||
ENV UV_LINK_MODE=copy \
|
||||
UV_PYTHON_DOWNLOADS=0 \
|
||||
UV_PROJECT_ENVIRONMENT=/app/.venv
|
||||
# Lockfile first so dependency layers cache independently of the app source.
|
||||
COPY pyproject.toml uv.lock ./
|
||||
RUN uv sync --frozen --no-dev --no-cache
|
||||
|
||||
# ---------- Stage 2: runtime ----------
|
||||
FROM docker.io/python:3.14-slim AS runtime
|
||||
ENV PATH="/app/.venv/bin:$PATH" \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1
|
||||
RUN useradd --create-home --uid 10001 monitor
|
||||
WORKDIR /app
|
||||
COPY --from=python /app/.venv /app/.venv
|
||||
COPY app.py ./app.py
|
||||
COPY static ./static
|
||||
RUN chown -R monitor:monitor /app
|
||||
USER monitor
|
||||
# The dashboard listens on 8090 on purpose: 8000 is vLLM's port in this fleet and 8731 is
|
||||
# gufo's, so binding either would collide with an engine (or scrape itself).
|
||||
EXPOSE 8090
|
||||
ENV DASHBOARD_HOST=0.0.0.0 \
|
||||
DASHBOARD_PORT=8090
|
||||
# /api/config answers 200 with no upstream configured and never dials an engine, so a dead
|
||||
# gufo or vLLM can never make the dashboard container unhealthy.
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 \
|
||||
CMD ["python", "-c", "import sys, httpx; sys.exit(0 if httpx.get('http://127.0.0.1:8090/api/config', timeout=4).status_code == 200 else 1)"]
|
||||
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8090"]
|
||||
@@ -0,0 +1,432 @@
|
||||
# llm_monitor — gufo + vLLM real-time dashboard
|
||||
|
||||
One page, two engines, live. Watches a **gufo** (llama.cpp-family) server and a **vLLM**
|
||||
server side by side: sessions/slots, queueing, KV/GPU cache pressure, throughput, and
|
||||
latency histograms — refreshed every second without a page reload.
|
||||
|
||||
- **Read-only by construction.** The backend only ever calls `client.get()`. There is no
|
||||
POST/PUT/PATCH/DELETE route anywhere (`curl -X POST /api/gufo/slots` → `405`), nothing is
|
||||
written to either engine, and upstream bodies are parsed and re-emitted as JSON instead of
|
||||
echoed.
|
||||
- **Per-engine health.** Each engine is polled and tracked independently. One engine being
|
||||
down never blocks or blanks the other.
|
||||
- **Opt-in engines.** A server only appears if its `*_BASE_URL` env var is set. Send this to
|
||||
someone who runs just vLLM and they see one clean vLLM panel — no gufo half stuck in red.
|
||||
- **No build step, no CDN, no browser dependencies.** 2 app files (`app.py`,
|
||||
`static/index.html`). Sparklines are hand-rolled inline SVG — the engines live on an
|
||||
internet-disconnected podman network, so a CDN chart library would just be a dead request.
|
||||
|
||||
## Endpoints and ports (source of truth)
|
||||
|
||||
Taken from `~/Deployments/reeseapps/ai/deployments`:
|
||||
|
||||
| Engine | Deployment | Host | Reachable as | Port | Upstream endpoints used |
|
||||
| -------------------- | ---------------------------- | ---------------- | -------------------------------- | ---- | ------------------------------------------------ |
|
||||
| gufo | `deployments/gufo` | `deskwork-ai` | `deskwork.reeselink.com` | 8731 | `GET /slots` (alias `/v1/slots`), `GET /metrics` |
|
||||
| vLLM (radiance) | `deployments/vllm-radiance` | `driveripper-ai` | `driveripper.reeselink.com` | 8000 | `GET /metrics` |
|
||||
| **this dashboard** | — | wherever you run it | — | **8090** | serves the page + `/api/*` |
|
||||
|
||||
> Note the port: `8000` is already vLLM/juggernaut in that fleet and `8731` is gufo, so the
|
||||
> dashboard listens on **8090** by default. The `8080`/`8000` defaults you may expect for the
|
||||
> *upstreams* are wrong for this fleet — `8731` is gufo here.
|
||||
|
||||
## Install & run (uv)
|
||||
|
||||
This is a [uv](https://docs.astral.sh/uv/) project: `pyproject.toml` + `uv.lock`, no build
|
||||
backend (`package = false` — it is an app, not a library). uv manages the venv; you never
|
||||
activate it manually.
|
||||
|
||||
```bash
|
||||
uv sync # create .venv from uv.lock (incl. dev group)
|
||||
|
||||
# Point at the engines you actually run (these are the fleet's real endpoints), then start:
|
||||
export GUFO_BASE_URL=http://deskwork.reeselink.com:8731 # gufo — deskwork-ai:8731
|
||||
export VLLM_BASE_URL=http://driveripper.reeselink.com:8000 # vLLM — driveripper-ai:8000
|
||||
uv run uvicorn app:app --host 127.0.0.1 --port 8090
|
||||
```
|
||||
|
||||
Set only what exists. With just `VLLM_BASE_URL` the page is a single-engine vLLM dashboard
|
||||
and `/api/gufo/*` answers `404 {"configured": false}` without ever dialling a port; with
|
||||
neither set it shows a "no engines configured" card instead of two permanently red panels.
|
||||
|
||||
Open <http://127.0.0.1:8090>.
|
||||
|
||||
```bash
|
||||
uv run python app.py # same thing, reads DASHBOARD_HOST/DASHBOARD_PORT
|
||||
uv run uvicorn app:app --port 8090 --reload # dev loop (watches app.py + static/)
|
||||
uv sync --no-dev # production install: skips the pytest group
|
||||
```
|
||||
|
||||
> Keep the `--port 8090`. Bare `uvicorn app:app` binds `:8000`, which is where vLLM already
|
||||
> lives on `driveripper-ai` — the dashboard would refuse to start there (and if it started on
|
||||
> a machine without vLLM, `VLLM_BASE_URL` would point at itself).
|
||||
|
||||
Can't reach those hosts directly (VPN, firewall, different network)? Tunnel them to loopback
|
||||
and point the env vars at the tunnel:
|
||||
|
||||
```bash
|
||||
ssh -N -L 8731:127.0.0.1:8731 -L 8000:127.0.0.1:8000 deskwork-ai
|
||||
export GUFO_BASE_URL=http://127.0.0.1:8731 VLLM_BASE_URL=http://127.0.0.1:8000
|
||||
```
|
||||
|
||||
After editing dependencies: `uv lock` (updates `uv.lock`), then regenerate the pip shim:
|
||||
`uv export --no-dev --no-hashes > requirements.txt`.
|
||||
|
||||
### Without uv
|
||||
|
||||
`requirements.txt` is a pinned export of `uv.lock`, so a host that has no uv still works:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv && source .venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
uvicorn app:app --host 127.0.0.1 --port 8090
|
||||
```
|
||||
|
||||
### In a container
|
||||
|
||||
The `Containerfile` builds a slim, non-root image (uv-locked deps, no build tools, no
|
||||
frontend build step — the page is copied verbatim). CI publishes it to
|
||||
`gitea.reeseapps.com/Experimental/llm-monitor` on every push to `main` and on release tags
|
||||
(`.gitea/workflows/build-and-push.yml`).
|
||||
|
||||
```bash
|
||||
podman build -t localhost/llm-monitor:main -f Containerfile .
|
||||
podman run -d --name llm-monitor -p 8090:8090 \
|
||||
-e GUFO_BASE_URL=http://deskwork.reeselink.com:8731 \
|
||||
-e VLLM_BASE_URL=http://driveripper.reeselink.com:8000 \
|
||||
localhost/llm-monitor:main
|
||||
```
|
||||
|
||||
Nothing is baked in: no host guesses and no API keys live in the image, so the same image
|
||||
serves a two-engine deployment, a vLLM-only deployment, or a gufo-only one — set (or omit)
|
||||
`GUFO_BASE_URL` / `VLLM_API_KEY` at run time. Inside the container the dashboard binds
|
||||
`0.0.0.0:8090`; reach the engines by whatever name is resolvable from that network (or
|
||||
publish the container onto the engines' podman network and use the quadlet hostnames,
|
||||
`deskwork-ai` / `driveripper-ai`). The `HEALTHCHECK` probes `/api/config`, which never dials
|
||||
an engine — a dead engine must never mark the dashboard itself unhealthy.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
uv run pytest # 114 tests, no network, no real engines
|
||||
uv run pytest --cov # with coverage (app.py 98%)
|
||||
uv run pytest -k parsing -vv # just the Prometheus parser / field maps
|
||||
uv run -p 3.10 pytest # check the requires-python floor
|
||||
```
|
||||
|
||||
Supported Python: **>=3.10** (CI-worthy runs verified on 3.10 / 3.13 / 3.14). There is no
|
||||
`.python-version` pin on purpose — uv uses whatever interpreter is present, since this gets
|
||||
dropped onto whatever box hosts the engines.
|
||||
|
||||
`tests/conftest.py` swaps `app.state.client` for a `FakeUpstream` that records the URLs it
|
||||
was asked for and can raise `httpx.ConnectError` / `TimeoutException` on demand — so engine
|
||||
outages are asserted, not hand-tested. Suite covers: parser edge cases (aliases, absent
|
||||
series, label escaping, histogram interpolation), passthrough fidelity, 405 on every
|
||||
non-GET verb, 502/504 translation, per-engine auth, and the no-third-party-assets rule.
|
||||
|
||||
## Configuration (env vars only)
|
||||
|
||||
| Var | Default | Meaning |
|
||||
| ------------------------- | ------------------------ | -------------------------------------------------------------- |
|
||||
| `GUFO_BASE_URL` | **_unset_ → gufo hidden** | gufo origin (scheme://host[:port]); unset/blank means this deployment has no gufo |
|
||||
| `VLLM_BASE_URL` | **_unset_ → vLLM hidden** | vLLM origin; unset/blank means this deployment has no vLLM |
|
||||
| `GUFO_API_KEY` | _unset_ | if set, sent upstream as `Authorization: Bearer <key>` |
|
||||
| `VLLM_API_KEY` | _unset_ | if set, sent upstream as `Authorization: Bearer <key>` |
|
||||
| `UPSTREAM_TIMEOUT_SECONDS`| `3` | per-request upstream timeout (short: the UI polls every 1s) |
|
||||
| `RATE_WINDOW_SECONDS` | `10` | rolling window for vLLM Δcounter rates (see “Where prompt tok/s comes from”) |
|
||||
| `POLL_MS` | `1000` | browser poll interval |
|
||||
| `HISTORY_LEN` | `60` | sparkline samples kept in the browser |
|
||||
| `DASHBOARD_HOST`/`DASHBOARD_PORT` | `127.0.0.1` / `8090` | only used by `python app.py` |
|
||||
|
||||
There is deliberately **no default host** for the two engines. A built-in `127.0.0.1:8731`
|
||||
looked convenient until a vLLM-only user got a red gufo panel for a server they never
|
||||
installed; "not configured" and "configured but down" are different statements, so only the
|
||||
env var decides. `http://host:port/` is accepted (the trailing slash is trimmed) and a
|
||||
whitespace-only value counts as unset. `/api/config` exposes `gufo_enabled` / `vllm_enabled`,
|
||||
and the page builds its layout from those.
|
||||
|
||||
No auth on the dashboard itself by default — bind it to `127.0.0.1` (default) or put it
|
||||
behind your reverse proxy. The keys are never returned by any endpoint; `/api/config`
|
||||
reports only `gufo_auth: true/false`.
|
||||
|
||||
## Upstream start commands
|
||||
|
||||
These are the commands the dashboard expects to be able to scrape. Full worked examples are
|
||||
in each deployment's `README.md` in the deployments repo.
|
||||
|
||||
**gufo** — `/slots` and `/metrics` are built in; `--sessions N` is what sizes the slot grid:
|
||||
|
||||
```bash
|
||||
gufo serve llm --sessions 4 \
|
||||
--model /models/qwen3.8-flash-next/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
|
||||
--context 262144 --prefill-chunk 4096 --served-model-name turbo \
|
||||
--host 0.0.0.0 --port 8731
|
||||
# quadlet form: gufo serve --host 0.0.0.0 --port 8731 llm --sessions 2 ...
|
||||
```
|
||||
|
||||
**vLLM** — metrics are on by default at `/metrics` (no flag needed; `--disable-metrics` would
|
||||
break the vLLM panel):
|
||||
|
||||
```bash
|
||||
vllm serve amd/Qwen3.8-27B-Quark-AWQ-MXFP4 \
|
||||
--served-model-name=juggernaut \
|
||||
--max-model-len=131072 --max-num-seqs=4 --max-num-batched-tokens=4096 \
|
||||
--kv-cache-dtype=fp8 --enable-prefix-caching --gpu-memory-utilization=0.95 \
|
||||
--host=0.0.0.0 --port=8000
|
||||
```
|
||||
|
||||
## Backend routes
|
||||
|
||||
| Route | Upstream | Returns |
|
||||
| -------------------- | --------------------------- | ----------------------------------------------------------------------- |
|
||||
| `GET /` | — | the dashboard (`static/index.html`) |
|
||||
| `GET /api/config` | — | base URLs, `*_enabled` (which engines exist), poll/history settings, `*_auth` booleans (never the keys) |
|
||||
| `GET /api/gufo/slots`| `GET {GUFO_BASE_URL}/slots` | upstream JSON **passed through byte-for-byte**; falls back to `/v1/slots` |
|
||||
| `GET /api/gufo/metrics` | `GET {GUFO_BASE_URL}/metrics` | Prometheus text → JSON, gufo field map |
|
||||
| `GET /api/vllm/metrics` | `GET {VLLM_BASE_URL}/metrics` | Prometheus text → JSON, vLLM field map |
|
||||
|
||||
Envelope for the JSON routes:
|
||||
|
||||
```jsonc
|
||||
{ "ok": true, "engine": "vllm", "upstream": "http://127.0.0.1:8000/metrics",
|
||||
"scraped_at": 1760000000.123, "series": 21, "metrics": { "...": {} } }
|
||||
```
|
||||
|
||||
If an engine's `*_BASE_URL` was unset, its two routes answer
|
||||
`404 {"ok": false, "engine": "gufo", "configured": false, "detail": "gufo is not configured —
|
||||
set GUFO_BASE_URL and restart"}` and **no upstream request is made** — a server that is not
|
||||
installed is not an outage, so it must not look like one.
|
||||
|
||||
Every metric cell is `{"value": <number|null>, "source": "<series or computed:...>|null",
|
||||
"derived": <bool>}`. A missing series is `value: null`, which the page renders as **`n/a`** —
|
||||
it never 500s because a metric name changed. Upstream failures return `502` (unreachable /
|
||||
HTTP error) or `504` (timeout) with `{"ok": false, "detail": "..."}`, and that `detail` is
|
||||
what the panel shows.
|
||||
|
||||
## One Prometheus parser, two field maps
|
||||
|
||||
`app.py` contains a single Prometheus text parser (`class Prom`: tolerant line matcher,
|
||||
label unescaping, `sum`/`max`/`avg` aggregation, by-label grouping, histogram
|
||||
`_sum`/`_count`/`_bucket` → mean + interpolated p95). The engines differ only in their field
|
||||
maps:
|
||||
|
||||
The tables below are **what the deployed engines actually export** (captured live; see
|
||||
`tests/fixtures/*_real.txt`). Each field is an alias list, so the names from the original
|
||||
spec and the legacy `llama:` prefix still work — but the deployed name comes first, and the
|
||||
live fixtures are the regression test that keeps them honest.
|
||||
|
||||
**gufo** — note `tokens_predicted_*` (not `generated`), and the speed gauges are named
|
||||
`*_tokens_seconds` while their unit is tokens/second:
|
||||
|
||||
| UI field | Series (deployed name first) | Agg |
|
||||
| ------------------- | ------------------------------------------------------------------- | ---- |
|
||||
| Running | `llamacpp:requests_processing` | sum |
|
||||
| Waiting | `llamacpp:requests_deferred` | sum |
|
||||
| KV cache | `llamacpp:kv_cache_usage_ratio` | max |
|
||||
| prompt tok·s⁻¹ (all slots) | **`computed:Δllamacpp:prompt_tokens_total/Δt`** (`RateEstimator`), else `llamacpp:prompt_tokens_seconds` | Δ / avg |
|
||||
| prompt tok·s⁻¹ (last pass) | `llamacpp:prompt_tokens_seconds` — **one request's last prefill pass**; exposed as `prompt_tokens_per_second_native` | avg |
|
||||
| decode tok·s⁻¹ (all slots) | **`computed:Δllamacpp:tokens_predicted_total/Δt`** (`RateEstimator`), else `llamacpp:predicted_tokens_seconds` | Δ / avg |
|
||||
| decode tok·s⁻¹ (last pass) | `llamacpp:predicted_tokens_seconds` — **one request's last decode pass**, holds that value while the box idles; exposed as `decode_tokens_per_second_native` | avg |
|
||||
| prompt tokens | `llamacpp:prompt_tokens_total` — **excludes cache hits** | sum |
|
||||
| cache-hit tokens | `llamacpp:prompt_tokens_cached_total` | sum |
|
||||
| generated tokens | `llamacpp:tokens_predicted_total` | sum |
|
||||
| total tokens | `computed:prompt+cached+generated` | — |
|
||||
| prompt cache | `computed:cached/(prompt+cached)` | — |
|
||||
| peak ctx | `llamacpp:n_tokens_max` | max |
|
||||
| gpu losses | `gufo_device_lost_total` (non-zero = incident, shown red) | sum |
|
||||
| spec accept | `computed:accepted/draft_tokens` from `llamacpp:spec_decode_num_*` | — |
|
||||
|
||||
**vLLM** — `kv_cache_usage_perc` (not `gpu_cache_usage`), `generation_tokens_total` (not
|
||||
`generation_tokens_total`), TPOT as `inter_token_latency_seconds`, and every series tagged
|
||||
`engine=` / `model_name=`:
|
||||
|
||||
| UI field | Series (deployed name first) | Notes |
|
||||
| --------------------- | ---------------------------------------------------------------- | ------------------------------------- |
|
||||
| Running | `vllm:num_requests_running` | sum across `engine=` label sets |
|
||||
| Waiting | `vllm:num_requests_waiting` | sum |
|
||||
| Waiting · capacity | `vllm:num_requests_waiting_by_reason{reason="capacity"}` | live reasons: `capacity`, `deferred` |
|
||||
| GPU cache | `vllm:kv_cache_usage_perc`, `vllm:gpu_cache_usage` | max (never sums past 1.0) |
|
||||
| TTFT | `vllm:time_to_first_token_seconds` | histogram → mean, p95 |
|
||||
| TPOT | `vllm:inter_token_latency_seconds`, `vllm:time_per_output_token_seconds`, `vllm:request_time_per_output_token_seconds` | histogram → mean, p95 |
|
||||
| Queue time | `vllm:request_queue_time_seconds` | histogram → mean, p95 |
|
||||
| prompt / generation tokens | `vllm:prompt_tokens_total`, `vllm:generation_tokens_total` | sum |
|
||||
| prefix cache hit | `computed:hits/queries` from `vllm:prefix_cache_{hits,queries}_total` | — |
|
||||
| preemptions | `vllm:num_preemptions_total` (non-zero = KV pressure, shown red) | sum |
|
||||
| prompt tok·s⁻¹ | `vllm:prompt_throughput`, `vllm:avg_prompt_throughput_toks_per_s` — **neither exists on this build** — else **`computed:Δprompt_tokens_total/Δt`**, else **`computed:mean(request_prompt_tokens)/mean(request_prefill_time_seconds)`** | see below |
|
||||
| prefill avg | `computed:mean(request_prompt_tokens)/mean(request_prefill_time_seconds)` | lifetime average, ~4.9k tok/s live |
|
||||
| decode tok·s⁻¹ | `vllm:generation_throughput` if present, else **`computed:Δgeneration_tokens_total/Δt`**, else **`computed:running/tpot`** | `derived: true`; the UI tags the source |
|
||||
| per-seq tok·s⁻¹ | `computed:1/tpot` | from TPOT mean |
|
||||
|
||||
### Where vLLM's prompt tok/s comes from
|
||||
|
||||
That vLLM exports **no rate gauge at all** — no `*_throughput`, no `*_toks_per_s`, nothing
|
||||
with a rate-ish name (grep its `/metrics` and you get zero hits). Counters only. So the
|
||||
number is computed, in this order, and the UI labels which one you are looking at:
|
||||
|
||||
1. **`vllm:prompt_throughput`** if a future build provides it — the only real scraped gauge.
|
||||
2. **Measured Δ** — `Δprompt_tokens_total / Δt` over a rolling `RATE_WINDOW_SECONDS` window
|
||||
(default 10s). This is *actual* current throughput, tagged `Δ10.0s`. A bare 1-second delta
|
||||
jitters between 0 and spikes because prompts arrive in bursts, hence the window. Counter
|
||||
resets (engine restart) yield `n/a` for one scrape rather than a huge negative rate.
|
||||
3. **Lifetime average prefill speed** — `mean(request_prompt_tokens) / mean(request_prefill_time_seconds)`,
|
||||
i.e. tokens-per-second this engine has achieved on prefill since it started (~4.9k tok/s
|
||||
live). Used only until there are two scrapes to diff, and shown on its own as **prefill avg**.
|
||||
|
||||
Decode tok/s follows the same precedence, and the measured Δ replaced the old
|
||||
`running/tpot` estimate as step 2 — the Δ counts real output, the estimate assumed every
|
||||
running request was decoding at that instant. The per-request figure is still `1/TPOT`.
|
||||
|
||||
An `n/a` now means the counters themselves are gone, not that a gauge name changed.
|
||||
|
||||
### Where gufo's decode tok/s comes from
|
||||
|
||||
gufo's **decode tok/s is the aggregate over every slot generating at once**, and it is
|
||||
measured, not scraped. `llamacpp:predicted_tokens_seconds` is a **single unlabeled gauge**
|
||||
holding the last decode pass of **one** request: with two slots decoding at 35 tok/s it still
|
||||
reads 35, and when the box goes idle it keeps serving that stale number indefinitely (live:
|
||||
stuck at 38.0 with `requests_processing == 0`). There are no label sets to `sum`, so no
|
||||
aggregation rule could ever fix it.
|
||||
|
||||
`llamacpp:tokens_predicted_total` advances once per token from every slot, so the same
|
||||
`RateEstimator` used for vLLM answers the question directly — two slots at 35 tok/s advance
|
||||
the counter 70 tokens per second and the cell reads ~70:
|
||||
|
||||
1. **Measured Δ** — `Δtokens_predicted_total / Δt` over the `RATE_WINDOW_SECONDS` window,
|
||||
tagged `Δ10.0s`. Goes to a genuine `0.0` when nothing is generating.
|
||||
2. **The scraped gauge** — only until there are two scrapes to diff, or on a build missing
|
||||
the counter; rendered with no tag (it is a real scrape) and still exposed separately as
|
||||
`decode_tokens_per_second_native`.
|
||||
|
||||
This is the one place where measured **outranks** a native gauge, deliberately inverting the
|
||||
usual precedence: vLLM's native throughput gauge really is an aggregate, gufo's is not. Both
|
||||
engines share the single `RATES` estimator and cannot cross-read, because it keys off the
|
||||
*resolved* series name — which also means a counter rename starts a fresh window instead of
|
||||
diffing two different series and inventing throughput.
|
||||
|
||||
gufo's `/slots` is passed through untouched and returns a **bare JSON array** (older builds
|
||||
wrap it as `{"slots": [...]}`; the page accepts both). Slot fields read: `id`,
|
||||
`is_processing` (authoritative; `state` 0/1 is the fallback), `id_task`/`task_id`,
|
||||
`n_prompt_tokens`, `n_prompt_tokens_cache`, `n_prompt_tokens_processed`, `n_ctx`, `model`,
|
||||
`speculative`, and `next_token[0].{n_decoded,n_remain}` (`-1` means "nothing pending" → `n/a`).
|
||||
|
||||
## Dashboard layout
|
||||
|
||||
- **Fleet total** (very top): one number per metric summed across both engines — running,
|
||||
waiting, prompt tok·s⁻¹, decode tok·s⁻¹, total tokens. Hidden entirely when fewer than two
|
||||
engines are configured (a cross-engine sum of one engine just duplicates its own card).
|
||||
Rules:
|
||||
- **Only engines answering now are counted.** A dead engine keeps its last good sample in
|
||||
its own panel (labelled stale, correct there), but adding a frozen number to a fleet total
|
||||
would present hours-old data as current — so it is dropped, the header drops to `1 of 2
|
||||
engines`, and the accent/dot go amber. Zero answering → every cell `n/a`, dot red.
|
||||
Liveness is keyed on each engine's **`/metrics` job only**: gufo's `/slots` failing must
|
||||
not remove it from a total that came from `/metrics`.
|
||||
- **Summed in the browser**, not on the backend. The page already holds both parsed payloads,
|
||||
so the API stays one-scrape-pure with no second polling path hammering the engines.
|
||||
- Hover any cell for the itemised breakdown (`gufo 45 + vLLM 37 · excluded: …`), with each
|
||||
part marked `(measured)` when it is a derived rate.
|
||||
- Sums are computed on the raw values then rounded once, so a cell can differ from the two
|
||||
rounded panel numbers above it by 1 (`308 + 177` shown, `486` total).
|
||||
- **The prompt sum is the one imperfect addition**: gufo's prompt counter *excludes* cache
|
||||
hits (so it measures newly-computed prefill work) while vLLM's *includes* cached prefix
|
||||
tokens. Both are per-engine aggregates and additive within an engine, but the fleet number
|
||||
leans slightly low for gufo. The `prompt tok/s` label carries the caveat on hover.
|
||||
- **Header line** names only the engines that exist (`vLLM http://host:8000 · poll 1000ms ·
|
||||
read-only`), and the tab title follows (`LLM Monitor · vLLM`).
|
||||
- **Shared stats strip** (top): per engine — cache-usage bar (blue → amber ≥75% → red ≥90%),
|
||||
prompt tok·s⁻¹, decode tok·s⁻¹, total tokens, plus slots-busy (gufo) / per-seq tok·s⁻¹
|
||||
(vLLM). Strip labels stay terse (they share one dense grid), so gufo's cell just reads
|
||||
**decode tok/s** with its all-slots meaning on hover; the gufo panel's sparkline caption
|
||||
spells it out as **decode tok/s · all slots**. Every derived rate carries its provenance tag
|
||||
(`Δ10.0s`, `from TPOT`, `lifetime avg`) with the exact `computed:` source on hover.
|
||||
- **gufo panel**: vLLM-style banner `Running: N reqs, Waiting: M`, sparklines for the
|
||||
all-slots decode tok·s⁻¹ and KV usage, then a grid with **one card per `/slots` entry**: slot id + served
|
||||
model (`+spec` when speculative decoding is on), status pill (green `processing` / grey
|
||||
`idle`), request id, prompt tokens, cache hits (with % of prompt), generated so far
|
||||
(`next_token[0].n_decoded`), remaining budget, and slot context size. Queued requests are
|
||||
not in `/slots` — that is what `Waiting` (deferred) covers.
|
||||
- **Alarm colours**: `gpu losses` (gufo `gufo_device_lost_total`) and `preemptions`
|
||||
(`vllm:num_preemptions_total`) turn red above zero — both are cumulative counters, so a
|
||||
moving number there means something is wrong, not busy.
|
||||
- **vLLM panel**: banner `Running: N reqs, Waiting: M`, sparklines for the all-slots decode
|
||||
tok·s⁻¹ and GPU cache usage — deliberately the **same pair of charts as the gufo panel**, both
|
||||
fed by `decode_tokens_per_second` (`Δvllm:generation_tokens_total/Δt` here) so the two engines
|
||||
read alike. The running count is no longer charted; it lives in the banner and the fleet
|
||||
total. Then metric cards for waiting-by-reason, GPU cache bar, TTFT / TPOT / queue
|
||||
time (mean + p95), prompt + generation token totals, and series scraped.
|
||||
- **Sparkline y-axis**: each chart labels the top, middle and bottom of the scale it actually
|
||||
plotted against (`49 / 43 / 37`, or `100% / 50% / 0%` for the cache charts, which are pinned
|
||||
to 0–1 rather than autoscaled), with faint rules at those levels. The plots autoscale, so
|
||||
without the labels a 1% drift and a 2× spike are the same squiggle. Labels are rendered as
|
||||
HTML beside the SVG, never inside it — `preserveAspectRatio="none"` stretches the viewBox to
|
||||
the card width and would smear the text. Precision adapts so the three never collide (a 1–2
|
||||
request axis reads `2.0 / 1.5 / 1.0`, not `2 / 2 / 1`), and a **flat** window labels all three
|
||||
levels with the value itself — an idle box says `0 / 0 / 0`, not the `±0.001` the plot's
|
||||
zero-range padding happens to produce. Fewer than two points → axis `n/a`.
|
||||
- **Every second**: all four `/api/*` endpoints are polled independently
|
||||
(`setInterval` + fire-and-forget, with an in-flight guard so a slow upstream can't pile up
|
||||
requests). DOM is updated in place — slot cards are keyed by slot id and reused, so the
|
||||
poll is invisible. Per-engine green/red dot, "updated HH:MM:SS" stamp, and a
|
||||
`stale Ns` counter that only appears when that engine stopped answering. `Space` or the
|
||||
**pause** button freezes polling for reading a snapshot.
|
||||
|
||||
## Failure behaviour
|
||||
|
||||
| Situation | Result |
|
||||
| ---------------------------------------- | ----------------------------------------------------------------- |
|
||||
| Engine's `*_BASE_URL` unset | that engine is **not on the page at all** (no strip card, no panel, no polling, no fleet share) and its routes answer `404 configured:false`; neither configured → a setup card naming the env vars |
|
||||
| Engine stopped / unreachable | that panel's dot goes red, red banner with the upstream error, last good values kept and greyed, `stale Ns` grows; **the other engine keeps polling normally** |
|
||||
| Engine comes back | dot goes green and the banner clears on the next successful poll — no reload needed |
|
||||
| A metric series is absent | that one cell shows `n/a` (and `series absent → n/a` in its foot), everything else renders |
|
||||
| Upstream returns non-Prometheus garbage | parsed as 0 series → all cells `n/a`, page still healthy |
|
||||
| Sparkline has < 2 points | chart stays faded and its y-axis reads `n/a` rather than inventing a scale |
|
||||
| Upstream slower than `UPSTREAM_TIMEOUT_SECONDS` | `504`, dot red, and the request is abandoned rather than queued |
|
||||
|
||||
## Smoke test without live engines
|
||||
|
||||
Start the app first (`uv run uvicorn app:app --host 127.0.0.1 --port 8090`), then:
|
||||
|
||||
```bash
|
||||
curl -s localhost:8090/api/config | python3 -m json.tool # what it is pointed at
|
||||
curl -s localhost:8090/api/gufo/slots | head -c 200 # proxy passthrough
|
||||
curl -s localhost:8090/api/gufo/metrics | python3 -m json.tool | head -30
|
||||
curl -s localhost:8090/api/vllm/metrics | python3 -m json.tool | head -30
|
||||
curl -s -o /dev/null -w '%{http_code}\n' -X POST localhost:8090/api/gufo/slots # -> 405 (read-only)
|
||||
```
|
||||
|
||||
One-engine deployment (what a vLLM-only recipient sees) — gufo answers `404 configured:false`
|
||||
and the page has no gufo section at all:
|
||||
|
||||
```bash
|
||||
env -u GUFO_BASE_URL VLLM_BASE_URL=http://127.0.0.1:8000 \
|
||||
uv run uvicorn app:app --host 127.0.0.1 --port 8090
|
||||
curl -s localhost:8090/api/config | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["gufo_enabled"], d["vllm_enabled"])' # False True
|
||||
curl -s -o /dev/null -w '%{http_code}\n' localhost:8090/api/gufo/metrics # -> 404, nothing dialled
|
||||
```
|
||||
|
||||
`n/a` + red-dot path described above.
|
||||
|
||||
To see what an engine really exposes (do this before trusting any metric name, including the
|
||||
ones written in this README):
|
||||
|
||||
```bash
|
||||
curl -s "$GUFO_BASE_URL/metrics" | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
|
||||
curl -s "$VLLM_BASE_URL/metrics" | grep -v '^#' | sed -E 's/[{ ].*//' | sort -u
|
||||
curl -s "$GUFO_BASE_URL/slots" | python3 -m json.tool | head # a bare array
|
||||
```
|
||||
|
||||
## Files
|
||||
|
||||
```
|
||||
pyproject.toml uv project: deps, dev group, package=false, pytest/coverage config
|
||||
uv.lock pinned resolution (commit it)
|
||||
app.py FastAPI app: config, Prometheus parser, 2 field maps, 5 GET routes
|
||||
static/index.html the dashboard: CSS + vanilla JS + inline-SVG sparklines
|
||||
tests/ conftest.py (FakeUpstream), test_parsing.py, test_proxy.py,
|
||||
test_real_contracts.py + fixtures/ (verbatim captures from the live engines)
|
||||
requirements.txt pip shim, generated from uv.lock
|
||||
AGENTS.md working rules for humans and agents on this repo
|
||||
README.md
|
||||
```
|
||||
|
||||
There are no JS dependencies and nothing to build: `static/index.html` is served verbatim.
|
||||
@@ -0,0 +1,819 @@
|
||||
"""llm_monitor — read-only, real-time dashboard for a gufo server and a vLLM server.
|
||||
|
||||
Two upstream engines, one page. The backend exists purely to sidestep CORS: the
|
||||
browser only ever talks to this origin, and this origin forwards a small set of
|
||||
read-only GETs to the engines.
|
||||
|
||||
GET / -> the dashboard (static/index.html)
|
||||
GET /api/config -> upstream base URLs (no secrets)
|
||||
GET /api/gufo/slots -> proxied passthrough of gufo /slots
|
||||
GET /api/gufo/metrics -> gufo /metrics (Prometheus text) as JSON
|
||||
GET /api/vllm/metrics -> vLLM /metrics (Prometheus text) as JSON
|
||||
|
||||
STRICTLY READ-ONLY: this module only ever issues `client.get(...)`. There is no
|
||||
POST/PUT/PATCH/DELETE route, nothing writes to the engines, and upstream bodies
|
||||
are parsed and re-emitted as JSON rather than echoed.
|
||||
|
||||
An engine only exists when its *_BASE_URL env var is set: unset means this deployment
|
||||
does not run it, its routes answer 404 "not configured" without dialling anything, and
|
||||
the page leaves its section out entirely.
|
||||
|
||||
Both /metrics endpoints share ONE Prometheus text parser (class `Prom`) — the
|
||||
gufo (llama.cpp-style `llamacpp:*`) and vLLM (`vllm:*`) expositions are the same
|
||||
format, so there is exactly one parsing path and per-engine field maps on top.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from collections import deque
|
||||
from contextlib import asynccontextmanager
|
||||
from pathlib import Path
|
||||
from typing import Any, Sequence
|
||||
|
||||
import httpx
|
||||
from fastapi import Depends, FastAPI, Request
|
||||
from fastapi.responses import FileResponse, JSONResponse, Response
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Configuration (env only — no config file, no CLI flags)
|
||||
# --------------------------------------------------------------------------- #
|
||||
# An engine is part of this deployment only when its *_BASE_URL is set. There is
|
||||
# deliberately NO default host: a built-in 127.0.0.1 guess made a vLLM-only box (or
|
||||
# anyone who just cloned this repo) render a permanently red gufo panel for a server
|
||||
# they do not run. Unset -> the endpoint 404s as "not configured" and the page hides
|
||||
# that section entirely.
|
||||
#
|
||||
# Real values live in ~/Deployments/reeseapps/ai/deployments:
|
||||
# gufo/quadlets/gufo.container -> PublishPort=8731:8731/tcp
|
||||
# vllm-radiance/quadlets/...container -> PublishPort=8000:8000/tcp
|
||||
# Keep them tunnel-friendly on 127.0.0.1 and put real hosts in the env.
|
||||
def _base_url_from_env(name: str) -> str:
|
||||
"""`http://host:port/` -> `http://host:port`; unset or blank -> `""` (not configured).
|
||||
|
||||
Split out so the "does this env configure the engine?" rule is testable without
|
||||
re-importing the app.
|
||||
"""
|
||||
return os.environ.get(name, "").strip().rstrip("/")
|
||||
|
||||
|
||||
GUFO_BASE_URL = _base_url_from_env("GUFO_BASE_URL")
|
||||
VLLM_BASE_URL = _base_url_from_env("VLLM_BASE_URL")
|
||||
GUFO_API_KEY = os.environ.get("GUFO_API_KEY", "").strip()
|
||||
VLLM_API_KEY = os.environ.get("VLLM_API_KEY", "").strip()
|
||||
# Short on purpose: the UI polls every second, so a wedged engine must fail fast
|
||||
# rather than pile up requests.
|
||||
UPSTREAM_TIMEOUT = float(os.environ.get("UPSTREAM_TIMEOUT_SECONDS", "3"))
|
||||
|
||||
STATIC_DIR = Path(__file__).resolve().parent / "static"
|
||||
|
||||
POLL_MS = int(os.environ.get("POLL_MS", "1000"))
|
||||
HISTORY_LEN = int(os.environ.get("HISTORY_LEN", "60"))
|
||||
|
||||
|
||||
def _auth_headers(api_key: str) -> dict[str, str]:
|
||||
"""Optional bearer token. Empty key -> no Authorization header at all."""
|
||||
return {"Authorization": f"Bearer {api_key}"} if api_key else {}
|
||||
|
||||
|
||||
def not_configured(engine: str, env_var: str) -> JSONResponse:
|
||||
"""Body for an engine this deployment does not run.
|
||||
|
||||
404 rather than 502: nothing is broken upstream, the upstream was never called. The
|
||||
`configured: false` flag lets the page keep "absent engine" and "engine is down"
|
||||
visibly different instead of showing a red dot for a server nobody installed.
|
||||
"""
|
||||
return JSONResponse(
|
||||
{
|
||||
"ok": False,
|
||||
"engine": engine,
|
||||
"configured": False,
|
||||
"detail": f"{engine} is not configured — set {env_var} and restart",
|
||||
},
|
||||
status_code=404,
|
||||
headers=NO_STORE,
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Prometheus text exposition parser (shared by gufo and vLLM)
|
||||
# --------------------------------------------------------------------------- #
|
||||
_NAME_RE = r"[a-zA-Z_:][a-zA-Z0-9_:]*"
|
||||
# name{label="v",...} value [timestamp]
|
||||
# - the label block is greedy so a '}' inside a quoted value cannot truncate the
|
||||
# match; the anchored value/end-of-line does that instead.
|
||||
# - whitespace before '{' and a trailing '# ...' comment are not part of the
|
||||
# official exposition format, but tolerating them means a quirky exporter shows
|
||||
# a number here instead of 'n/a'.
|
||||
_LINE_RE = re.compile(
|
||||
rf"^(?P<name>{_NAME_RE})(?:[ \t]*\{{(?P<labels>.*)\}})?[ \t]+"
|
||||
r"(?P<value>[+-]?(\d+\.?\d*(e[+-]?\d+)?|inf|nan))"
|
||||
r"(?:[ \t]+(?P<ts>-?\d+))?[ \t]*(?:#.*)?$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_LABEL_RE = re.compile(r'([a-zA-Z_][a-zA-Z0-9_.:]*)="((?:[^"\\]|\\.)*)"')
|
||||
|
||||
|
||||
def _unquote(value: str) -> str:
|
||||
return re.sub(r"\\(.)", r"\1", value)
|
||||
|
||||
|
||||
class Prom:
|
||||
"""Parsed Prometheus exposition: name -> [(labels, value), ...]."""
|
||||
|
||||
def __init__(self, text: str) -> None:
|
||||
self.series: dict[str, list[tuple[dict[str, str], float]]] = {}
|
||||
self.parse_errors = 0
|
||||
for raw in text.splitlines():
|
||||
line = raw.strip()
|
||||
if not line or line[0] == "#": # comments / # TYPE / # HELP
|
||||
continue
|
||||
m = _LINE_RE.match(line)
|
||||
if not m:
|
||||
self.parse_errors += 1
|
||||
continue
|
||||
try:
|
||||
value = float(m.group("value"))
|
||||
except ValueError: # pragma: no cover - regex keeps this unreachable
|
||||
self.parse_errors += 1
|
||||
continue
|
||||
labels = {
|
||||
_unquote(k): _unquote(v) for k, v in _LABEL_RE.findall(m.group("labels") or "")
|
||||
}
|
||||
self.series.setdefault(m.group("name"), []).append((labels, value))
|
||||
|
||||
# -- lookup helpers ----------------------------------------------------- #
|
||||
def resolve(self, names: Sequence[str]) -> str | None:
|
||||
"""First name from `names` that is actually present."""
|
||||
return next((n for n in names if n in self.series), None)
|
||||
|
||||
def resolve_hist(self, names: Sequence[str]) -> str | None:
|
||||
"""Like resolve(), but for histograms: the bare name never appears in the
|
||||
exposition, only <name>_bucket / _sum / _count do."""
|
||||
for n in names:
|
||||
if any(f"{n}{suffix}" in self.series for suffix in ("_bucket", "_sum", "_count")):
|
||||
return n
|
||||
return None
|
||||
|
||||
def values(
|
||||
self,
|
||||
names: Sequence[str],
|
||||
label_match: dict[str, str] | None = None,
|
||||
) -> list[float]:
|
||||
name = self.resolve(names)
|
||||
if name is None:
|
||||
return []
|
||||
if not label_match:
|
||||
return [v for _labels, v in self.series[name]]
|
||||
return [v for labels, v in self.series[name] if all(labels.get(k) == x for k, x in label_match.items())]
|
||||
|
||||
def scalar(
|
||||
self,
|
||||
names: Sequence[str],
|
||||
agg: str = "sum",
|
||||
label_match: dict[str, str] | None = None,
|
||||
) -> float | None:
|
||||
"""Aggregate every matching series, or None when the metric is absent.
|
||||
|
||||
agg="sum" counters and per-engine gauges (running/waiting) — summing
|
||||
across label sets is the right fleet-wide total.
|
||||
agg="max" usage ratios — report the most pressured engine, never a sum
|
||||
that could exceed 1.0.
|
||||
agg="avg" rate gauges — an average across label sets.
|
||||
"""
|
||||
found = self.values(names, label_match)
|
||||
finite = [v for v in found if math.isfinite(v)]
|
||||
if not finite:
|
||||
return None
|
||||
if agg == "max":
|
||||
return max(finite)
|
||||
if agg == "avg":
|
||||
return sum(finite) / len(finite)
|
||||
return sum(finite)
|
||||
|
||||
def by_label(self, names: Sequence[str], label: str) -> dict[str, float]:
|
||||
"""Sum series grouped by `label` (e.g. vLLM waiting-by-reason)."""
|
||||
name = self.resolve(names)
|
||||
if name is None:
|
||||
return {}
|
||||
out: dict[str, float] = {}
|
||||
for labels, value in self.series[name]:
|
||||
key = labels.get(label, "")
|
||||
if not math.isfinite(value):
|
||||
continue
|
||||
out[key] = out.get(key, 0.0) + value
|
||||
return out
|
||||
|
||||
def histogram(self, names: Sequence[str]) -> dict[str, Any] | None:
|
||||
"""Mean (+ p95 when interpolable) of a `<name>_sum` / `_count` / `_bucket`."""
|
||||
base = self.resolve_hist(names)
|
||||
if base is None:
|
||||
return None
|
||||
total = self.scalar([f"{base}_sum"])
|
||||
count = self.scalar([f"{base}_count"])
|
||||
buckets = self._buckets(base)
|
||||
out: dict[str, Any] = {
|
||||
"count": count,
|
||||
"sum": total,
|
||||
"mean": (total / count) if (total is not None and count) else None,
|
||||
"p95": None,
|
||||
"source": base,
|
||||
}
|
||||
if buckets and count:
|
||||
# Interpolate only for a single label set: mixing per-DP-rank
|
||||
# histograms into one cumulative curve would invent a bogus p95.
|
||||
distinct_groups = self._bucket_label_groups(base)
|
||||
if distinct_groups <= 1:
|
||||
out["p95"] = _quantile(buckets, count, 0.95)
|
||||
return out
|
||||
|
||||
def _bucket_label_groups(self, base: str) -> int:
|
||||
groups = set()
|
||||
for labels, _v in self.series.get(f"{base}_bucket", []):
|
||||
groups.add(tuple(sorted((k, v) for k, v in labels.items() if k != "le")))
|
||||
return len(groups)
|
||||
|
||||
def _buckets(self, base: str) -> list[tuple[float, float]]:
|
||||
"""(le, cumulative_count) sorted ascending by bound."""
|
||||
out: list[tuple[float, float]] = []
|
||||
for labels, value in self.series.get(f"{base}_bucket", []):
|
||||
le = labels.get("le")
|
||||
if le is None or not math.isfinite(value):
|
||||
continue
|
||||
bound = math.inf if le in ("+Inf", "Inf", "inf") else float(le)
|
||||
out.append((bound, value))
|
||||
out.sort(key=lambda item: item[0])
|
||||
return out
|
||||
|
||||
|
||||
def _quantile(buckets: list[tuple[float, float]], count: float, q: float) -> float | None:
|
||||
"""Linear interpolation inside the bucket that contains the `q` quantile."""
|
||||
target = q * count
|
||||
prev_bound, prev_cum = 0.0, 0.0
|
||||
for bound, cum in buckets:
|
||||
if cum >= target:
|
||||
if math.isinf(bound):
|
||||
# Only mass above the last real bucket — best honest guess.
|
||||
return prev_bound if math.isfinite(prev_bound) else None
|
||||
span = cum - prev_cum
|
||||
if span <= 0:
|
||||
return bound
|
||||
return prev_bound + (bound - prev_bound) * ((target - prev_cum) / span)
|
||||
prev_bound, prev_cum = bound, cum
|
||||
return None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Per-engine field maps (the only engine-specific knowledge on the backend)
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _field(
|
||||
value: float | None,
|
||||
source: str | None,
|
||||
derived: bool = False,
|
||||
window: float | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Uniform metric cell: the UI renders `n/a` whenever value is None.
|
||||
|
||||
`window` is the Δt a rate was measured over, so the UI can say `Δ1.0s` instead of
|
||||
implying a scraped gauge it never got.
|
||||
"""
|
||||
cell: dict[str, Any] = {"value": value, "source": source, "derived": derived}
|
||||
if window is not None:
|
||||
cell["window"] = window
|
||||
return cell
|
||||
|
||||
|
||||
class RateEstimator:
|
||||
"""Δcounter / Δt across consecutive scrapes.
|
||||
|
||||
Stock vLLM exports no throughput gauge at all — only counters — so the only way to
|
||||
answer "how many tokens per second RIGHT NOW" is to difference `*_total` between two
|
||||
scrapes and label it as exactly that. Deliberately injected rather than global so the
|
||||
parsers stay pure and testable; a counter reset (engine restart) yields None, never a
|
||||
huge negative spike.
|
||||
|
||||
One instance per process: with several uvicorn workers each keeps its own previous
|
||||
sample, which only skews the window, not the meaning.
|
||||
"""
|
||||
|
||||
def __init__(self, span: float = 10.0, max_samples: int = 512) -> None:
|
||||
self.span = span
|
||||
self.max_samples = max_samples
|
||||
self._samples: dict[str, deque[tuple[float, float]]] = {}
|
||||
|
||||
def rate(
|
||||
self, key: str, value: float | None, now: float | None = None
|
||||
) -> tuple[float | None, float | None]:
|
||||
"""(tokens per second, window actually used) or (None, window|None)."""
|
||||
if value is None:
|
||||
return None, None
|
||||
now = time.time() if now is None else now
|
||||
samples = self._samples.setdefault(key, deque())
|
||||
if samples and value < samples[-1][1]:
|
||||
samples.clear() # counter reset = engine restart; the old history is a lie
|
||||
samples.append((now, value))
|
||||
while len(samples) > self.max_samples:
|
||||
samples.popleft()
|
||||
if len(samples) < 2:
|
||||
return None, None # first scrape after start: nothing to diff against
|
||||
|
||||
# Oldest sample still inside the span; if only the newest one is, fall back to
|
||||
# the previous sample so a fast poller still gets a rate (over a shorter window).
|
||||
inside = [point for point in samples if now - point[0] <= self.span]
|
||||
base = inside[0] if len(inside) > 1 else samples[-2]
|
||||
dt, dv = now - base[0], value - base[1]
|
||||
# forget anything older than we could ever use again
|
||||
cutoff = now - 2 * self.span
|
||||
while samples and samples[0][0] < cutoff and len(samples) > 2:
|
||||
samples.popleft()
|
||||
if dt <= 0 or dv < 0:
|
||||
return None, dt if dt > 0 else None
|
||||
return dv / dt, dt
|
||||
|
||||
|
||||
# ui key -> (candidate upstream series in preference order, aggregation over label sets).
|
||||
#
|
||||
# The first entry of each tuple is what the DEPLOYED gufo actually exports
|
||||
# (verified against deskwork-ai:8731/metrics). gufo/llama.cpp does not use the names
|
||||
# people expect: `prompt_tokens_total` not `tokens_prompt_total`, `tokens_predicted_total`
|
||||
# not `tokens_generated_total`, and the speed gauges are named `*_tokens_seconds` while
|
||||
# their unit is tokens/second. Later entries are the older `llama:` prefix and the
|
||||
# spec'd names, kept as fallbacks so an older/other build still renders.
|
||||
GUFO_SERIES: dict[str, tuple[tuple[str, ...], str]] = {
|
||||
"requests_processing": (("llamacpp:requests_processing", "llama:requests_processing"), "sum"),
|
||||
"requests_deferred": (("llamacpp:requests_deferred", "llama:requests_deferred"), "sum"),
|
||||
"kv_cache_usage_ratio": (("llamacpp:kv_cache_usage_ratio", "llama:kv_cache_usage_ratio"), "max"),
|
||||
# gufo counts cache hits separately: prompt_tokens_total EXCLUDES cached tokens.
|
||||
"tokens_prompt_total": (
|
||||
("llamacpp:prompt_tokens_total", "llama:prompt_tokens_total",
|
||||
"llamacpp:tokens_prompt_total", "llama:tokens_prompt_total"),
|
||||
"sum",
|
||||
),
|
||||
"tokens_prompt_cached_total": (
|
||||
("llamacpp:prompt_tokens_cached_total", "llama:prompt_tokens_cached_total"),
|
||||
"sum",
|
||||
),
|
||||
"tokens_generated_total": (
|
||||
("llamacpp:tokens_predicted_total", "llamacpp:tokens_generated_total",
|
||||
"llama:tokens_predicted_total", "llama:tokens_generated_total"),
|
||||
"sum",
|
||||
),
|
||||
# NOT the throughput cell: prefill speed of one request's last pass (see below).
|
||||
"prompt_tokens_per_second_native": (
|
||||
("llamacpp:prompt_tokens_seconds", "llamacpp:prompt_tokens_per_second",
|
||||
"llama:prompt_tokens_seconds"),
|
||||
"avg",
|
||||
),
|
||||
# NOT the throughput cell: one unlabeled gauge holding the speed of the most recent
|
||||
# decode pass, i.e. a single request. Kept as a fallback + per-request reference.
|
||||
"decode_tokens_per_second_native": (
|
||||
("llamacpp:predicted_tokens_seconds", "llamacpp:decode_tokens_per_second",
|
||||
"llama:predicted_tokens_seconds"),
|
||||
"avg",
|
||||
),
|
||||
"n_tokens_max": (("llamacpp:n_tokens_max", "llama:n_tokens_max"), "max"),
|
||||
# GPU context losses (gufo-specific): any non-zero value here is an incident.
|
||||
"device_lost_total": (("gufo_device_lost_total",), "sum"),
|
||||
# speculative decoding (MTP on the fleet): accept rate is the useful number.
|
||||
"spec_drafts_total": (("llamacpp:spec_decode_num_drafts_total",), "sum"),
|
||||
"spec_draft_tokens_total": (("llamacpp:spec_decode_num_draft_tokens_total",), "sum"),
|
||||
"spec_accepted_tokens_total": (("llamacpp:spec_decode_num_accepted_tokens_total",), "sum"),
|
||||
}
|
||||
|
||||
|
||||
def _ratio(numerator: dict[str, Any], denominator: dict[str, Any], source: str) -> dict[str, Any]:
|
||||
num, den = numerator["value"], denominator["value"]
|
||||
if num is None or den is None or den == 0:
|
||||
return _field(None, None)
|
||||
return _field(num / den, source, derived=True)
|
||||
|
||||
|
||||
def parse_gufo_metrics(
|
||||
text: str, rates: RateEstimator | None = None, now: float | None = None
|
||||
) -> dict[str, Any]:
|
||||
"""`rates` enables the Δ-based aggregate decode rate; omitted, the parse stays pure
|
||||
(deterministic) and that cell falls back to the scraped gauge.
|
||||
"""
|
||||
p = Prom(text)
|
||||
out: dict[str, Any] = {}
|
||||
|
||||
for key, (names, agg) in GUFO_SERIES.items():
|
||||
out[key] = _field(p.scalar(names, agg), p.resolve(names))
|
||||
|
||||
# ------- aggregate throughput: the counters add slots, gufo's gauges cannot ------- #
|
||||
# gufo exports both speeds as ONE unlabeled gauge each, holding the last decode (or
|
||||
# prefill) pass of ONE request: with two slots busy it still reports a single
|
||||
# request's speed, and it holds that number stale once the box goes idle. `avg` over
|
||||
# label sets can never sum concurrency — there are no label sets to sum. The token
|
||||
# counters advance once per token from every slot, so Δtokens/Δt over the window IS
|
||||
# the server-aggregate: two slots at 35 tok/s read ~70. Measured therefore wins over
|
||||
# the gauge for both cells, which deliberately inverts the usual precedence (vLLM's
|
||||
# native gauge really is an aggregate, gufo's are not). Each gauge stays the fallback
|
||||
# for the first scrape and for builds missing the counter, and the UI tags provenance.
|
||||
def measured(total_key: str) -> dict[str, Any] | None:
|
||||
"""Δcounter/Δt for one cumulative counter, keyed by the RESOLVED series name so an
|
||||
alias rename starts a fresh window instead of diffing two different series into a
|
||||
fabricated rate.
|
||||
"""
|
||||
if rates is None:
|
||||
return None
|
||||
source = out[total_key]["source"]
|
||||
if source is None:
|
||||
return None
|
||||
rate, window = rates.rate(source, out[total_key]["value"], now)
|
||||
if rate is None:
|
||||
return None
|
||||
return _field(rate, f"computed:Δ{source}/Δt", derived=True, window=window)
|
||||
|
||||
for ui_key, native_key, total_key in (
|
||||
("decode_tokens_per_second", "decode_tokens_per_second_native", "tokens_generated_total"),
|
||||
("prompt_tokens_per_second", "prompt_tokens_per_second_native", "tokens_prompt_total"),
|
||||
):
|
||||
gauge = out[native_key]
|
||||
out[ui_key] = measured(total_key) or _field(gauge["value"], gauge["source"])
|
||||
|
||||
# gufo counts prompt cache hits separately and prompt_tokens_total EXCLUDES them, so
|
||||
# the prompt rate above is newly-computed prefill work, not prompt tokens accepted.
|
||||
# That is the number the gauge it replaces was reporting; vLLM's prompt counter
|
||||
# includes cached prefix tokens, so a cross-engine prompt-rate sum is close to, not
|
||||
# identical to, like-for-like — the UI says so on hover.
|
||||
|
||||
prompt = out["tokens_prompt_total"]["value"]
|
||||
cached = out["tokens_prompt_cached_total"]["value"] or 0.0
|
||||
generated = out["tokens_generated_total"]["value"]
|
||||
parts = [v for v in (prompt, generated) if v is not None]
|
||||
out["tokens_total"] = (
|
||||
_field(sum(parts) + cached, "computed:prompt+cached+generated")
|
||||
if parts
|
||||
else _field(None, None)
|
||||
)
|
||||
# share of prompt tokens served from cache, over everything the model was asked
|
||||
if prompt is not None:
|
||||
out["prompt_cache_hit_ratio"] = _field(
|
||||
cached / (prompt + cached) if (prompt + cached) > 0 else None,
|
||||
"computed:cached/(prompt+cached)",
|
||||
derived=True,
|
||||
)
|
||||
else:
|
||||
out["prompt_cache_hit_ratio"] = _field(None, None)
|
||||
out["spec_accept_rate"] = _ratio(
|
||||
out["spec_accepted_tokens_total"], out["spec_draft_tokens_total"],
|
||||
"computed:accepted/draft_tokens",
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
# Same idea as GUFO_SERIES: deployed vLLM first, older/spec'd names as fallbacks.
|
||||
# Verified against driveripper-ai:8000/metrics, which exports `kv_cache_usage_perc`
|
||||
# (NOT `gpu_cache_usage`) and `generation_tokens_total` (NOT `generation_tokens_total`),
|
||||
# tags every series with engine= / model_name=, and reports TPOT as
|
||||
# `inter_token_latency_seconds`.
|
||||
VLLM_SERIES: dict[str, tuple[tuple[str, ...], str]] = {
|
||||
"num_requests_running": (("vllm:num_requests_running",), "sum"),
|
||||
"num_requests_waiting": (("vllm:num_requests_waiting",), "sum"),
|
||||
"gpu_cache_usage": (("vllm:kv_cache_usage_perc", "vllm:gpu_cache_usage"), "max"),
|
||||
"prompt_tokens_total": (("vllm:prompt_tokens_total",), "sum"),
|
||||
"generation_tokens_total": (("vllm:generation_tokens_total", "vllm:generation_tokens_total"), "sum"),
|
||||
# throughput gauges only exist on some vLLM builds; absent -> derived from TPOT
|
||||
"prompt_tokens_per_second": (
|
||||
("vllm:prompt_throughput", "vllm:avg_prompt_throughput_toks_per_s"),
|
||||
"avg",
|
||||
),
|
||||
"decode_tokens_per_second_native": (
|
||||
("vllm:generation_throughput", "vllm:avg_generation_throughput_toks_per_s"),
|
||||
"avg",
|
||||
),
|
||||
"prefix_cache_queries_total": (("vllm:prefix_cache_queries_total",), "sum"),
|
||||
"prefix_cache_hits_total": (("vllm:prefix_cache_hits_total",), "sum"),
|
||||
# preemption = a running request kicked back to the queue: the real KV-pressure signal
|
||||
"num_preemptions_total": (("vllm:num_preemptions_total",), "sum"),
|
||||
}
|
||||
|
||||
VLLM_HISTOGRAMS = {
|
||||
# ui key -> candidate upstream names
|
||||
"time_to_first_token_seconds": ("vllm:time_to_first_token_seconds",),
|
||||
"time_per_output_token_seconds": (
|
||||
"vllm:time_per_output_token_seconds",
|
||||
"vllm:inter_token_latency_seconds",
|
||||
"vllm:request_time_per_output_token_seconds",
|
||||
),
|
||||
"request_queue_time_seconds": ("vllm:request_queue_time_seconds",),
|
||||
# not shown directly: mean prompt length / mean prefill time = lifetime prefill
|
||||
# speed, which is what makes prompt tok/s non-n/a on a build with no rate gauge
|
||||
"request_prefill_time_seconds": ("vllm:request_prefill_time_seconds",),
|
||||
"request_prompt_tokens": ("vllm:request_prompt_tokens",),
|
||||
}
|
||||
|
||||
|
||||
def out_histogram_mean(out: dict[str, Any], key: str) -> float | None:
|
||||
"""Mean of a histogram already parsed into `out` under `key`."""
|
||||
cell = out.get(key)
|
||||
mean = cell.get("mean") if isinstance(cell, dict) else None
|
||||
return mean if isinstance(mean, float) and math.isfinite(mean) else None
|
||||
|
||||
|
||||
def parse_vllm_metrics(
|
||||
text: str, rates: RateEstimator | None = None, now: float | None = None
|
||||
) -> dict[str, Any]:
|
||||
"""`rates` enables the Δ-based rates; omitted, the parse stays pure and those cells
|
||||
fall back to whatever the engine natively reports (or None)."""
|
||||
p = Prom(text)
|
||||
out: dict[str, Any] = {}
|
||||
|
||||
for key, (names, agg) in VLLM_SERIES.items():
|
||||
out[key] = _field(p.scalar(names, agg), p.resolve(names))
|
||||
|
||||
# Waiting, split by reason (capacity / defragmentation / ...).
|
||||
reasons = p.by_label(
|
||||
("vllm:num_requests_waiting_by_reason", "vllm:num_requests_waiting_by_reason_total"),
|
||||
"reason",
|
||||
)
|
||||
out["waiting_by_reason"] = {
|
||||
"value": reasons,
|
||||
"source": p.resolve(
|
||||
("vllm:num_requests_waiting_by_reason", "vllm:num_requests_waiting_by_reason_total")
|
||||
),
|
||||
"derived": False,
|
||||
}
|
||||
out["waiting_capacity"] = _field(
|
||||
reasons.get("capacity"),
|
||||
p.resolve(("vllm:num_requests_waiting_by_reason",)),
|
||||
)
|
||||
|
||||
# Histograms -> mean (and p95 when interpolable).
|
||||
for ui_key, names in VLLM_HISTOGRAMS.items():
|
||||
out[ui_key] = p.histogram(names)
|
||||
# Flattened scalar form for the simple card renderer.
|
||||
out[f"{ui_key}__mean"] = _field(
|
||||
out[ui_key]["mean"] if out[ui_key] else None,
|
||||
out[ui_key]["source"] if out[ui_key] else None,
|
||||
)
|
||||
out[f"{ui_key}__p95"] = _field(
|
||||
out[ui_key]["p95"] if out[ui_key] else None,
|
||||
out[ui_key]["source"] if out[ui_key] else None,
|
||||
)
|
||||
|
||||
# Decode throughput: prefer a native gauge, else derive from TPOT so the
|
||||
# stats strip is never blank on a stock vLLM.
|
||||
tpot = out["time_per_output_token_seconds"]["mean"] if out["time_per_output_token_seconds"] else None
|
||||
per_seq = (1.0 / tpot) if tpot else None
|
||||
out["per_request_decode_tokens_per_second"] = _field(per_seq, "computed:1/tpot", derived=True)
|
||||
|
||||
# ---- throughput: native gauge > measured Δcounter > estimate, in that order ---- #
|
||||
def measured(key: str, total_key: str) -> dict[str, Any] | None:
|
||||
if rates is None:
|
||||
return None
|
||||
rate, window = rates.rate(key, out[total_key]["value"], now)
|
||||
if rate is None:
|
||||
return None
|
||||
return _field(
|
||||
rate, f"computed:Δ{VLLM_SERIES[total_key][0][0]}/Δt", derived=True, window=window
|
||||
)
|
||||
|
||||
decode = out["decode_tokens_per_second_native"]["value"]
|
||||
running = out["num_requests_running"]["value"]
|
||||
if decode is not None:
|
||||
out["decode_tokens_per_second"] = _field(
|
||||
decode, out["decode_tokens_per_second_native"]["source"]
|
||||
)
|
||||
else:
|
||||
# measured aggregate beats the running/tpot estimate: it counts real output, the
|
||||
# estimate assumes every running request is decoding this very instant
|
||||
out["decode_tokens_per_second"] = (
|
||||
measured("vllm:generation_tokens_total", "generation_tokens_total")
|
||||
or (
|
||||
_field((running or 0) * per_seq, "computed:running/tpot", derived=True)
|
||||
if per_seq is not None
|
||||
else _field(None, None)
|
||||
)
|
||||
)
|
||||
|
||||
# prompt tok/s: the deployed build has no rate gauge, so diff the counter. On the
|
||||
# first scrape after start there is nothing to diff, and we fall back to the lifetime
|
||||
# average prefill speed (mean prompt length / mean prefill time) rather than n/a.
|
||||
prefill_avg = None
|
||||
prefill_time = out_histogram_mean(out, "request_prefill_time_seconds")
|
||||
prompt_len = out_histogram_mean(out, "request_prompt_tokens")
|
||||
if prefill_time and prompt_len:
|
||||
prefill_avg = _field(
|
||||
prompt_len / prefill_time,
|
||||
"computed:mean(request_prompt_tokens)/mean(request_prefill_time_seconds)",
|
||||
derived=True,
|
||||
)
|
||||
if out["prompt_tokens_per_second"]["value"] is None:
|
||||
out["prompt_tokens_per_second"] = (
|
||||
measured("vllm:prompt_tokens_total", "prompt_tokens_total")
|
||||
or prefill_avg
|
||||
or _field(None, None)
|
||||
)
|
||||
out["prefill_tokens_per_second_lifetime"] = prefill_avg or _field(None, None)
|
||||
|
||||
# sum only the parts that exist: a partial exposition (one counter renamed away)
|
||||
# must degrade to a smaller total, not raise and 500 the endpoint
|
||||
parts = [v for v in (out["prompt_tokens_total"]["value"], out["generation_tokens_total"]["value"]) if v is not None]
|
||||
out["tokens_total"] = (
|
||||
_field(sum(parts), "computed:prompt+generated") if parts else _field(None, None)
|
||||
)
|
||||
out["prefix_cache_hit_ratio"] = _ratio(
|
||||
out["prefix_cache_hits_total"], out["prefix_cache_queries_total"],
|
||||
"computed:hits/queries",
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# App + upstream fetch helpers
|
||||
# --------------------------------------------------------------------------- #
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
app.state.client = httpx.AsyncClient(
|
||||
timeout=httpx.Timeout(UPSTREAM_TIMEOUT),
|
||||
follow_redirects=False,
|
||||
limits=httpx.Limits(max_connections=32, max_keepalive_connections=16),
|
||||
)
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
await app.state.client.aclose()
|
||||
|
||||
|
||||
app = FastAPI(title="llm_monitor", version="1.0.0", lifespan=lifespan, docs_url=None, redoc_url=None)
|
||||
|
||||
# Rolling counters behind every Δcounter/Δt rate. Module-level and deliberately the only
|
||||
# mutable state in the app: everything else is a pure function of one scrape. The ~10s
|
||||
# span keeps the strip readable — a bare 1s delta jitters between 0 and big spikes
|
||||
# because prompts arrive in bursts, not continuously.
|
||||
#
|
||||
# ONE instance for both engines: it is keyed by the resolved upstream series name
|
||||
# (`llamacpp:tokens_predicted_total` vs `vllm:generation_tokens_total`), so the engines
|
||||
# can never read each other's samples.
|
||||
RATES = RateEstimator(span=float(os.environ.get("RATE_WINDOW_SECONDS", "10")))
|
||||
|
||||
NO_STORE = {"cache-control": "no-store, max-age=0"}
|
||||
|
||||
|
||||
def client_of(request: Request) -> httpx.AsyncClient:
|
||||
return request.app.state.client
|
||||
|
||||
|
||||
class UpstreamError(Exception):
|
||||
"""An upstream problem. Carries the upstream HTTP status when there was one,
|
||||
so a caller can branch (gufo /slots -> /v1/slots fallback) before it is
|
||||
rendered as a JSON error for the browser."""
|
||||
|
||||
def __init__(self, message: str, *, upstream_status: int | None = None) -> None:
|
||||
super().__init__(message)
|
||||
self.message = message
|
||||
self.upstream_status = upstream_status
|
||||
|
||||
|
||||
@app.exception_handler(UpstreamError)
|
||||
async def upstream_error_handler(request: Request, exc: UpstreamError) -> JSONResponse:
|
||||
# 502 = upstream answered badly / not at all, 504 = upstream too slow.
|
||||
status = 504 if exc.upstream_status == 504 else 502
|
||||
return JSONResponse({"ok": False, "detail": exc.message}, status_code=status, headers=NO_STORE)
|
||||
|
||||
|
||||
async def fetch_text(client: httpx.AsyncClient, url: str, headers: dict[str, str]) -> str:
|
||||
return (await fetch(client, url, headers)).text
|
||||
|
||||
|
||||
async def fetch(client: httpx.AsyncClient, url: str, headers: dict[str, str]) -> httpx.Response:
|
||||
"""GET only. Any upstream problem becomes an UpstreamError so the UI can flip
|
||||
that engine's dot red while the other engine keeps polling."""
|
||||
try:
|
||||
resp = await client.get(url, headers=headers)
|
||||
except httpx.TimeoutException as exc:
|
||||
raise UpstreamError(f"{url} timed out after {UPSTREAM_TIMEOUT:g}s", upstream_status=504) from exc
|
||||
except httpx.HTTPError as exc:
|
||||
raise UpstreamError(f"{url} unreachable: {type(exc).__name__}") from exc
|
||||
if resp.status_code >= 400:
|
||||
raise UpstreamError(
|
||||
f"{url} -> HTTP {resp.status_code} {resp.reason_phrase}".strip(),
|
||||
upstream_status=resp.status_code,
|
||||
)
|
||||
return resp
|
||||
|
||||
|
||||
def ok_payload(engine: str, upstream: str, metrics: Any, parsed: Prom | None = None) -> JSONResponse:
|
||||
"""`series` / `unparsed` make a metric-name change diagnosable from the browser
|
||||
without ssh'ing to the engine: 0 series, or a big `unparsed`, means the exposition
|
||||
format drifted from what `parse_gufo_metrics` / `parse_vllm_metrics` expect."""
|
||||
body: dict[str, Any] = {
|
||||
"ok": True,
|
||||
"engine": engine,
|
||||
"upstream": upstream,
|
||||
"scraped_at": time.time(),
|
||||
"metrics": metrics,
|
||||
}
|
||||
if parsed is not None:
|
||||
body["series"] = len(parsed.series)
|
||||
body["unparsed"] = parsed.parse_errors
|
||||
return JSONResponse(body, headers=NO_STORE)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Routes
|
||||
# --------------------------------------------------------------------------- #
|
||||
@app.get("/", include_in_schema=False)
|
||||
async def dashboard() -> FileResponse:
|
||||
return FileResponse(STATIC_DIR / "index.html", media_type="text/html", headers=NO_STORE)
|
||||
|
||||
|
||||
@app.get("/api/config")
|
||||
async def config() -> JSONResponse:
|
||||
"""What the page is pointed at, and which engines exist at all. Never includes API keys.
|
||||
|
||||
The `*_enabled` flags are what the page renders its layout from: an engine with no
|
||||
base URL gets no panel, no strip card and no polling. Derived from the base URL on
|
||||
purpose — there is no separate ENABLED flag that could disagree with it.
|
||||
"""
|
||||
return JSONResponse(
|
||||
{
|
||||
"gufo_base_url": GUFO_BASE_URL,
|
||||
"vllm_base_url": VLLM_BASE_URL,
|
||||
"gufo_enabled": bool(GUFO_BASE_URL),
|
||||
"vllm_enabled": bool(VLLM_BASE_URL),
|
||||
"gufo_auth": bool(GUFO_API_KEY),
|
||||
"vllm_auth": bool(VLLM_API_KEY),
|
||||
"poll_ms": POLL_MS,
|
||||
"history_len": HISTORY_LEN,
|
||||
"upstream_timeout_seconds": UPSTREAM_TIMEOUT,
|
||||
},
|
||||
headers=NO_STORE,
|
||||
)
|
||||
|
||||
|
||||
@app.get("/api/gufo/slots")
|
||||
async def gufo_slots(client: httpx.AsyncClient = Depends(client_of)) -> Response:
|
||||
"""Pass gufo's /slots JSON straight through (it is already JSON).
|
||||
|
||||
/v1/slots is the documented alias, so a server that only exposes the alias
|
||||
still works. Queued/deferred requests never appear here — the deferred count
|
||||
comes from /metrics instead.
|
||||
"""
|
||||
if not GUFO_BASE_URL:
|
||||
return not_configured("gufo", "GUFO_BASE_URL")
|
||||
headers = _auth_headers(GUFO_API_KEY)
|
||||
tried: list[str] = []
|
||||
last_error: UpstreamError | None = None
|
||||
for path in ("/slots", "/v1/slots"):
|
||||
url = GUFO_BASE_URL + path
|
||||
tried.append(path)
|
||||
try:
|
||||
resp = await fetch(client, url, headers)
|
||||
except UpstreamError as exc:
|
||||
if exc.upstream_status == 404: # try the documented alias
|
||||
last_error = exc
|
||||
continue
|
||||
raise
|
||||
return Response(
|
||||
content=resp.content,
|
||||
media_type=resp.headers.get("content-type", "application/json"),
|
||||
headers=NO_STORE,
|
||||
)
|
||||
detail = f"{GUFO_BASE_URL} has no {' or '.join(tried)}"
|
||||
raise UpstreamError(detail if last_error is None else f"{detail} ({last_error.message})")
|
||||
|
||||
|
||||
@app.get("/api/gufo/metrics")
|
||||
async def gufo_metrics(client: httpx.AsyncClient = Depends(client_of)) -> JSONResponse:
|
||||
if not GUFO_BASE_URL:
|
||||
return not_configured("gufo", "GUFO_BASE_URL")
|
||||
url = GUFO_BASE_URL + "/metrics"
|
||||
text = await fetch_text(client, url, _auth_headers(GUFO_API_KEY))
|
||||
parsed = Prom(text)
|
||||
# RATES turns the cumulative token counter into an aggregate tok/s, which is the only
|
||||
# way to count every concurrent slot — gufo's speed gauge tracks one request.
|
||||
return ok_payload("gufo", url, parse_gufo_metrics(text, RATES), parsed)
|
||||
|
||||
|
||||
@app.get("/api/vllm/metrics")
|
||||
async def vllm_metrics(client: httpx.AsyncClient = Depends(client_of)) -> JSONResponse:
|
||||
if not VLLM_BASE_URL:
|
||||
return not_configured("vllm", "VLLM_BASE_URL")
|
||||
url = VLLM_BASE_URL + "/metrics"
|
||||
text = await fetch_text(client, url, _auth_headers(VLLM_API_KEY))
|
||||
parsed = Prom(text)
|
||||
# RATES carries the previous scrape's counters, so prompt/decode tok/s can be
|
||||
# measured as Δcounter/Δt (this vLLM build exports no rate gauge of its own).
|
||||
return ok_payload("vllm", url, parse_vllm_metrics(text, RATES), parsed)
|
||||
|
||||
|
||||
@app.exception_handler(404)
|
||||
async def not_found(request: Request, exc: Any) -> JSONResponse:
|
||||
return JSONResponse({"ok": False, "detail": f"no such route: {request.url.path}"}, status_code=404)
|
||||
|
||||
|
||||
# Convenience for `python app.py` — equivalent to `uvicorn app:app --port 8090`.
|
||||
if __name__ == "__main__":
|
||||
import uvicorn
|
||||
|
||||
uvicorn.run(
|
||||
app,
|
||||
host=os.environ.get("DASHBOARD_HOST", "127.0.0.1"),
|
||||
port=int(os.environ.get("DASHBOARD_PORT", "8090")),
|
||||
)
|
||||
@@ -0,0 +1,38 @@
|
||||
[project]
|
||||
name = "llm-monitor"
|
||||
version = "1.0.0"
|
||||
description = "Read-only real-time dashboard for a gufo server and a vLLM server (sessions, queueing, cache pressure, throughput, latency)"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
# Deliberately tiny: the Prometheus exposition is parsed in-process (class Prom in
|
||||
# app.py) rather than pulled in as prometheus_client, and the frontend has zero
|
||||
# runtime deps (vanilla JS + inline SVG, no CDN).
|
||||
dependencies = [
|
||||
"fastapi>=0.111",
|
||||
"httpx>=0.27",
|
||||
"uvicorn[standard]>=0.30",
|
||||
]
|
||||
|
||||
# uv dev group: installed by plain `uv sync` (skip with `uv sync --no-dev`).
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
"pytest>=8",
|
||||
"pytest-cov>=5",
|
||||
]
|
||||
|
||||
# Not a distributable package — a single-file app run as `uv run uvicorn app:app`.
|
||||
# package=false means no build backend, nothing to install, `uv run` just syncs deps.
|
||||
[tool.uv]
|
||||
package = false
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
addopts = "-q --strict-markers"
|
||||
|
||||
[tool.coverage.run]
|
||||
source = ["app"]
|
||||
omit = ["tests/*"]
|
||||
|
||||
[tool.coverage.report]
|
||||
show_missing = true
|
||||
skip_covered = false
|
||||
@@ -0,0 +1,80 @@
|
||||
# Generated from uv.lock — DO NOT EDIT BY HAND.
|
||||
# uv lock # after changing pyproject.toml
|
||||
# uv export --no-dev --no-hashes > requirements.txt
|
||||
# uv users do not need this file: 'uv sync' installs straight from uv.lock.
|
||||
# It exists so a host without uv can still: pip install -r requirements.txt
|
||||
|
||||
# This file was autogenerated by uv via the following command:
|
||||
# uv export --no-dev --no-hashes
|
||||
annotated-doc==0.0.5
|
||||
# via fastapi
|
||||
annotated-types==0.8.0
|
||||
# via pydantic
|
||||
anyio==4.15.1
|
||||
# via
|
||||
# httpx
|
||||
# starlette
|
||||
# watchfiles
|
||||
certifi==2026.7.22
|
||||
# via
|
||||
# httpcore
|
||||
# httpx
|
||||
click==8.5.0
|
||||
# via uvicorn
|
||||
exceptiongroup==1.3.1 ; python_full_version < '3.11'
|
||||
# via anyio
|
||||
fastapi==0.143.2
|
||||
# via llm-monitor
|
||||
h11==0.16.0
|
||||
# via
|
||||
# httpcore
|
||||
# uvicorn
|
||||
httpcore==1.0.9
|
||||
# via httpx
|
||||
httptools==0.9.0
|
||||
# via uvicorn
|
||||
httpx==0.28.1
|
||||
# via llm-monitor
|
||||
idna==3.20
|
||||
# via
|
||||
# anyio
|
||||
# httpx
|
||||
opentelemetry-api==1.45.1
|
||||
# via fastapi
|
||||
pydantic==2.14.1
|
||||
# via fastapi
|
||||
pydantic-core==2.50.1
|
||||
# via pydantic
|
||||
python-dotenv==1.2.4
|
||||
# via uvicorn
|
||||
pyyaml==6.0.3
|
||||
# via uvicorn
|
||||
starlette==1.7.0 ; python_full_version < '3.11'
|
||||
# via fastapi
|
||||
starlette==1.8.0 ; python_full_version >= '3.11'
|
||||
# via fastapi
|
||||
typing-extensions==4.16.0
|
||||
# via
|
||||
# anyio
|
||||
# exceptiongroup
|
||||
# fastapi
|
||||
# opentelemetry-api
|
||||
# pydantic
|
||||
# pydantic-core
|
||||
# starlette
|
||||
# typing-inspection
|
||||
# uvicorn
|
||||
typing-inspection==0.4.4
|
||||
# via
|
||||
# fastapi
|
||||
# pydantic
|
||||
uvicorn==0.54.0
|
||||
# via llm-monitor
|
||||
uvloop==0.23.0 ; platform_python_implementation != 'PyPy' and sys_platform != 'cygwin' and sys_platform != 'win32'
|
||||
# via uvicorn
|
||||
watchfiles==1.3.0
|
||||
# via uvicorn
|
||||
websockets==16.1.1 ; python_full_version < '3.11'
|
||||
# via uvicorn
|
||||
websockets==17.2 ; python_full_version >= '3.11'
|
||||
# via uvicorn
|
||||
+1673
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,128 @@
|
||||
"""Test fixtures: a fake upstream (httpx-shaped) so no sockets are involved.
|
||||
|
||||
The app only ever calls `client.get(...)`, so a stub with an async `get` is a
|
||||
complete substitute for httpx.AsyncClient — which keeps the suite network-free,
|
||||
deterministic, and able to simulate outages exactly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
import app as app_module
|
||||
|
||||
GUFO_FIXTURE = """# HELP llamacpp:requests_processing how many requests are processing
|
||||
# TYPE llamacpp:requests_processing gauge
|
||||
llama:requests_processing 99
|
||||
llamacpp:requests_processing 2
|
||||
# TYPE llamacpp:requests_deferred gauge
|
||||
llamacpp:requests_deferred 1
|
||||
# TYPE llamacpp:kv_cache_usage_ratio gauge
|
||||
llamacpp:kv_cache_usage_ratio 0.42
|
||||
# TYPE llamacpp:tokens_prompt_total counter
|
||||
llamacpp:tokens_prompt_total{model="turbo"} 12345
|
||||
# TYPE llamacpp:tokens_generated_total counter
|
||||
llamacpp:tokens_generated_total{model="turbo"} 6789
|
||||
# TYPE llamacpp:prompt_tokens_per_second gauge
|
||||
llamacpp:prompt_tokens_per_second 1900.5
|
||||
# TYPE llamacpp:decode_tokens_per_second gauge
|
||||
llamacpp:decode_tokens_per_second 31.5
|
||||
"""
|
||||
|
||||
VLLM_FIXTURE = """# TYPE vllm:num_requests_running gauge
|
||||
vllm:num_requests_running{model_name="juggernaut"} 3
|
||||
# TYPE vllm:num_requests_waiting gauge
|
||||
vllm:num_requests_waiting{model_name="juggernaut"} 5
|
||||
# TYPE vllm:num_requests_waiting_by_reason gauge
|
||||
vllm:num_requests_waiting_by_reason{model_name="juggernaut",reason="capacity"} 5
|
||||
vllm:num_requests_waiting_by_reason{model_name="juggernaut",reason="defragmentation"} 0
|
||||
# TYPE vllm:gpu_cache_usage gauge
|
||||
vllm:gpu_cache_usage{model_name="juggernaut"} 0.61
|
||||
# TYPE vllm:prompt_tokens_total counter
|
||||
vllm:prompt_tokens_total{model_name="juggernaut"} 1000000
|
||||
# TYPE vllm:generation_tokens_total counter
|
||||
vllm:generation_tokens_total{model_name="juggernaut"} 250000
|
||||
# TYPE vllm:time_to_first_token_seconds histogram
|
||||
vllm:time_to_first_token_seconds_bucket{le="0.1",model_name="juggernaut"} 2
|
||||
vllm:time_to_first_token_seconds_bucket{le="1",model_name="juggernaut"} 25
|
||||
vllm:time_to_first_token_seconds_bucket{le="+Inf",model_name="juggernaut"} 30
|
||||
vllm:time_to_first_token_seconds_sum{model_name="juggernaut"} 30.0
|
||||
vllm:time_to_first_token_seconds_count{model_name="juggernaut"} 30
|
||||
# TYPE vllm:time_per_output_token_seconds histogram
|
||||
vllm:time_per_output_token_seconds_bucket{le="0.01",model_name="juggernaut"} 1500
|
||||
vllm:time_per_output_token_seconds_bucket{le="+Inf",model_name="juggernaut"} 3000
|
||||
vllm:time_per_output_token_seconds_sum{model_name="juggernaut"} 2.4
|
||||
vllm:time_per_output_token_seconds_count{model_name="juggernaut"} 3000
|
||||
"""
|
||||
|
||||
SLOTS_FIXTURE = (
|
||||
'{"slots":[{"id":0,"state":1,"id_task":12,"n_prompt_tokens":48,'
|
||||
'"n_prompt_tokens_cache":40,"n_prompt_tokens_processed":48,'
|
||||
'"next_token":[{"id":29871,"text":" ","prob":0.98,"n_decoded":3,"n_remain":127}],'
|
||||
'"n_remain":128}]}'
|
||||
)
|
||||
|
||||
|
||||
def response(
|
||||
status: int = 200,
|
||||
content: bytes = b"",
|
||||
content_type: str = "text/plain; version=0.0.4",
|
||||
) -> httpx.Response:
|
||||
# A real httpx.Response rather than a mock: .text / .content / .reason_phrase
|
||||
# then behave exactly as app.fetch() expects. Attaching a request is mandatory
|
||||
# before a body is read.
|
||||
return httpx.Response(
|
||||
status,
|
||||
content=content,
|
||||
headers={"content-type": content_type},
|
||||
request=httpx.Request("GET", "http://upstream.test"),
|
||||
)
|
||||
|
||||
|
||||
class FakeUpstream:
|
||||
"""Stands in for httpx.AsyncClient and records what was requested."""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.routes: dict[str, object] = {}
|
||||
self.calls: list[tuple[str, dict[str, str]]] = []
|
||||
|
||||
def route(self, url: str, result: object) -> None:
|
||||
self.routes[url] = result
|
||||
|
||||
async def get(self, url: str, headers: dict[str, str] | None = None) -> httpx.Response:
|
||||
self.calls.append((url, dict(headers or {})))
|
||||
result = self.routes.get(url)
|
||||
if isinstance(result, Exception):
|
||||
raise result
|
||||
if result is None:
|
||||
return response(404, b'{"detail":"Not Found"}', "application/json")
|
||||
return result # type: ignore[return-value]
|
||||
|
||||
@property
|
||||
def requested_urls(self) -> list[str]:
|
||||
return [url for url, _headers in self.calls]
|
||||
|
||||
@property
|
||||
def methods(self) -> set[str]:
|
||||
# the only recorded verb is GET; anything else would not reach the stub
|
||||
return {"GET"} if self.calls else set()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def upstream(monkeypatch) -> FakeUpstream:
|
||||
fake = FakeUpstream()
|
||||
monkeypatch.setattr(app_module, "GUFO_BASE_URL", "http://gufo.test")
|
||||
monkeypatch.setattr(app_module, "VLLM_BASE_URL", "http://vllm.test")
|
||||
monkeypatch.setattr(app_module, "GUFO_API_KEY", "")
|
||||
monkeypatch.setattr(app_module, "VLLM_API_KEY", "")
|
||||
return fake
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def client(upstream: FakeUpstream):
|
||||
"""TestClient without `with` => lifespan skipped, so we inject the stub client."""
|
||||
test_client = TestClient(app_module.app)
|
||||
app_module.app.state.client = upstream
|
||||
return test_client
|
||||
Vendored
+49
@@ -0,0 +1,49 @@
|
||||
# Captured verbatim from http://deskwork.reeselink.com:8731/metrics (gufo 0.16.0, Qwen3.8 Flash Next,
|
||||
# --sessions 2, --speculative mtp). Re-capture after an engine upgrade; see tests/test_real_contracts.py.
|
||||
# Values are point-in-time and drift between captures — assert on names/sources, not exact numbers.
|
||||
|
||||
# HELP gufo_device_lost_total Confirmed GPU context losses
|
||||
# TYPE gufo_device_lost_total counter
|
||||
gufo_device_lost_total 0
|
||||
# HELP llamacpp:prompt_tokens_total Total prompt tokens processed, excluding cache hits
|
||||
# TYPE llamacpp:prompt_tokens_total counter
|
||||
llamacpp:prompt_tokens_total 374263
|
||||
# HELP llamacpp:prompt_tokens_cached_total Total prompt tokens reused from cache
|
||||
# TYPE llamacpp:prompt_tokens_cached_total counter
|
||||
llamacpp:prompt_tokens_cached_total 29650554
|
||||
# HELP llamacpp:prompt_seconds_total Prompt process time
|
||||
# TYPE llamacpp:prompt_seconds_total counter
|
||||
llamacpp:prompt_seconds_total 341.91072852000008
|
||||
# HELP llamacpp:tokens_predicted_total Total tokens generated
|
||||
# TYPE llamacpp:tokens_predicted_total counter
|
||||
llamacpp:tokens_predicted_total 173091
|
||||
# HELP llamacpp:tokens_predicted_seconds_total Predict process time
|
||||
# TYPE llamacpp:tokens_predicted_seconds_total counter
|
||||
llamacpp:tokens_predicted_seconds_total 5458.0845807790029
|
||||
# HELP llamacpp:n_tokens_max Largest observed n_tokens.
|
||||
# TYPE llamacpp:n_tokens_max counter
|
||||
llamacpp:n_tokens_max 252622
|
||||
# HELP llamacpp:spec_decode_num_drafts_total Total speculative verification rounds
|
||||
# TYPE llamacpp:spec_decode_num_drafts_total counter
|
||||
llamacpp:spec_decode_num_drafts_total 54515
|
||||
# HELP llamacpp:spec_decode_num_draft_tokens_total Total draft tokens proposed
|
||||
# TYPE llamacpp:spec_decode_num_draft_tokens_total counter
|
||||
llamacpp:spec_decode_num_draft_tokens_total 158803
|
||||
# HELP llamacpp:spec_decode_num_accepted_tokens_total Total draft tokens accepted
|
||||
# TYPE llamacpp:spec_decode_num_accepted_tokens_total counter
|
||||
llamacpp:spec_decode_num_accepted_tokens_total 117552
|
||||
# HELP llamacpp:prompt_tokens_seconds Prompt processing speed in tokens per second
|
||||
# TYPE llamacpp:prompt_tokens_seconds gauge
|
||||
llamacpp:prompt_tokens_seconds 820.2841172907755
|
||||
# HELP llamacpp:predicted_tokens_seconds Generation speed in tokens per second
|
||||
# TYPE llamacpp:predicted_tokens_seconds gauge
|
||||
llamacpp:predicted_tokens_seconds 43.015060732721352
|
||||
# HELP llamacpp:requests_processing Number of admitted requests, including cache preparation
|
||||
# TYPE llamacpp:requests_processing gauge
|
||||
llamacpp:requests_processing 0
|
||||
# HELP llamacpp:requests_deferred Number of requests waiting for a session
|
||||
# TYPE llamacpp:requests_deferred gauge
|
||||
llamacpp:requests_deferred 0
|
||||
# HELP llamacpp:kv_cache_usage_ratio In-flight prompt and generated tokens over sessions times context; excludes retained cache
|
||||
# TYPE llamacpp:kv_cache_usage_ratio gauge
|
||||
llamacpp:kv_cache_usage_ratio 0
|
||||
Vendored
+1
@@ -0,0 +1 @@
|
||||
[{"id":0,"n_ctx":262144,"speculative":true,"is_processing":false,"id_task":-1,"task_id":-1,"state":0,"n_prompt_tokens":0,"n_prompt_tokens_cache":0,"n_prompt_tokens_processed":0,"prompt":"","next_token":[{"has_next_token":false,"has_new_line":false,"n_remain":-1,"n_decoded":0}],"model":"turbo"},{"id":1,"n_ctx":262144,"speculative":true,"is_processing":false,"id_task":-1,"task_id":-1,"state":0,"n_prompt_tokens":0,"n_prompt_tokens_cache":0,"n_prompt_tokens_processed":0,"prompt":"","next_token":[{"has_next_token":false,"has_new_line":false,"n_remain":-1,"n_decoded":0}],"model":"turbo"}]
|
||||
Vendored
+163
@@ -0,0 +1,163 @@
|
||||
# Captured verbatim from http://driveripper.reeselink.com:8000/metrics (vllm-radiance).
|
||||
# Trimmed only by dropping series the dashboard does not read; every kept line is real.
|
||||
process_cpu_seconds_total 240.62
|
||||
# HELP vllm:num_requests_running Number of requests in model execution batches.
|
||||
# TYPE vllm:num_requests_running gauge
|
||||
vllm:num_requests_running{engine="0",model_name="juggernaut"} 2.0
|
||||
# HELP vllm:num_requests_waiting Number of requests waiting to be processed.
|
||||
# TYPE vllm:num_requests_waiting gauge
|
||||
vllm:num_requests_waiting{engine="0",model_name="juggernaut"} 0.0
|
||||
# HELP vllm:num_requests_waiting_by_reason Number of waiting requests by reason. Reason labels: 'capacity' = waiting for scheduling capacity; 'deferred' = deferred by transient constraints (LoRA budget, KV transfer, blocked status). Sum of all reasons equals vllm:num_requests_waiting.
|
||||
# TYPE vllm:num_requests_waiting_by_reason gauge
|
||||
vllm:num_requests_waiting_by_reason{engine="0",model_name="juggernaut",reason="capacity"} 0.0
|
||||
vllm:num_requests_waiting_by_reason{engine="0",model_name="juggernaut",reason="deferred"} 0.0
|
||||
# HELP vllm:kv_cache_usage_perc KV-cache usage. 1 means 100 percent usage.
|
||||
# TYPE vllm:kv_cache_usage_perc gauge
|
||||
vllm:kv_cache_usage_perc{engine="0",model_name="juggernaut"} 0.39790575916230364
|
||||
# HELP vllm:prefix_cache_queries_total Prefix cache queries, in terms of number of queried tokens.
|
||||
# TYPE vllm:prefix_cache_queries_total counter
|
||||
vllm:prefix_cache_queries_total{engine="0",model_name="juggernaut"} 8.582355e+06
|
||||
# HELP vllm:prefix_cache_hits_total Prefix cache hits, in terms of number of cached tokens.
|
||||
# TYPE vllm:prefix_cache_hits_total counter
|
||||
vllm:prefix_cache_hits_total{engine="0",model_name="juggernaut"} 4.464096e+06
|
||||
# HELP vllm:num_preemptions_total Cumulative number of preemption from the engine.
|
||||
# TYPE vllm:num_preemptions_total counter
|
||||
vllm:num_preemptions_total{engine="0",model_name="juggernaut"} 0.0
|
||||
# HELP vllm:prompt_tokens_total Number of prefill tokens processed.
|
||||
# TYPE vllm:prompt_tokens_total counter
|
||||
vllm:prompt_tokens_total{engine="0",model_name="juggernaut"} 8.582355e+06
|
||||
# HELP vllm:generation_tokens_total Number of generation tokens processed.
|
||||
# TYPE vllm:generation_tokens_total counter
|
||||
vllm:generation_tokens_total{engine="0",model_name="juggernaut"} 339918.0
|
||||
vllm:request_success_total{engine="0",finished_reason="stop",model_name="juggernaut"} 2145.0
|
||||
vllm:request_success_total{engine="0",finished_reason="length",model_name="juggernaut"} 0.0
|
||||
vllm:request_success_total{engine="0",finished_reason="abort",model_name="juggernaut"} 0.0
|
||||
vllm:request_success_total{engine="0",finished_reason="error",model_name="juggernaut"} 0.0
|
||||
vllm:request_success_total{engine="0",finished_reason="repetition",model_name="juggernaut"} 0.0
|
||||
# HELP vllm:time_to_first_token_seconds Histogram of time to first token in seconds.
|
||||
# TYPE vllm:time_to_first_token_seconds histogram
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="0.001",model_name="juggernaut"} 0.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="0.005",model_name="juggernaut"} 0.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="0.01",model_name="juggernaut"} 0.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="0.02",model_name="juggernaut"} 0.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="0.04",model_name="juggernaut"} 0.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="0.06",model_name="juggernaut"} 0.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="0.08",model_name="juggernaut"} 3.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="0.1",model_name="juggernaut"} 3.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="0.25",model_name="juggernaut"} 341.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="0.5",model_name="juggernaut"} 724.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="0.75",model_name="juggernaut"} 1076.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="1.0",model_name="juggernaut"} 1291.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="2.5",model_name="juggernaut"} 2114.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="5.0",model_name="juggernaut"} 2148.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="7.5",model_name="juggernaut"} 2149.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="10.0",model_name="juggernaut"} 2149.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="20.0",model_name="juggernaut"} 2149.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="40.0",model_name="juggernaut"} 2149.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="80.0",model_name="juggernaut"} 2149.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="160.0",model_name="juggernaut"} 2149.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="640.0",model_name="juggernaut"} 2149.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="2560.0",model_name="juggernaut"} 2149.0
|
||||
vllm:time_to_first_token_seconds_bucket{engine="0",le="+Inf",model_name="juggernaut"} 2149.0
|
||||
vllm:time_to_first_token_seconds_count{engine="0",model_name="juggernaut"} 2149.0
|
||||
vllm:time_to_first_token_seconds_sum{engine="0",model_name="juggernaut"} 1861.7778627872467
|
||||
# HELP vllm:inter_token_latency_seconds Histogram of inter-token latency in seconds.
|
||||
# TYPE vllm:inter_token_latency_seconds histogram
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="0.01",model_name="juggernaut"} 0.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="0.025",model_name="juggernaut"} 0.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="0.05",model_name="juggernaut"} 327550.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="0.075",model_name="juggernaut"} 337018.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="0.1",model_name="juggernaut"} 337051.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="0.15",model_name="juggernaut"} 337141.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="0.2",model_name="juggernaut"} 337229.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="0.3",model_name="juggernaut"} 337313.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="0.4",model_name="juggernaut"} 337368.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="0.5",model_name="juggernaut"} 337430.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="0.75",model_name="juggernaut"} 337631.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="1.0",model_name="juggernaut"} 337651.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="2.5",model_name="juggernaut"} 337769.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="5.0",model_name="juggernaut"} 337769.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="7.5",model_name="juggernaut"} 337769.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="10.0",model_name="juggernaut"} 337769.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="20.0",model_name="juggernaut"} 337769.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="40.0",model_name="juggernaut"} 337769.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="80.0",model_name="juggernaut"} 337769.0
|
||||
vllm:inter_token_latency_seconds_bucket{engine="0",le="+Inf",model_name="juggernaut"} 337769.0
|
||||
vllm:inter_token_latency_seconds_count{engine="0",model_name="juggernaut"} 337769.0
|
||||
vllm:inter_token_latency_seconds_sum{engine="0",model_name="juggernaut"} 12194.350635488052
|
||||
# HELP vllm:request_queue_time_seconds Histogram of time spent in WAITING phase for request.
|
||||
# TYPE vllm:request_queue_time_seconds histogram
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="0.3",model_name="juggernaut"} 2143.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="0.5",model_name="juggernaut"} 2143.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="0.8",model_name="juggernaut"} 2143.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="1.0",model_name="juggernaut"} 2143.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="1.5",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="2.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="2.5",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="5.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="10.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="15.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="20.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="30.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="40.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="50.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="60.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="120.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="240.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="480.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="960.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="1920.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="7680.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_bucket{engine="0",le="+Inf",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_count{engine="0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_queue_time_seconds_sum{engine="0",model_name="juggernaut"} 2.707464350562077
|
||||
vllm:cache_config_info{_block_size_resolved="True",block_size="1568",cache_dtype="fp8",enable_mamba_fine_grained_prefix_cache="False",enable_prefix_caching="True",engine="0",gpu_memory_utilization="0.95",is_attention_free="False",kv_cache_dtype_skip_layers="[]",kv_cache_layout="None",kv_cache_max_concurrency="2.1333333333333333",kv_cache_memory_bytes="None",kv_cache_size_tokens="279620",kv_offloading_backend="native",kv_offloading_size="None",kv_sharing_fast_prefill="False",mamba_block_size="1568",mamba_cache_dtype="auto",mamba_cache_mode="align",mamba_page_size_padded="None",mamba_ssm_cache_dtype="float32",num_cpu_blocks="None",num_gpu_blocks="192",num_gpu_blocks_override="None",prefix_cache_retention_interval="0",prefix_caching_hash_algo="sha256",prefix_match_unit="None",replayssm_buffer_len="16",skip_page_size_padded="None",sliding_window="None",use_kda_recoverssm="False",use_replayssm="False",user_specified_block_size="False",user_specified_mamba_block_size="False"} 1.0
|
||||
|
||||
# prefill series (used for the lifetime prompt-speed derivation)
|
||||
# HELP vllm:request_prompt_tokens Number of prefill tokens processed.
|
||||
# TYPE vllm:request_prompt_tokens histogram
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="1.0",model_name="juggernaut"} 0.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="2.0",model_name="juggernaut"} 0.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="5.0",model_name="juggernaut"} 0.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="10.0",model_name="juggernaut"} 0.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="20.0",model_name="juggernaut"} 5.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="50.0",model_name="juggernaut"} 5.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="100.0",model_name="juggernaut"} 20.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="200.0",model_name="juggernaut"} 71.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="500.0",model_name="juggernaut"} 361.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="1000.0",model_name="juggernaut"} 638.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="2000.0",model_name="juggernaut"} 1134.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="5000.0",model_name="juggernaut"} 1943.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="10000.0",model_name="juggernaut"} 2026.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="20000.0",model_name="juggernaut"} 2072.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="50000.0",model_name="juggernaut"} 2112.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="100000.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prompt_tokens_bucket{engine="0",le="+Inf",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prompt_tokens_count{engine="0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prompt_tokens_sum{engine="0",model_name="juggernaut"} 8.441975e+06
|
||||
# HELP vllm:request_prefill_time_seconds Histogram of time spent in PREFILL phase for request.
|
||||
# TYPE vllm:request_prefill_time_seconds histogram
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="0.3",model_name="juggernaut"} 472.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="0.5",model_name="juggernaut"} 742.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="0.8",model_name="juggernaut"} 1138.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="1.0",model_name="juggernaut"} 1318.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="1.5",model_name="juggernaut"} 1976.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="2.0",model_name="juggernaut"} 2078.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="2.5",model_name="juggernaut"} 2117.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="5.0",model_name="juggernaut"} 2144.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="10.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="15.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="20.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="30.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="40.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="50.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="60.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="120.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="240.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="480.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="960.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="1920.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="7680.0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_bucket{engine="0",le="+Inf",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_count{engine="0",model_name="juggernaut"} 2145.0
|
||||
vllm:request_prefill_time_seconds_sum{engine="0",model_name="juggernaut"} 1796.7229758803733
|
||||
@@ -0,0 +1,252 @@
|
||||
"""Prometheus text parser + per-engine field maps.
|
||||
|
||||
This is the code most likely to silently rot: an engine renames a series and the
|
||||
dashboard must degrade to `n/a`, not crash and not misreport. The aggregation
|
||||
rules (sum / max / avg) are asserted explicitly because getting them wrong is
|
||||
how a cache ratio ends up over 100%.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
|
||||
import pytest
|
||||
|
||||
from app import Prom, parse_gufo_metrics, parse_vllm_metrics
|
||||
from tests.conftest import GUFO_FIXTURE, VLLM_FIXTURE
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Prom
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_ignores_comments_help_and_type_lines() -> None:
|
||||
p = Prom("# HELP a hi\n# TYPE a gauge\na 1\n\n")
|
||||
assert list(p.series) == ["a"]
|
||||
assert p.parse_errors == 0
|
||||
|
||||
|
||||
def test_label_block_with_brace_and_escape() -> None:
|
||||
p = Prom(r'a{x="a}b\"c"} 5')
|
||||
assert p.series["a"][0][0] == {"x": 'a}b"c'}
|
||||
assert p.series["a"][0][1] == 5.0
|
||||
|
||||
|
||||
def test_tolerates_space_before_labels_and_trailing_comment() -> None:
|
||||
p = Prom('a {m="x"} 1 # a comment\nb 2 1700000000000\n')
|
||||
assert p.scalar(["a"]) == 1.0
|
||||
assert p.scalar(["b"]) == 2.0
|
||||
|
||||
|
||||
def test_counts_garbage_instead_of_raising() -> None:
|
||||
p = Prom("this is not prometheus\nalso bad x y z\n")
|
||||
assert p.series == {}
|
||||
assert p.parse_errors == 2
|
||||
|
||||
|
||||
def test_non_finite_values_are_dropped_from_aggregates() -> None:
|
||||
p = Prom("a 1\na nan\na +Inf\na 3\n")
|
||||
assert p.scalar(["a"]) == 4.0 # sum of the finite ones only
|
||||
assert p.scalar(["a"], "max") == 3.0
|
||||
assert p.scalar(["a"], "avg") == 2.0
|
||||
|
||||
|
||||
def test_absent_series_is_none_never_zero() -> None:
|
||||
p = Prom("other 1\n")
|
||||
assert p.scalar(["nope"]) is None
|
||||
assert p.resolve(["nope", "other"]) == "other"
|
||||
|
||||
|
||||
def test_alias_resolution_prefers_first_present() -> None:
|
||||
p = Prom("llama:requests_processing 7\n")
|
||||
assert p.resolve(["llamacpp:requests_processing", "llama:requests_processing"]) == (
|
||||
"llama:requests_processing"
|
||||
)
|
||||
|
||||
|
||||
def test_by_label_grouping() -> None:
|
||||
p = Prom('w{reason="capacity"} 2\nw{reason="capacity"} 3\nw{reason="other"} 1\n')
|
||||
assert p.by_label(["w"], "reason") == {"capacity": 5.0, "other": 1.0}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Histograms
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_histogram_mean_and_p95() -> None:
|
||||
p = Prom('b_bucket{le="1"} 1\nb_bucket{le="10"} 10\nb_bucket{le="+Inf"} 10\nb_sum 40\nb_count 10\n')
|
||||
h = p.histogram(["b"])
|
||||
assert h is not None
|
||||
assert h["mean"] == 4.0
|
||||
# 95th of 10 observations is the 9.5th, interpolated inside the (1,10] bucket
|
||||
assert h["p95"] == pytest.approx(9.5)
|
||||
|
||||
|
||||
def test_histogram_p95_when_mass_exceeds_the_last_finite_bucket() -> None:
|
||||
# almost everything above the last real bound: report that bound, never +Inf
|
||||
p = Prom('b_bucket{le="1"} 5\nb_bucket{le="10"} 9\nb_bucket{le="+Inf"} 10\nb_sum 40\nb_count 10\n')
|
||||
assert p.histogram(["b"])["p95"] == pytest.approx(10.0)
|
||||
|
||||
|
||||
def test_histogram_resolves_without_a_bare_series_line() -> None:
|
||||
# real expositions never emit a series named after the histogram itself
|
||||
p = Prom('vllm:x_sum 6\nvllm:x_count 3\nvllm:x_bucket{le="+Inf"} 3\n')
|
||||
assert p.histogram(["vllm:x"])["mean"] == 2.0
|
||||
|
||||
|
||||
def test_histogram_p95_suppressed_across_label_sets() -> None:
|
||||
p = Prom(
|
||||
'c_sum{dp="0"} 10\nc_sum{dp="1"} 20\n'
|
||||
'c_count{dp="0"} 2\nc_count{dp="1"} 3\n'
|
||||
'c_bucket{dp="0",le="1"} 2\nc_bucket{dp="1",le="1"} 3\n'
|
||||
)
|
||||
h = p.histogram(["c"])
|
||||
assert h["mean"] == 6.0 # still aggregated
|
||||
assert h["p95"] is None # but blending label sets would invent a percentile
|
||||
|
||||
|
||||
def test_histogram_missing_is_none() -> None:
|
||||
assert Prom("unrelated 1\n").histogram(["b"]) is None
|
||||
|
||||
|
||||
def test_quantile_on_empty_buckets() -> None:
|
||||
from app import _quantile
|
||||
|
||||
assert _quantile([], 0, 0.95) is None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# gufo field map
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_gufo_field_map() -> None:
|
||||
m = parse_gufo_metrics(GUFO_FIXTURE)
|
||||
assert m["requests_processing"] == {"value": 2.0, "source": "llamacpp:requests_processing", "derived": False}
|
||||
assert m["requests_deferred"]["value"] == 1.0
|
||||
assert m["kv_cache_usage_ratio"]["value"] == pytest.approx(0.42)
|
||||
assert m["tokens_prompt_total"]["value"] == 12345.0
|
||||
assert m["tokens_generated_total"]["value"] == 6789.0
|
||||
assert m["prompt_tokens_per_second"]["value"] == pytest.approx(1900.5)
|
||||
assert m["prompt_tokens_per_second_native"]["value"] == pytest.approx(1900.5)
|
||||
# no rates injected -> the cell falls back to the scraped single-request gauge
|
||||
assert m["decode_tokens_per_second"]["value"] == pytest.approx(31.5)
|
||||
assert m["decode_tokens_per_second"]["source"] == "llamacpp:decode_tokens_per_second"
|
||||
assert m["decode_tokens_per_second_native"]["value"] == pytest.approx(31.5)
|
||||
assert m["tokens_total"] == {
|
||||
"value": 19134.0, # no cached series in this fixture -> prompt + generated
|
||||
"source": "computed:prompt+cached+generated",
|
||||
"derived": False,
|
||||
}
|
||||
|
||||
|
||||
def test_gufo_total_counts_cache_hits_once() -> None:
|
||||
# gufo's prompt_tokens_total EXCLUDES cache hits, so a naive sum under-reports
|
||||
m = parse_gufo_metrics(
|
||||
"llamacpp:prompt_tokens_total 100\n"
|
||||
"llamacpp:prompt_tokens_cached_total 900\n"
|
||||
"llamacpp:tokens_predicted_total 50\n"
|
||||
)
|
||||
assert m["tokens_total"]["value"] == 1050.0
|
||||
assert m["prompt_cache_hit_ratio"]["value"] == pytest.approx(0.9)
|
||||
|
||||
|
||||
def test_gufo_spec_accept_rate_absent_without_the_series() -> None:
|
||||
assert parse_gufo_metrics(GUFO_FIXTURE)["spec_accept_rate"]["value"] is None
|
||||
|
||||
|
||||
def test_gufo_spec_accept_rate_from_real_names() -> None:
|
||||
m = parse_gufo_metrics(
|
||||
"llamacpp:spec_decode_num_drafts_total 100\n"
|
||||
"llamacpp:spec_decode_num_draft_tokens_total 400\n"
|
||||
"llamacpp:spec_decode_num_accepted_tokens_total 200\n"
|
||||
)
|
||||
assert m["spec_accept_rate"]["value"] == pytest.approx(0.5)
|
||||
assert m["spec_accept_rate"]["derived"] is True
|
||||
|
||||
|
||||
def test_gufo_kv_ratio_uses_max_not_sum() -> None:
|
||||
# two contexts at 0.4 and 0.5 must report 0.5, never 0.9
|
||||
m = parse_gufo_metrics("llamacpp:kv_cache_usage_ratio 0.4\nllamacpp:kv_cache_usage_ratio 0.5\n")
|
||||
assert m["kv_cache_usage_ratio"]["value"] == pytest.approx(0.5)
|
||||
assert m["kv_cache_usage_ratio"]["value"] <= 1.0
|
||||
|
||||
|
||||
def test_gufo_empty_exposition_is_all_none() -> None:
|
||||
m = parse_gufo_metrics("")
|
||||
assert all(cell["value"] is None for cell in m.values())
|
||||
assert m["tokens_total"]["value"] is None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# vLLM field map
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_vllm_field_map() -> None:
|
||||
m = parse_vllm_metrics(VLLM_FIXTURE)
|
||||
assert m["num_requests_running"]["value"] == 3.0
|
||||
assert m["num_requests_waiting"]["value"] == 5.0
|
||||
assert m["waiting_by_reason"]["value"] == {"capacity": 5.0, "defragmentation": 0.0}
|
||||
assert m["waiting_capacity"]["value"] == 5.0
|
||||
assert m["gpu_cache_usage"]["value"] == pytest.approx(0.61)
|
||||
assert m["prompt_tokens_total"]["value"] == 1_000_000.0
|
||||
assert m["generation_tokens_total"]["value"] == 250_000.0
|
||||
assert m["time_to_first_token_seconds"]["mean"] == pytest.approx(1.0)
|
||||
assert m["time_to_first_token_seconds__mean"]["value"] == pytest.approx(1.0)
|
||||
assert m["request_queue_time_seconds"] is None # absent on this server -> n/a
|
||||
|
||||
|
||||
def test_vllm_decode_throughput_derived_from_tpot() -> None:
|
||||
m = parse_vllm_metrics(VLLM_FIXTURE)
|
||||
tpot = m["time_per_output_token_seconds"]["mean"]
|
||||
assert m["decode_tokens_per_second"]["derived"] is True
|
||||
assert m["decode_tokens_per_second"]["source"] == "computed:running/tpot"
|
||||
assert m["decode_tokens_per_second"]["value"] == pytest.approx(3 / tpot)
|
||||
assert m["per_request_decode_tokens_per_second"]["value"] == pytest.approx(1 / tpot)
|
||||
|
||||
|
||||
def test_vllm_native_throughput_wins_over_derivation() -> None:
|
||||
m = parse_vllm_metrics(
|
||||
'vllm:num_requests_running{model_name="m"} 2\n'
|
||||
'vllm:generation_throughput{model_name="m"} 77.5\n'
|
||||
"vllm:time_per_output_token_seconds_sum 1.0\n"
|
||||
"vllm:time_per_output_token_seconds_count 100\n"
|
||||
)
|
||||
assert m["decode_tokens_per_second"] == {
|
||||
"value": 77.5,
|
||||
"source": "vllm:generation_throughput",
|
||||
"derived": False,
|
||||
}
|
||||
# per-request figure is still useful next to a native aggregate
|
||||
assert m["per_request_decode_tokens_per_second"]["value"] == pytest.approx(100.0)
|
||||
|
||||
|
||||
def test_vllm_inter_token_latency_alias() -> None:
|
||||
m = parse_vllm_metrics(
|
||||
"vllm:inter_token_latency_seconds_sum 4\nvllm:inter_token_latency_seconds_count 8\n"
|
||||
)
|
||||
assert m["time_per_output_token_seconds"]["source"] == "vllm:inter_token_latency_seconds"
|
||||
assert m["time_per_output_token_seconds"]["mean"] == pytest.approx(0.5)
|
||||
|
||||
|
||||
def test_partial_exposition_totals_do_not_raise() -> None:
|
||||
"""Regression: `None + value` used to 500 the endpoint when one of the two token
|
||||
counters was missing — a renamed series must shrink the total, not break it."""
|
||||
assert parse_vllm_metrics("vllm:prompt_tokens_total 100\n")["tokens_total"]["value"] == 100.0
|
||||
assert parse_vllm_metrics("vllm:generation_tokens_total 7\n")["tokens_total"]["value"] == 7.0
|
||||
|
||||
|
||||
def test_vllm_empty_exposition_is_all_none() -> None:
|
||||
m = parse_vllm_metrics("# nothing here\nunrelated_metric_total 1\n")
|
||||
for key in (
|
||||
"num_requests_running",
|
||||
"num_requests_waiting",
|
||||
"gpu_cache_usage",
|
||||
"prompt_tokens_total",
|
||||
"decode_tokens_per_second",
|
||||
"tokens_total",
|
||||
):
|
||||
assert m[key]["value"] is None, key
|
||||
assert m["time_to_first_token_seconds"] is None
|
||||
assert m["waiting_by_reason"]["value"] == {}
|
||||
|
||||
|
||||
def test_values_are_finite_or_none() -> None:
|
||||
for cell in parse_vllm_metrics(VLLM_FIXTURE).values():
|
||||
if isinstance(cell, dict) and "value" in cell and isinstance(cell["value"], float):
|
||||
assert math.isfinite(cell["value"])
|
||||
@@ -0,0 +1,299 @@
|
||||
"""Proxy routes: read-only guarantee, upstream failure translation, JSON shape.
|
||||
|
||||
Nothing here touches a socket — see conftest.FakeUpstream. The point is to pin
|
||||
the contract the browser depends on: what passes through, what becomes JSON, and
|
||||
exactly how an engine outage shows up.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
|
||||
import app as app_module
|
||||
from tests.conftest import GUFO_FIXTURE, SLOTS_FIXTURE, VLLM_FIXTURE, response
|
||||
|
||||
API_PATHS = ["/api/gufo/slots", "/api/gufo/metrics", "/api/vllm/metrics", "/api/config"]
|
||||
GUFO_PATHS = ["/api/gufo/slots", "/api/gufo/metrics"]
|
||||
VLLM_PATHS = ["/api/vllm/metrics"]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# read-only by construction
|
||||
# --------------------------------------------------------------------------- #
|
||||
@pytest.mark.parametrize("method", ["post", "put", "patch", "delete"])
|
||||
@pytest.mark.parametrize("path", API_PATHS)
|
||||
def test_no_writing_verbs(client, method: str, path: str) -> None:
|
||||
assert getattr(client, method)(path).status_code == 405
|
||||
|
||||
|
||||
def test_only_get_reaches_upstream(client, upstream) -> None:
|
||||
upstream.route("http://gufo.test/slots", response(200, SLOTS_FIXTURE.encode()))
|
||||
client.get("/api/gufo/slots")
|
||||
upstream.route("http://gufo.test/metrics", response(200, GUFO_FIXTURE.encode()))
|
||||
client.get("/api/gufo/metrics")
|
||||
assert upstream.methods == {"GET"}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# /api/gufo/slots — passthrough
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_slots_pass_through_byte_for_byte(client, upstream) -> None:
|
||||
upstream.route(
|
||||
"http://gufo.test/slots", response(200, SLOTS_FIXTURE.encode(), "application/json")
|
||||
)
|
||||
r = client.get("/api/gufo/slots")
|
||||
assert r.status_code == 200
|
||||
assert r.content.decode() == SLOTS_FIXTURE # not re-serialised, not reformatted
|
||||
assert r.headers["content-type"].startswith("application/json")
|
||||
assert "no-store" in r.headers["cache-control"]
|
||||
assert r.json()["slots"][0]["id_task"] == 12
|
||||
|
||||
|
||||
def test_slots_accepts_a_bare_array_payload(client, upstream) -> None:
|
||||
"""The deployed gufo returns `[{...}]`, not `{"slots": [...]}`. The proxy must not
|
||||
care — it forwards whatever JSON the engine sent, unchanged."""
|
||||
bare = '[{"id":0,"is_processing":true,"id_task":7,"next_token":[{"n_decoded":3,"n_remain":9}]}]'
|
||||
upstream.route("http://gufo.test/slots", response(200, bare.encode(), "application/json"))
|
||||
r = client.get("/api/gufo/slots")
|
||||
assert r.status_code == 200
|
||||
assert r.json() == json.loads(bare)
|
||||
assert isinstance(r.json(), list)
|
||||
|
||||
|
||||
def test_upstream_content_type_is_passed_through(client, upstream) -> None:
|
||||
# whatever gufo claims, the browser gets the same thing — no re-typing
|
||||
upstream.route(
|
||||
"http://gufo.test/slots",
|
||||
response(200, SLOTS_FIXTURE.encode(), "application/stream+json"),
|
||||
)
|
||||
assert client.get("/api/gufo/slots").headers["content-type"] == "application/stream+json"
|
||||
|
||||
|
||||
def test_slots_falls_back_to_the_alias(client, upstream) -> None:
|
||||
# /slots 404s (the stub's default for unrouted URLs), the alias answers
|
||||
upstream.route("http://gufo.test/v1/slots", response(200, SLOTS_FIXTURE.encode(), "application/json"))
|
||||
r = client.get("/api/gufo/slots")
|
||||
assert r.status_code == 200
|
||||
assert upstream.requested_urls == ["http://gufo.test/slots", "http://gufo.test/v1/slots"]
|
||||
|
||||
|
||||
def test_slots_reports_missing_endpoints(client, upstream) -> None:
|
||||
r = client.get("/api/gufo/slots")
|
||||
assert r.status_code == 502
|
||||
assert "/slots" in r.json()["detail"] and "/v1/slots" in r.json()["detail"]
|
||||
|
||||
|
||||
def test_slots_does_not_swallow_a_server_error(client, upstream) -> None:
|
||||
upstream.route("http://gufo.test/slots", response(500, b"boom"))
|
||||
r = client.get("/api/gufo/slots")
|
||||
assert r.status_code == 502 # engine is unhealthy: that is a bad gateway, not a fallback
|
||||
assert "HTTP 500" in r.json()["detail"]
|
||||
assert upstream.requested_urls == ["http://gufo.test/slots"] # no pointless alias retry
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# /metrics — Prometheus -> JSON
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_gufo_metrics_endpoint(client, upstream) -> None:
|
||||
upstream.route("http://gufo.test/metrics", response(200, GUFO_FIXTURE.encode()))
|
||||
body = client.get("/api/gufo/metrics").json()
|
||||
assert body["ok"] is True
|
||||
assert body["engine"] == "gufo"
|
||||
assert body["upstream"] == "http://gufo.test/metrics"
|
||||
assert body["series"] == 8 # 7 llamacpp:* series + the legacy llama:* one
|
||||
assert body["unparsed"] == 0
|
||||
assert body["metrics"]["requests_processing"]["value"] == 2.0
|
||||
assert body["scraped_at"] > 0
|
||||
|
||||
|
||||
def test_vllm_metrics_endpoint(client, upstream) -> None:
|
||||
upstream.route("http://vllm.test/metrics", response(200, VLLM_FIXTURE.encode()))
|
||||
m = client.get("/api/vllm/metrics").json()["metrics"]
|
||||
assert m["num_requests_running"]["value"] == 3.0
|
||||
assert m["waiting_capacity"]["value"] == 5.0
|
||||
assert m["time_to_first_token_seconds"]["mean"] == pytest.approx(1.0)
|
||||
|
||||
|
||||
def test_garbage_exposition_is_ok_with_no_series(client, upstream) -> None:
|
||||
upstream.route("http://vllm.test/metrics", response(200, b"not prometheus at all\n"))
|
||||
body = client.get("/api/vllm/metrics").json()
|
||||
assert body["ok"] is True
|
||||
assert body["series"] == 0 and body["unparsed"] == 1
|
||||
assert body["metrics"]["num_requests_running"]["value"] is None # renders as n/a
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# engine outage translation (the per-engine health contract)
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_unreachable_engine_is_a_502_with_a_readable_detail(client, upstream) -> None:
|
||||
upstream.route("http://vllm.test/metrics", httpx.ConnectError("connection refused"))
|
||||
r = client.get("/api/vllm/metrics")
|
||||
assert r.status_code == 502
|
||||
assert r.json()["ok"] is False
|
||||
assert "unreachable" in r.json()["detail"]
|
||||
|
||||
|
||||
def test_slow_engine_is_a_504(client, upstream) -> None:
|
||||
upstream.route("http://gufo.test/metrics", httpx.TimeoutException("too slow"))
|
||||
r = client.get("/api/gufo/metrics")
|
||||
assert r.status_code == 504
|
||||
assert "timed out" in r.json()["detail"]
|
||||
|
||||
|
||||
def test_one_engine_being_down_does_not_affect_the_other(client, upstream) -> None:
|
||||
upstream.route("http://gufo.test/slots", response(200, SLOTS_FIXTURE.encode(), "application/json"))
|
||||
upstream.route("http://vllm.test/metrics", httpx.ConnectError("connection refused"))
|
||||
assert client.get("/api/vllm/metrics").status_code == 502
|
||||
assert client.get("/api/gufo/slots").status_code == 200
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# auth + config
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_no_authorization_header_without_keys(client, upstream) -> None:
|
||||
upstream.route("http://gufo.test/slots", response(200, b"{}", "application/json"))
|
||||
upstream.route("http://vllm.test/metrics", response(200, b"", "text/plain"))
|
||||
client.get("/api/gufo/slots")
|
||||
client.get("/api/vllm/metrics")
|
||||
assert all("authorization" not in headers for _url, headers in upstream.calls)
|
||||
|
||||
|
||||
def test_bearer_tokens_are_per_engine(client, upstream, monkeypatch) -> None:
|
||||
monkeypatch.setattr(app_module, "GUFO_API_KEY", "sk-gufo")
|
||||
monkeypatch.setattr(app_module, "VLLM_API_KEY", "sk-vllm")
|
||||
upstream.route("http://gufo.test/slots", response(200, b"{}", "application/json"))
|
||||
upstream.route("http://vllm.test/metrics", response(200, b"", "text/plain"))
|
||||
client.get("/api/gufo/slots")
|
||||
client.get("/api/vllm/metrics")
|
||||
assert upstream.calls[0][1]["Authorization"] == "Bearer sk-gufo"
|
||||
assert upstream.calls[1][1]["Authorization"] == "Bearer sk-vllm"
|
||||
|
||||
|
||||
def test_config_never_leaks_keys(client, monkeypatch) -> None:
|
||||
monkeypatch.setattr(app_module, "GUFO_API_KEY", "sk-super-secret")
|
||||
r = client.get("/api/config")
|
||||
assert "sk-super-secret" not in r.text
|
||||
body = r.json()
|
||||
assert body["gufo_auth"] is True and body["vllm_auth"] is False
|
||||
assert body["poll_ms"] == app_module.POLL_MS
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# opt-in engines: an unset *_BASE_URL means this deployment does not run that server
|
||||
# --------------------------------------------------------------------------- #
|
||||
@pytest.mark.parametrize(
|
||||
"raw, expected",
|
||||
[
|
||||
("http://box:1", "http://box:1"),
|
||||
("http://box:1/", "http://box:1"), # never a doubled //metrics
|
||||
(" http://box:1 ", "http://box:1"),
|
||||
("", ""), # set-but-empty is 'not configured', not 'http:///metrics'
|
||||
(" ", ""),
|
||||
],
|
||||
)
|
||||
def test_base_url_env_parsing(monkeypatch, raw: str, expected: str) -> None:
|
||||
monkeypatch.setenv("TEST_ENGINE_BASE_URL", raw)
|
||||
assert app_module._base_url_from_env("TEST_ENGINE_BASE_URL") == expected
|
||||
# and unset means the same as blank: no default host is invented
|
||||
monkeypatch.delenv("TEST_ENGINE_BASE_URL")
|
||||
assert app_module._base_url_from_env("TEST_ENGINE_BASE_URL") == ""
|
||||
|
||||
|
||||
@pytest.mark.parametrize("path", GUFO_PATHS + VLLM_PATHS)
|
||||
def test_unset_base_url_is_404_not_configured(client, upstream, monkeypatch, path: str) -> None:
|
||||
"""A server nobody runs must not look like a server that is down: no upstream call,
|
||||
no 502, and a detail that names the env var to set."""
|
||||
engine = "gufo" if "gufo" in path else "vllm"
|
||||
monkeypatch.setattr(app_module, f"{engine.upper()}_BASE_URL", "")
|
||||
r = client.get(path)
|
||||
assert r.status_code == 404
|
||||
body = r.json()
|
||||
assert body["ok"] is False
|
||||
assert body["configured"] is False
|
||||
assert body["engine"] == engine
|
||||
assert f"{engine.upper()}_BASE_URL" in body["detail"]
|
||||
assert "no such route" not in body["detail"] # distinct from an unknown path
|
||||
assert upstream.requested_urls == [] # nothing was dialled
|
||||
|
||||
|
||||
def test_one_configured_engine_is_unaffected_by_the_other_being_absent(
|
||||
client, upstream, monkeypatch
|
||||
) -> None:
|
||||
"""The vLLM-only recipient of this repo gets a working dashboard, not a half-broken one."""
|
||||
monkeypatch.setattr(app_module, "GUFO_BASE_URL", "")
|
||||
upstream.route("http://vllm.test/metrics", response(200, VLLM_FIXTURE.encode()))
|
||||
assert client.get("/api/gufo/metrics").status_code == 404
|
||||
assert client.get("/api/gufo/slots").status_code == 404
|
||||
body = client.get("/api/vllm/metrics").json()
|
||||
assert body["ok"] is True
|
||||
assert upstream.requested_urls == ["http://vllm.test/metrics"]
|
||||
|
||||
|
||||
def test_config_reports_which_engines_exist(client, monkeypatch) -> None:
|
||||
body = client.get("/api/config").json()
|
||||
assert body["gufo_enabled"] is True and body["vllm_enabled"] is True
|
||||
|
||||
monkeypatch.setattr(app_module, "GUFO_BASE_URL", "")
|
||||
body = client.get("/api/config").json()
|
||||
assert body["gufo_enabled"] is False
|
||||
assert body["gufo_base_url"] == ""
|
||||
assert body["vllm_enabled"] is True # the other engine keeps its section
|
||||
|
||||
|
||||
def test_lifespan_owns_one_shared_client(client) -> None:
|
||||
"""Entering lifespan installs exactly one AsyncClient on app.state and closes it
|
||||
on shutdown — the poll loop depends on connection reuse, and a leaked client
|
||||
would keep sockets open after Ctrl-C."""
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
with TestClient(app_module.app) as running:
|
||||
shared = app_module.app.state.client
|
||||
assert isinstance(shared, httpx.AsyncClient)
|
||||
assert not shared.is_closed
|
||||
assert running.get("/api/config").status_code == 200
|
||||
assert app_module.app.state.client is shared
|
||||
assert shared.is_closed
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# page + misc
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_dashboard_is_served(client) -> None:
|
||||
r = client.get("/")
|
||||
assert r.status_code == 200
|
||||
assert "text/html" in r.headers["content-type"]
|
||||
assert "LLM Monitor" in r.text
|
||||
|
||||
|
||||
def test_dashboard_layout_is_driven_by_the_config_flags(client) -> None:
|
||||
"""The page must render from /api/config's *_enabled flags instead of hardcoding two
|
||||
engines: an absent engine loses its panel, its strip card, its polling and its share
|
||||
of the fleet totals, and nothing configured at all shows the setup card. Asserted on
|
||||
the shipped HTML because the layout logic lives in the inline script."""
|
||||
html = client.get("/").text
|
||||
# one generic engine list, revealed from the backend's flags
|
||||
assert "const ENGINE_IDS" in html and "_enabled`" in html
|
||||
assert "isEnabled(job.engine)" in html # an absent engine is never polled
|
||||
# what gets hidden, and what shows instead when there is nothing to show
|
||||
for marker in ("panel-${engine}", "strip-${engine}", 'id="no-engines"'):
|
||||
assert marker in html
|
||||
|
||||
|
||||
def test_dashboard_loads_no_third_party_assets(client) -> None:
|
||||
"""The engines live on an internet-disconnected network: no CDN, no webfonts,
|
||||
no build step. Only same-origin requests are allowed from the page."""
|
||||
import re
|
||||
|
||||
html = client.get("/").text
|
||||
refs = re.findall(r"""(?:src|href)\s*=\s*["']([^"']+)["']""", html)
|
||||
external = [r for r in refs if r.startswith(("http://", "//"))]
|
||||
assert external == [], f"page references external assets: {external}"
|
||||
|
||||
|
||||
def test_unknown_route_is_json(client) -> None:
|
||||
r = client.get("/api/nope")
|
||||
assert r.status_code == 404
|
||||
assert "no such route" in r.json()["detail"]
|
||||
@@ -0,0 +1,299 @@
|
||||
"""Derived throughput: Δcounter/Δt and the lifetime-average fallback.
|
||||
|
||||
The deployed vLLM exports no rate gauge of any kind (verified: zero series with a
|
||||
throughput/rate name), so prompt tok/s can only be *computed*. These tests pin the
|
||||
precedence — native gauge > measured Δcounter > estimate > n/a — and the counter-reset
|
||||
handling, because a naive delta shows up as a giant negative spike after an engine
|
||||
restart.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from app import RateEstimator, parse_gufo_metrics, parse_vllm_metrics
|
||||
|
||||
# gufo's generation-speed gauge is ONE request's last decode pass; the counter is every
|
||||
# slot's tokens. 30 tok/s on the gauge with 60 tokens landing per second = two slots.
|
||||
GUFO_SCRAPE = """
|
||||
llamacpp:requests_processing 2
|
||||
llamacpp:tokens_predicted_total 1000
|
||||
llamacpp:predicted_tokens_seconds 30
|
||||
"""
|
||||
|
||||
COUNTERS = """
|
||||
vllm:num_requests_running{engine="0"} 2
|
||||
vllm:prompt_tokens_total{engine="0"} 1000
|
||||
vllm:generation_tokens_total{engine="0"} 400
|
||||
vllm:inter_token_latency_seconds_sum 0.04
|
||||
vllm:inter_token_latency_seconds_count 2
|
||||
"""
|
||||
|
||||
|
||||
def test_first_scrape_has_nothing_to_diff_against() -> None:
|
||||
rates = RateEstimator()
|
||||
assert rates.rate("k", 100.0, now=10.0) == (None, None)
|
||||
|
||||
|
||||
def test_second_scrape_measures_the_delta() -> None:
|
||||
rates = RateEstimator()
|
||||
rates.rate("k", 100.0, now=10.0)
|
||||
rate, window = rates.rate("k", 130.0, now=11.0)
|
||||
assert rate == pytest.approx(30.0)
|
||||
assert window == pytest.approx(1.0)
|
||||
|
||||
|
||||
def test_counter_reset_yields_none_not_a_negative_spike() -> None:
|
||||
rates = RateEstimator()
|
||||
rates.rate("k", 5000.0, now=10.0)
|
||||
assert rates.rate("k", 12.0, now=11.0) == (None, None) # engine restarted
|
||||
# and it recovers on the following scrape
|
||||
assert rates.rate("k", 22.0, now=12.0)[0] == pytest.approx(10.0)
|
||||
|
||||
|
||||
def test_absent_counter_and_zero_window_are_none() -> None:
|
||||
rates = RateEstimator()
|
||||
assert rates.rate("k", None, now=1.0) == (None, None)
|
||||
rates.rate("k", 1.0, now=5.0)
|
||||
assert rates.rate("k", 2.0, now=5.0) == (None, None) # dt == 0
|
||||
assert rates.rate("other", 1.0, now=6.0) == (None, None) # unknown key
|
||||
|
||||
|
||||
def test_rate_smooths_over_the_span_once_enough_samples_exist() -> None:
|
||||
# a bare 1s delta jitters (prompts arrive in bursts); the span averages it out
|
||||
rates = RateEstimator(span=10.0)
|
||||
rates.rate("k", 0.0, now=0.0) # outside the span by t=12
|
||||
rates.rate("k", 100.0, now=5.0)
|
||||
rate, window = rates.rate("k", 400.0, now=12.0)
|
||||
assert window == pytest.approx(7.0) # measured from the oldest in-span sample
|
||||
assert rate == pytest.approx(300.0 / 7.0)
|
||||
|
||||
|
||||
def test_only_two_in_window_samples_still_produce_a_shorter_rate() -> None:
|
||||
rates = RateEstimator(span=10.0)
|
||||
rates.rate("k", 0.0, now=0.0)
|
||||
rates.rate("k", 100.0, now=20.0) # first sample is long gone
|
||||
rate, window = rates.rate("k", 150.0, now=21.0)
|
||||
assert window == pytest.approx(1.0)
|
||||
assert rate == pytest.approx(50.0)
|
||||
|
||||
|
||||
def test_history_is_bounded() -> None:
|
||||
rates = RateEstimator(span=1.0, max_samples=8)
|
||||
for i in range(50):
|
||||
rates.rate("k", float(i), now=float(i))
|
||||
assert len(rates._samples["k"]) <= 8
|
||||
|
||||
|
||||
def test_keys_are_independent() -> None:
|
||||
rates = RateEstimator()
|
||||
rates.rate("a", 0.0, now=0.0)
|
||||
rates.rate("b", 0.0, now=0.0)
|
||||
assert rates.rate("a", 10.0, now=1.0)[0] == pytest.approx(10.0)
|
||||
assert rates.rate("b", 4.0, now=1.0)[0] == pytest.approx(4.0)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# gufo aggregate decode rate: the counter adds slots, the gauge cannot
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_gufo_decode_rate_adds_every_slot_generating_at_once() -> None:
|
||||
"""The whole point: two slots at 30 tok/s must read 60, not 30.
|
||||
|
||||
gufo's `predicted_tokens_seconds` is a single unlabeled gauge holding one request's
|
||||
last decode pass, so it can never sum concurrency. `tokens_predicted_total` advances
|
||||
once per token from every slot, so Δ/Δt is the server-aggregate.
|
||||
"""
|
||||
rates = RateEstimator()
|
||||
parse_gufo_metrics(GUFO_SCRAPE, rates, now=100.0)
|
||||
m = parse_gufo_metrics(GUFO_SCRAPE.replace("1000", "1060"), rates, now=101.0)
|
||||
d = m["decode_tokens_per_second"]
|
||||
assert d["value"] == pytest.approx(60.0)
|
||||
assert d["derived"] is True
|
||||
assert d["window"] == pytest.approx(1.0)
|
||||
assert d["source"] == "computed:Δllamacpp:tokens_predicted_total/Δt"
|
||||
# the single-request gauge stays exposed, untouched, as the native reference
|
||||
assert m["decode_tokens_per_second_native"]["value"] == pytest.approx(30.0)
|
||||
|
||||
|
||||
def test_gufo_first_scrape_falls_back_to_the_gauge_not_na() -> None:
|
||||
# nothing to diff against yet, but the scrape can still answer
|
||||
m = parse_gufo_metrics(GUFO_SCRAPE, RateEstimator(), now=10.0)
|
||||
assert m["decode_tokens_per_second"] == {
|
||||
"value": 30.0,
|
||||
"source": "llamacpp:predicted_tokens_seconds",
|
||||
"derived": False,
|
||||
}
|
||||
|
||||
|
||||
def test_gufo_idle_reads_zero_not_the_stale_gauge() -> None:
|
||||
# an engine that generated nothing in the window is at 0 tok/s; the gauge would
|
||||
# happily keep showing the last session's speed forever
|
||||
rates = RateEstimator()
|
||||
parse_gufo_metrics(GUFO_SCRAPE, rates, now=0.0)
|
||||
m = parse_gufo_metrics(GUFO_SCRAPE, rates, now=1.0)
|
||||
assert m["decode_tokens_per_second"]["value"] == 0.0
|
||||
assert m["decode_tokens_per_second_native"]["value"] == pytest.approx(30.0)
|
||||
|
||||
|
||||
def test_gufo_counter_reset_falls_back_instead_of_going_negative() -> None:
|
||||
rates = RateEstimator()
|
||||
parse_gufo_metrics(GUFO_SCRAPE, rates, now=0.0)
|
||||
# engine restarted: the counter dropped. Never -995 tok/s.
|
||||
m = parse_gufo_metrics(GUFO_SCRAPE.replace("1000", "5"), rates, now=1.0)
|
||||
assert m["decode_tokens_per_second"]["value"] == pytest.approx(30.0)
|
||||
assert m["decode_tokens_per_second"]["derived"] is False
|
||||
# and it recovers on the following scrape
|
||||
m = parse_gufo_metrics(GUFO_SCRAPE.replace("1000", "25"), rates, now=2.0)
|
||||
assert m["decode_tokens_per_second"]["value"] == pytest.approx(20.0)
|
||||
|
||||
|
||||
def test_gufo_alias_rename_starts_a_fresh_window_instead_of_a_fake_rate() -> None:
|
||||
# the estimator keys off the RESOLVED series name, so a renamed counter cannot diff
|
||||
# against a different series' history (and a flapping alias cannot invent throughput)
|
||||
rates = RateEstimator()
|
||||
parse_gufo_metrics("llamacpp:tokens_generated_total 1000\n", rates, now=0.0)
|
||||
m = parse_gufo_metrics("llamacpp:tokens_predicted_total 1000\n", rates, now=1.0)
|
||||
assert m["decode_tokens_per_second"]["value"] is None
|
||||
|
||||
|
||||
def test_gufo_prompt_rate_is_measured_and_adds_slots_too() -> None:
|
||||
"""Prefill gets the same treatment as decode, so a cross-engine prompt sum is additive.
|
||||
|
||||
gufo's `prompt_tokens_seconds` is one request's last prefill pass; the counter is every
|
||||
slot's newly-computed prompt tokens.
|
||||
"""
|
||||
text = "llamacpp:prompt_tokens_total 1000\nllamacpp:prompt_tokens_seconds 700\n"
|
||||
rates = RateEstimator()
|
||||
parse_gufo_metrics(text, rates, now=0.0)
|
||||
m = parse_gufo_metrics(text.replace("1000", "3400"), rates, now=2.0)
|
||||
p = m["prompt_tokens_per_second"]
|
||||
assert p["value"] == pytest.approx(1200.0)
|
||||
assert p["source"] == "computed:Δllamacpp:prompt_tokens_total/Δt"
|
||||
assert p["derived"] is True and p["window"] == pytest.approx(2.0)
|
||||
# the one-request gauge stays exposed next to it
|
||||
assert m["prompt_tokens_per_second_native"]["value"] == pytest.approx(700.0)
|
||||
|
||||
|
||||
def test_gufo_prompt_first_scrape_falls_back_to_the_gauge() -> None:
|
||||
text = "llamacpp:prompt_tokens_total 1000\nllamacpp:prompt_tokens_seconds 700\n"
|
||||
m = parse_gufo_metrics(text, RateEstimator(), now=5.0)
|
||||
assert m["prompt_tokens_per_second"] == {
|
||||
"value": 700.0,
|
||||
"source": "llamacpp:prompt_tokens_seconds",
|
||||
"derived": False,
|
||||
}
|
||||
|
||||
|
||||
def test_gufo_prompt_and_decode_keep_separate_windows() -> None:
|
||||
# one estimator, but the two counters must not read each other's history
|
||||
rates = RateEstimator()
|
||||
base = "llamacpp:prompt_tokens_total 1000\nllamacpp:tokens_predicted_total 500\n"
|
||||
parse_gufo_metrics(base, rates, now=0.0)
|
||||
m = parse_gufo_metrics(base.replace("1000", "3000").replace("500", "550"), rates, now=2.0)
|
||||
assert m["prompt_tokens_per_second"]["value"] == pytest.approx(1000.0)
|
||||
assert m["decode_tokens_per_second"]["value"] == pytest.approx(25.0)
|
||||
|
||||
|
||||
def test_gufo_prompt_idle_reads_zero_not_the_stale_gauge() -> None:
|
||||
text = "llamacpp:prompt_tokens_total 1000\nllamacpp:prompt_tokens_seconds 700\n"
|
||||
rates = RateEstimator()
|
||||
parse_gufo_metrics(text, rates, now=0.0)
|
||||
m = parse_gufo_metrics(text, rates, now=1.0)
|
||||
assert m["prompt_tokens_per_second"]["value"] == 0.0
|
||||
assert m["prompt_tokens_per_second_native"]["value"] == pytest.approx(700.0)
|
||||
|
||||
|
||||
def test_gufo_parse_without_rates_stays_pure() -> None:
|
||||
assert parse_gufo_metrics(GUFO_SCRAPE) == parse_gufo_metrics(GUFO_SCRAPE)
|
||||
assert parse_gufo_metrics(GUFO_SCRAPE)["decode_tokens_per_second"]["derived"] is False
|
||||
|
||||
|
||||
def test_both_engines_share_one_estimator_without_cross_reading() -> None:
|
||||
# RATES is one object for both engines; the series names keep the histories apart
|
||||
rates = RateEstimator()
|
||||
parse_gufo_metrics(GUFO_SCRAPE, rates, now=0.0)
|
||||
parse_vllm_metrics(COUNTERS, rates, now=0.0)
|
||||
g = parse_gufo_metrics(GUFO_SCRAPE.replace("1000", "1040"), rates, now=1.0)
|
||||
v = parse_vllm_metrics(COUNTERS.replace("400", "410"), rates, now=1.0)
|
||||
assert g["decode_tokens_per_second"]["value"] == pytest.approx(40.0)
|
||||
assert v["decode_tokens_per_second"]["source"] == "computed:Δvllm:generation_tokens_total/Δt"
|
||||
assert v["decode_tokens_per_second"]["value"] == pytest.approx(10.0)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# parser integration
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_prompt_and_decode_use_measured_deltas_when_rates_are_supplied() -> None:
|
||||
rates = RateEstimator()
|
||||
parse_vllm_metrics(COUNTERS, rates, now=100.0) # seeds the previous sample
|
||||
m = parse_vllm_metrics(
|
||||
COUNTERS.replace("1000", "1500").replace("400", "450"), rates, now=101.0
|
||||
)
|
||||
p = m["prompt_tokens_per_second"]
|
||||
d = m["decode_tokens_per_second"]
|
||||
assert p["derived"] is True and p["window"] == pytest.approx(1.0)
|
||||
assert p["value"] == pytest.approx(500.0)
|
||||
assert p["source"] == "computed:Δvllm:prompt_tokens_total/Δt"
|
||||
assert d["value"] == pytest.approx(50.0)
|
||||
assert d["source"] == "computed:Δvllm:generation_tokens_total/Δt"
|
||||
|
||||
|
||||
def test_first_scrape_falls_back_to_the_lifetime_average() -> None:
|
||||
# no previous sample to diff against, but prefill time + prompt length still answer
|
||||
text = (
|
||||
"vllm:prompt_tokens_total 1000\n"
|
||||
"vllm:request_prompt_tokens_sum 8000\nvllm:request_prompt_tokens_count 2\n"
|
||||
"vllm:request_prefill_time_seconds_sum 1.6\nvllm:request_prefill_time_seconds_count 2\n"
|
||||
)
|
||||
m = parse_vllm_metrics(text, RateEstimator(), now=10.0)
|
||||
p = m["prompt_tokens_per_second"]
|
||||
assert p["value"] == pytest.approx(5000.0) # 4000 tok mean / 0.8 s mean
|
||||
assert p["source"] == "computed:mean(request_prompt_tokens)/mean(request_prefill_time_seconds)"
|
||||
assert m["prefill_tokens_per_second_lifetime"]["value"] == pytest.approx(5000.0)
|
||||
|
||||
|
||||
def test_measured_delta_beats_the_lifetime_average() -> None:
|
||||
rates = RateEstimator()
|
||||
text = (
|
||||
"vllm:prompt_tokens_total 1000\n"
|
||||
"vllm:request_prompt_tokens_sum 8000\nvllm:request_prompt_tokens_count 2\n"
|
||||
"vllm:request_prefill_time_seconds_sum 1.6\nvllm:request_prefill_time_seconds_count 2\n"
|
||||
)
|
||||
parse_vllm_metrics(text, rates, now=0.0)
|
||||
m = parse_vllm_metrics(text.replace("1000", "1200"), rates, now=1.0)
|
||||
assert m["prompt_tokens_per_second"]["source"].startswith("computed:Δ")
|
||||
assert m["prompt_tokens_per_second"]["value"] == pytest.approx(200.0)
|
||||
|
||||
|
||||
def test_native_gauge_still_wins_over_any_derivation() -> None:
|
||||
m = parse_vllm_metrics(
|
||||
"vllm:prompt_throughput 1234.5\nvllm:prompt_tokens_total 10\n",
|
||||
RateEstimator(),
|
||||
now=1.0,
|
||||
)
|
||||
assert m["prompt_tokens_per_second"] == {
|
||||
"value": 1234.5,
|
||||
"source": "vllm:prompt_throughput",
|
||||
"derived": False,
|
||||
}
|
||||
|
||||
|
||||
def test_without_rates_the_parse_stays_pure() -> None:
|
||||
# parse(text) alone must not depend on wall-clock or hidden state
|
||||
a = parse_vllm_metrics(COUNTERS)
|
||||
b = parse_vllm_metrics(COUNTERS)
|
||||
assert a == b
|
||||
assert a["prompt_tokens_per_second"]["value"] is None # no prefill series here
|
||||
assert a["prefill_tokens_per_second_lifetime"]["value"] is None
|
||||
# decode still answers via TPOT, which is stateless
|
||||
assert a["decode_tokens_per_second"]["source"] == "computed:running/tpot"
|
||||
|
||||
|
||||
def test_prompt_tok_s_is_never_a_silent_zero() -> None:
|
||||
"""The whole point: an engine that IS working must not render n/a because it lacks
|
||||
a gauge, and an idle engine must show 0 rather than a stale estimate."""
|
||||
rates = RateEstimator()
|
||||
text = "vllm:prompt_tokens_total 500\n"
|
||||
parse_vllm_metrics(text, rates, now=0.0)
|
||||
m = parse_vllm_metrics(text, rates, now=1.0)
|
||||
assert m["prompt_tokens_per_second"]["value"] == 0.0 # genuinely idle, not unknown
|
||||
@@ -0,0 +1,229 @@
|
||||
"""Ground truth: the payloads captured from the DEPLOYED engines.
|
||||
|
||||
Neither engine exports the series names people expect, and gufo's `/slots` is a bare
|
||||
JSON array. These fixtures are verbatim captures (gufo on deskwork-ai:8731, vLLM on
|
||||
driveripper-ai:8000); if a field stops mapping, the engine renamed something and this
|
||||
test tells you which one instead of the dashboard quietly showing `n/a` in production.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app import RateEstimator, parse_gufo_metrics, parse_vllm_metrics
|
||||
|
||||
FIXTURES = Path(__file__).resolve().parent / "fixtures"
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def gufo_text() -> str:
|
||||
return (FIXTURES / "gufo_metrics_real.txt").read_text()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def vllm_text() -> str:
|
||||
return (FIXTURES / "vllm_metrics_real.txt").read_text()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# gufo
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_gufo_real_names_are_the_primary_aliases(gufo_text: str) -> None:
|
||||
m = parse_gufo_metrics(gufo_text)
|
||||
assert m["tokens_prompt_total"]["source"] == "llamacpp:prompt_tokens_total"
|
||||
assert m["tokens_generated_total"]["source"] == "llamacpp:tokens_predicted_total"
|
||||
# gufo names its speed gauges *_seconds while the unit is tokens/second
|
||||
assert m["prompt_tokens_per_second_native"]["source"] == "llamacpp:prompt_tokens_seconds"
|
||||
assert m["decode_tokens_per_second_native"]["source"] == "llamacpp:predicted_tokens_seconds"
|
||||
# parsed without an estimator, both throughput cells degrade to those one-request gauges
|
||||
assert m["prompt_tokens_per_second"]["source"] == "llamacpp:prompt_tokens_seconds"
|
||||
# parsed without an estimator, the throughput cell degrades to that one-request gauge
|
||||
assert m["decode_tokens_per_second"]["source"] == "llamacpp:predicted_tokens_seconds"
|
||||
|
||||
|
||||
def test_gufo_aggregate_decode_rate_resolves_on_the_real_counter(gufo_text: str) -> None:
|
||||
"""The Δ-rates key off the RESOLVED counter names, so they must work on gufo's real
|
||||
names (`tokens_predicted_total`, `prompt_tokens_total`) and land in the same ballpark as
|
||||
gufo's own busy-time counter (`tokens_predicted_seconds_total`).
|
||||
"""
|
||||
rates = RateEstimator()
|
||||
parse_gufo_metrics(gufo_text, rates, now=0.0)
|
||||
bumped = gufo_text.replace(
|
||||
"llamacpp:tokens_predicted_total 173091", "llamacpp:tokens_predicted_total 173191"
|
||||
).replace(
|
||||
# +1000 generated and +1000 prompt tokens over 2s -> 500 tok/s of each
|
||||
"llamacpp:prompt_tokens_total 374263", "llamacpp:prompt_tokens_total 375263"
|
||||
)
|
||||
assert bumped != gufo_text # the capture moved the counter; re-check this test
|
||||
m = parse_gufo_metrics(bumped, rates, now=2.0)
|
||||
d = m["decode_tokens_per_second"]
|
||||
assert d["source"] == "computed:Δllamacpp:tokens_predicted_total/Δt"
|
||||
assert d["value"] == pytest.approx(50.0)
|
||||
# sanity: the lifetime mean from the two cumulative counters is the same order of
|
||||
# magnitude, so the windowed aggregate is not quietly summing something else
|
||||
busy = float(re.search(r"tokens_predicted_seconds_total ([0-9.]+)", gufo_text).group(1))
|
||||
assert 5.0 < (173091.0 / busy) < 100.0
|
||||
|
||||
# prompt throughput from the same two scrapes: gufo's prompt counter EXCLUDES cache hits,
|
||||
# so this is newly-computed prefill work across all slots, not prompt tokens accepted
|
||||
p = m["prompt_tokens_per_second"]
|
||||
assert p["source"] == "computed:Δllamacpp:prompt_tokens_total/Δt"
|
||||
assert p["value"] == pytest.approx(500.0)
|
||||
assert m["prompt_tokens_per_second_native"]["value"] == pytest.approx(820.2841172907755)
|
||||
|
||||
|
||||
def test_gufo_every_banner_and_strip_field_resolves(gufo_text: str) -> None:
|
||||
m = parse_gufo_metrics(gufo_text)
|
||||
for key in (
|
||||
"requests_processing",
|
||||
"requests_deferred",
|
||||
"kv_cache_usage_ratio",
|
||||
"tokens_prompt_total",
|
||||
"tokens_generated_total",
|
||||
"prompt_tokens_per_second",
|
||||
"decode_tokens_per_second",
|
||||
"tokens_total",
|
||||
"n_tokens_max",
|
||||
"device_lost_total",
|
||||
):
|
||||
assert m[key]["value"] is not None, f"gufo {key} stopped resolving"
|
||||
assert 0.0 <= m["kv_cache_usage_ratio"]["value"] <= 1.0
|
||||
|
||||
|
||||
def test_gufo_totals_include_cache_hits_which_are_a_separate_counter(gufo_text: str) -> None:
|
||||
m = parse_gufo_metrics(gufo_text)
|
||||
prompt = m["tokens_prompt_total"]["value"]
|
||||
cached = m["tokens_prompt_cached_total"]["value"]
|
||||
generated = m["tokens_generated_total"]["value"]
|
||||
assert m["tokens_total"]["value"] == pytest.approx(prompt + cached + generated)
|
||||
# 29.6M cached vs 0.37M computed on the live box — cache hits dominate
|
||||
assert m["prompt_cache_hit_ratio"]["value"] == pytest.approx(cached / (prompt + cached))
|
||||
assert 0.0 < m["prompt_cache_hit_ratio"]["value"] < 1.0
|
||||
|
||||
|
||||
def test_gufo_speculative_decoding_accept_rate(gufo_text: str) -> None:
|
||||
m = parse_gufo_metrics(gufo_text)
|
||||
accept = m["spec_accept_rate"]
|
||||
assert accept["derived"] is True
|
||||
assert 0.0 <= accept["value"] <= 1.0
|
||||
|
||||
|
||||
def test_gufo_device_lost_is_zero_on_a_healthy_box(gufo_text: str) -> None:
|
||||
assert parse_gufo_metrics(gufo_text)["device_lost_total"]["value"] == 0.0
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# gufo /slots
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_gufo_slots_is_a_bare_array_not_an_object() -> None:
|
||||
"""The UI must not assume {"slots": [...]} — the deployed engine returns a list."""
|
||||
data = json.loads((FIXTURES / "gufo_slots_real.json").read_text())
|
||||
assert isinstance(data, list)
|
||||
assert len(data) >= 1
|
||||
assert all(isinstance(slot, dict) and "id" in slot for slot in data)
|
||||
|
||||
|
||||
def test_gufo_slot_fields_the_cards_depend_on() -> None:
|
||||
data = json.loads((FIXTURES / "gufo_slots_real.json").read_text())
|
||||
slot = data[0]
|
||||
for key in (
|
||||
"id",
|
||||
"state",
|
||||
"is_processing",
|
||||
"id_task",
|
||||
"n_prompt_tokens",
|
||||
"n_prompt_tokens_cache",
|
||||
"n_ctx",
|
||||
"next_token",
|
||||
"model",
|
||||
):
|
||||
assert key in slot, f"gufo slot no longer reports {key}"
|
||||
assert slot["next_token"], "next_token[] should still be present, with n_decoded/n_remain"
|
||||
assert "n_decoded" in slot["next_token"][0]
|
||||
# an idle slot reports -1 for "nothing pending", which is not a count
|
||||
if not slot["is_processing"]:
|
||||
assert slot["id_task"] == -1
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# vLLM
|
||||
# --------------------------------------------------------------------------- #
|
||||
def test_vllm_real_names_are_the_primary_aliases(vllm_text: str) -> None:
|
||||
m = parse_vllm_metrics(vllm_text)
|
||||
assert m["gpu_cache_usage"]["source"] == "vllm:kv_cache_usage_perc" # not gpu_cache_usage
|
||||
assert m["generation_tokens_total"]["source"] == "vllm:generation_tokens_total"
|
||||
assert m["time_per_output_token_seconds"]["source"] == "vllm:inter_token_latency_seconds"
|
||||
|
||||
|
||||
def test_vllm_every_banner_and_card_field_resolves(vllm_text: str) -> None:
|
||||
m = parse_vllm_metrics(vllm_text)
|
||||
for key in (
|
||||
"num_requests_running",
|
||||
"num_requests_waiting",
|
||||
"gpu_cache_usage",
|
||||
"prompt_tokens_total",
|
||||
"generation_tokens_total",
|
||||
"tokens_total",
|
||||
"waiting_capacity",
|
||||
"prefix_cache_hit_ratio",
|
||||
"num_preemptions_total",
|
||||
):
|
||||
assert m[key]["value"] is not None, f"vLLM {key} stopped resolving"
|
||||
for key in (
|
||||
"time_to_first_token_seconds__mean",
|
||||
"time_per_output_token_seconds__mean",
|
||||
"request_queue_time_seconds__mean",
|
||||
):
|
||||
assert m[key]["value"] is not None, f"vLLM {key} stopped resolving"
|
||||
assert 0.0 <= m["gpu_cache_usage"]["value"] <= 1.0
|
||||
assert 0.0 <= m["prefix_cache_hit_ratio"]["value"] <= 1.0
|
||||
|
||||
|
||||
def test_vllm_waiting_reasons_from_the_live_labels(vllm_text: str) -> None:
|
||||
# the deployed build tags reasons as capacity/deferred, not defragmentation
|
||||
m = parse_vllm_metrics(vllm_text)
|
||||
assert "capacity" in m["waiting_by_reason"]["value"]
|
||||
|
||||
|
||||
def test_vllm_histograms_survive_engine_and_model_labels(vllm_text: str) -> None:
|
||||
m = parse_vllm_metrics(vllm_text)
|
||||
ttft = m["time_to_first_token_seconds"]
|
||||
assert ttft["count"] > 0 and ttft["mean"] > 0
|
||||
# single engine label set here, so an interpolated p95 is legitimate
|
||||
assert ttft["p95"] is None or ttft["p95"] >= ttft["mean"]
|
||||
|
||||
|
||||
def test_vllm_prompt_tok_s_is_answered_even_without_a_rate_gauge(vllm_text: str) -> None:
|
||||
# this build exports no throughput series at all, so prompt tok/s comes from the
|
||||
# prefill histograms; it must not render n/a on a healthy, busy engine
|
||||
m = parse_vllm_metrics(vllm_text)
|
||||
p = m["prompt_tokens_per_second"]
|
||||
assert p["value"] is not None, "prompt tok/s regressed to n/a"
|
||||
assert p["derived"] is True
|
||||
assert p["source"] == "computed:mean(request_prompt_tokens)/mean(request_prefill_time_seconds)"
|
||||
assert p["value"] > 1000 # live box sits around 4.7k tok/s; single digits mean a bug
|
||||
assert m["prefill_tokens_per_second_lifetime"]["value"] == pytest.approx(p["value"])
|
||||
|
||||
|
||||
def test_vllm_prompt_tok_s_switches_to_measured_rate_once_scraping(vllm_text: str) -> None:
|
||||
from app import RateEstimator
|
||||
|
||||
rates = RateEstimator()
|
||||
parse_vllm_metrics(vllm_text, rates, now=1000.0)
|
||||
m = parse_vllm_metrics(vllm_text, rates, now=1001.0) # same counters -> zero throughput
|
||||
p = m["prompt_tokens_per_second"]
|
||||
assert p["source"] == "computed:Δvllm:prompt_tokens_total/Δt"
|
||||
assert p["window"] == pytest.approx(1.0)
|
||||
# unchanged capture -> genuinely zero throughput in that window, not a missing gauge
|
||||
assert p["value"] == 0.0
|
||||
|
||||
|
||||
def test_vllm_decode_throughput_is_derived_on_this_build(vllm_text: str) -> None:
|
||||
# no vllm:*throughput gauge exists on the deployed version -> computed from TPOT
|
||||
m = parse_vllm_metrics(vllm_text)
|
||||
assert m["decode_tokens_per_second"]["derived"] is True
|
||||
assert m["decode_tokens_per_second"]["source"] == "computed:running/tpot"
|
||||
Reference in New Issue
Block a user