phase: 23_llm_rate_responsiveness
Phase 23 final verification complete — all green.
```
VERIFIED (criteria)
1 /api/llm/snapshot windows ≈2 s live (1.98–2.00), /api/system window_seconds ≈10 s (9.996–10.005);
/api/config llm.rate_window_seconds=2=LLM_INTERVAL_SECONDS; browser re-check ALL CHECKS PASSED
(data/logs/23-window-{llm,system}.png — /llm prints Δ2.0s, /system footnote Δ10.0s)
2 Phase-13 pins unchanged: floor, 2×span ceiling, 8× jump gate + independent anchor — all pass
(only import/span-line diffs in tests/; git diff confirms no assertion rewritten)
3 Live proof re-derived from artifacts: 780 rows, 0 unlabelled, windows 0.61–3.38 s, 5 starts +
5 ends settled by poll 2 (0 unsettled), 1 m rollups max 42.3 vs raw 201.6, 0 over 300 tok/s;
counterfactual 10 s window: median settle poll 3, |ln| 0.924 vs 0.037
4 uv run pytest: 1494 passed / 15 skipped · coverage 98.94% (floor 94) · ruff clean · pyright 0
errors · node --check clean · bash .agents/validate.sh exit 0 · live 11 passed · store_perf 4 passed
Dev server up, all nine sources ok. No git add/commit run; everything left in the working tree.
DOCS PASS — 4 inconsistencies fixed in AGENTS.md (last two runs above include them)
- UI conventions hero example still said the `.sub` footnote reads `Δ10.0s`; the only hero that
prints a Δ is /llm's → now `Δ2.0s` on /llm, "at the current load" on /ups
- gap 9: "found 10 busy-edges" contradicted its own artifact (17 edges, 10 judged, 7 reported-not-
judged) → 17; and the poll-2 ratios were a 3-of-5 cherry-pick → full five values both windows
- gap 12: the responsive-table fix is now scheduled — names 27_mobile_tables_and_audit
README, config table, .env.example, rates/prom docstrings, app/api bullets and the testing-policy
pins all re-checked against code and found consistent (store pragmas, budgets, throttle, fleet).
NOTES
- Advisory only, not phase 23: fastapi 0.153.0 → 0.156.8 available; dependencies untouched here,
so rule 8's "update on purpose" is left to the operator.
- NOTES.md is empty in the working tree (its items became phases 24–27); left as-is, untracked-file
rules forbid me removing it.
- Next phase: 24_dev_mtls_rollout (25–27 follow).
```
This commit is contained in:
@@ -0,0 +1,78 @@
|
||||
"""The two Δ windows of phase 23, in one place so the pins cannot drift.
|
||||
|
||||
`app.build_hub` hands out **one `RateEstimator` per measuring cadence**:
|
||||
|
||||
| source | span | knob | cadence |
|
||||
| ------ | ------------- | ----------------------- | ------- |
|
||||
| host | `HOST_WINDOW` | `RATE_WINDOW_SECONDS` | 10 s |
|
||||
| llm | `LLM_WINDOW` | `LLM_INTERVAL_SECONDS` | 2 s |
|
||||
|
||||
Two numbers living in the same payload shape: every measured rate cell carries the `window` the
|
||||
estimator actually used, and `provenance()` in `page-llm.js` / `windowLabel()` in
|
||||
`page-system.js` print it as `Δ<window.toFixed(1)>s`. A pin that names only **one** of the two
|
||||
windows is how a later change could quietly give both sources the same span, so every pin reads
|
||||
them through the constants and helpers here:
|
||||
|
||||
* `tests/test_api.py` — the wiring: which estimator object each source's jobs receive, and the
|
||||
`/api/config` window the LLM page is allowed to echo;
|
||||
* `tests/test_llm_source.py` — the LLM payload: both throughput cells of both engines at the
|
||||
2 s shape, and the phase-13 floor still refusing the poll loop's catch-up gap at that span;
|
||||
* `tests/test_host_source.py` — the host payload: `window_seconds` at the 10 s shape, and the
|
||||
same 10 s cadence starved to `n/a` by the LLM's 2 s span (why one estimator cannot serve both);
|
||||
* `tests/test_rates.py` — the estimator's own gates at both spans, because they are properties of
|
||||
the estimator and not of the window size.
|
||||
|
||||
`config.py`'s defaults are the source of truth for the numbers themselves and
|
||||
`tests/test_config.py` pins them; this module exists so the pins cite one spelling of them
|
||||
instead of five literals that can each drift on their own day.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from dashboards.rates import MAX_WINDOW_RATIO, MIN_WINDOW_SECONDS
|
||||
|
||||
#: `RATE_WINDOW_SECONDS` — the Δ window the `host` jobs are measured over.
|
||||
HOST_WINDOW = 10.0
|
||||
|
||||
#: `LLM_INTERVAL_SECONDS` — the Δ window the `llm` jobs are measured over. It *is* their poll
|
||||
#: interval (phase 23): the smallest window a 2 s cadence can honestly fill, which is what stops
|
||||
#: a slot start ramping the hero over five polls.
|
||||
LLM_WINDOW = 2.0
|
||||
|
||||
|
||||
def delta_label(window: float | None) -> str:
|
||||
"""The provenance tag a page prints for a window — `provenance()`'s `toFixed(1)` idiom.
|
||||
|
||||
The pages build `Δ<window>s` out of the payload's own `window` field; this is the same
|
||||
rounding in Python, so a pin can assert *the label the operator reads* (`Δ2.0s`) instead of a
|
||||
bare float whose meaning nobody checked.
|
||||
"""
|
||||
return "" if window is None else f"Δ{window:.1f}s"
|
||||
|
||||
|
||||
def assert_window(window: float | None, span: float, *, where: str) -> None:
|
||||
"""Assert a measured number carries its own source's window, inside the estimator's bounds.
|
||||
|
||||
`window` is a rate cell's `window` field (both engines) or a host payload's `window_seconds`
|
||||
(`/system`'s footnote), and `where` names it in the failure message.
|
||||
|
||||
The bound is a **range**, never a literal, and its ceiling is the estimator's own
|
||||
`2 x span` rather than the span. A span equal to the cadence means only the current sample is
|
||||
ever strictly *inside* the window, so the base comes from the window-empty fallback — the
|
||||
previous sample, one poll back — and a published window is the real poll gap: the dev server
|
||||
reads 2.009-2.026 s on `/llm` and 9.978-10.002 s on `/system`. Anything past `2 x span` is a
|
||||
stall and answers `n/a`, which is the half of this the phase-13 pins hold. Because the window
|
||||
tracks a cadence knob, an operator who raises `LLM_INTERVAL_SECONDS` moves the window and this
|
||||
assertion together, while a pin hard-coded to `2.0` would rot.
|
||||
|
||||
The tight half is the label: `delta_label(window) == delta_label(span)` is what the browser
|
||||
prints, so `Δ2.0s` on `/llm` and `Δ10.0s` on `/system` are one assertion made at two different
|
||||
spans, and a window that had drifted a whole interval would print a different label.
|
||||
"""
|
||||
assert window is not None, f"{where} carries no window at all"
|
||||
assert MIN_WINDOW_SECONDS <= window <= MAX_WINDOW_RATIO * span, (
|
||||
f"{where} window {window} is outside its source's own bounds "
|
||||
f"(floor {MIN_WINDOW_SECONDS}, ceiling {MAX_WINDOW_RATIO} x span {span})"
|
||||
)
|
||||
assert delta_label(window) == delta_label(span), (
|
||||
f"{where} would print {delta_label(window)!r} beside a value measured over {span}s"
|
||||
)
|
||||
+93
-1
@@ -28,11 +28,12 @@ from auth_client import sign_in
|
||||
from auth_contract import ADMIN_PASSWORD, FAST_PBKDF2
|
||||
from card_contract import HOST_SHAPES, assert_card_contract
|
||||
from dashboards import alerts, auth, certs
|
||||
from dashboards import app as app_module
|
||||
from dashboards.api import MIN_PASSWORD_LENGTH, RANGE_SECONDS, _disk_visible, parse_range
|
||||
from dashboards.app import create_app
|
||||
from dashboards.collectors import UpstreamError
|
||||
from dashboards.config import Settings, Target
|
||||
from dashboards.rates import RateEstimator
|
||||
from dashboards.rates import MIN_WINDOW_SECONDS, RateEstimator
|
||||
from dashboards.severity import classify
|
||||
from dashboards.sources import nut
|
||||
from dashboards.sources.disks import SMART_SERIES, DisksCollector, parse_disks_snapshot
|
||||
@@ -43,6 +44,7 @@ from dashboards.sources.llm import (
|
||||
parse_vllm_metrics,
|
||||
)
|
||||
from dashboards.store import RES_RAW, Event, Sample, SampleStore
|
||||
from rate_window_contract import LLM_WINDOW, delta_label
|
||||
from slots_contract import (
|
||||
CANONICAL_TASK_ID_FIELD,
|
||||
OPTIONAL_NEXT_TOKEN_FIELDS,
|
||||
@@ -815,6 +817,77 @@ def test_engine_urls_are_redacted_the_same_way_hosts_are(tmp_path: Path) -> None
|
||||
store.close()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("llm_interval", "rate_window"), [(2.0, 10.0), (4.0, 20.0)])
|
||||
def test_build_hub_gives_each_source_the_estimator_its_own_cadence_fills(
|
||||
tmp_path: Path, monkeypatch: pytest.MonkeyPatch,
|
||||
llm_interval: float, rate_window: float) -> None:
|
||||
"""Two rate estimators, one per measuring cadence (phase 23, `TODO.md:8`).
|
||||
|
||||
The spy is the point: the numbers themselves are `config` defaults and a test on the default
|
||||
would pass even if `build_hub` handed one estimator to both sources. So this asserts the two
|
||||
**objects** the jobs receive, and does it at a non-default cadence pair to prove the LLM span
|
||||
tracks `LLM_INTERVAL_SECONDS` rather than a second knob nobody configured — one 10 s window
|
||||
smoothed a slot start across the five polls the operator was watching (the ramp), and a 2 s
|
||||
window over a 10 s `host` poll would answer `n/a` every poll.
|
||||
|
||||
The payload-level twin of this pin is the `Δ<window>` label each rate cell carries: this is
|
||||
the wiring, that is what the browser prints (phase 23 task 02 pins the labels).
|
||||
"""
|
||||
seen: dict[str, RateEstimator] = {}
|
||||
real_host_jobs = app_module.host_jobs
|
||||
real_llm_jobs = app_module.llm_jobs
|
||||
|
||||
def spy_host_jobs(targets: Any, http: Any, host_collector: Any, interval: float,
|
||||
**kwargs: Any) -> list[Any]:
|
||||
seen["host"] = host_collector.rates
|
||||
return real_host_jobs(targets, http, host_collector, interval, **kwargs)
|
||||
|
||||
def spy_llm_jobs(cfg: Any, http: Any, rates: RateEstimator) -> list[Any]:
|
||||
seen["llm"] = rates
|
||||
return real_llm_jobs(cfg, http, rates)
|
||||
|
||||
monkeypatch.setattr(app_module, "host_jobs", spy_host_jobs)
|
||||
monkeypatch.setattr(app_module, "llm_jobs", spy_llm_jobs)
|
||||
|
||||
db = str(tmp_path / "rate-windows.sqlite3")
|
||||
store = SampleStore(db)
|
||||
settings = Settings(db_path=db, admin_password=ADMIN_PASSWORD,
|
||||
pbkdf2_iterations=FAST_PBKDF2,
|
||||
host_targets=(Target("deskwork", DEAD_URL),),
|
||||
gufo_url="http://deskwork:8731", vllm_url="http://driveripper:8000",
|
||||
llm_interval=llm_interval, rate_window=rate_window)
|
||||
# `build_hub` only wires jobs; nothing here dials, so the shared client is never started
|
||||
# and there is no event loop to close one in.
|
||||
hub, _nut_monitor = app_module.build_hub(settings, store, httpx.AsyncClient())
|
||||
store.close()
|
||||
|
||||
assert sorted(seen) == ["host", "llm"], "a source stopped being handed an estimator"
|
||||
assert sorted(hub.jobs) == ["disks:deskwork", "host:deskwork", "llm:gufo", "llm:vllm"]
|
||||
assert seen["llm"] is not seen["host"], \
|
||||
"one estimator for both cadences is the bug this phase exists to fix"
|
||||
assert seen["llm"].span == pytest.approx(llm_interval), \
|
||||
"the LLM window must track LLM_INTERVAL_SECONDS, not a second knob"
|
||||
assert seen["host"].span == pytest.approx(rate_window), \
|
||||
"the host rates keep RATE_WINDOW_SECONDS: a 2 s span over a 10 s poll is `n/a` forever"
|
||||
# …and the smaller window still carries the production floor, so the phase-13 refusal of the
|
||||
# poll loop's ~0.2 s catch-up gap is not a consequence of the wider span.
|
||||
assert seen["llm"].min_window == MIN_WINDOW_SECONDS
|
||||
assert seen["host"].min_window == MIN_WINDOW_SECONDS
|
||||
assert seen["llm"].span > MIN_WINDOW_SECONDS, "the LLM window would refuse every poll"
|
||||
|
||||
|
||||
def test_config_reports_the_window_the_llm_rates_are_actually_measured_over(
|
||||
client: TestClient) -> None:
|
||||
"""`/api/config`'s llm window is a measurement claim, so it is the LLM's own span.
|
||||
|
||||
It used to echo `RATE_WINDOW_SECONDS` for both sources, which was harmless while one estimator
|
||||
served them all. Since phase 23 that same field would have advertised a 10 s window beside
|
||||
rates measured over 2 s — a number the page can print and no poll ever used (rule 2).
|
||||
"""
|
||||
llm = client.get("/api/config").json()["llm"]
|
||||
assert llm["rate_window_seconds"] == llm["interval_seconds"] == 2.0
|
||||
|
||||
|
||||
def wait_for_state(client: TestClient, path: str, *, timeout: float = 5.0) -> dict[str, Any]:
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
@@ -2586,6 +2659,25 @@ def test_llm_engine_panel_leads_with_two_throughput_heroes(client) -> None:
|
||||
assert "node.title = value.source" in js
|
||||
|
||||
|
||||
def test_llm_provenance_tag_prints_the_window_the_payload_sent(client: TestClient) -> None:
|
||||
"""`Δ2.0s` on `/llm` is the cell's own `window`, never a config echo (phase 23).
|
||||
|
||||
The window moved from 10 s to one poll interval without touching the page: `provenance()`
|
||||
reads `cell.window`, which is the number `tests/test_llm_source.py` pins to the LLM span, and
|
||||
both spell the label with the same one-decimal rounding (`delta_label` in
|
||||
`tests/rate_window_contract.py` is that rounding, used by both payload pins). A page printing
|
||||
the configured interval instead would advertise a window no poll measured (rule 2), and a
|
||||
literal `Δ10.0s` in the script is the same lie with the number typed in.
|
||||
"""
|
||||
js = client.get("/static/page-llm.js").text
|
||||
prov = js[js.index("function provenance("):js.index("function humanNumber(")]
|
||||
assert "`Δ${cell.window.toFixed(1)}s`" in prov, \
|
||||
f"the Δ tag is no longer built from the window the cell carries: {prov}"
|
||||
assert delta_label(LLM_WINDOW) == "Δ2.0s", "the rounding the page and the pins share moved"
|
||||
assert "rate_window_seconds" not in js, "the page echoes a config window beside a measurement"
|
||||
assert "Δ10.0s" not in js, "a hard-coded Δ is an invented window (rule 2)"
|
||||
|
||||
|
||||
def test_every_llm_cell_carries_a_window_sparkline_with_percentile_feet(client) -> None:
|
||||
"""Each panel row draws its own sparkline over the *selected* window and prints mean
|
||||
**and** p95 without a hover.
|
||||
|
||||
+21
-1
@@ -3,7 +3,7 @@ from __future__ import annotations
|
||||
|
||||
import importlib
|
||||
import os
|
||||
from dataclasses import FrozenInstanceError
|
||||
from dataclasses import FrozenInstanceError, fields
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
@@ -109,6 +109,26 @@ def test_the_disk_cadence_is_an_env_var_like_every_other_one(monkeypatch: pytest
|
||||
importlib.reload(config)
|
||||
|
||||
|
||||
def test_there_is_one_window_knob_per_measuring_cadence() -> None:
|
||||
"""Phase 23: `RATE_WINDOW_SECONDS` is the **host's** Δ window; the LLM's *is* its interval.
|
||||
|
||||
These are the defaults `app.build_hub` spans its two estimators with — the wiring is pinned in
|
||||
`tests/test_api.py`, the payload labels in `tests/test_llm_source.py` and
|
||||
`tests/test_host_source.py`, both through `tests/rate_window_contract.py`. And there is
|
||||
exactly **one** `*window*` setting: the LLM window tracks `LLM_INTERVAL_SECONDS` by
|
||||
construction, so it cannot drift from the cadence it belongs to, while the host's cannot
|
||||
shrink to match it either (a 2 s span over a 10 s poll has nothing inside it to diff, which
|
||||
`test_the_host_rates_keep_their_own_cadence_s_window` pins as `n/a`).
|
||||
"""
|
||||
settings = Settings()
|
||||
assert settings.rate_window == 10.0
|
||||
assert settings.llm_interval == 2.0
|
||||
assert settings.host_interval == 10.0
|
||||
window_knobs = {field.name for field in fields(Settings) if "window" in field.name}
|
||||
assert window_knobs == {"rate_window"}, (
|
||||
"the LLM window must stay `LLM_INTERVAL_SECONDS`: a second knob drifts from its cadence")
|
||||
|
||||
|
||||
def test_actions_need_the_flag_and_are_dead_under_read_only() -> None:
|
||||
assert Settings(nut_allow_actions=True, read_only=False).actions_allowed is True
|
||||
assert Settings(nut_allow_actions=True, read_only=True).actions_allowed is False
|
||||
|
||||
@@ -15,6 +15,7 @@ import pytest
|
||||
from dashboards.rates import RateEstimator
|
||||
from dashboards.severity import classify
|
||||
from dashboards.sources.host import HostCollector, fleet_summary, parse_host_snapshot
|
||||
from rate_window_contract import HOST_WINDOW, LLM_WINDOW, assert_window, delta_label
|
||||
|
||||
TICKS_PER_SECOND = 100.0
|
||||
|
||||
@@ -80,6 +81,41 @@ def test_second_scrape_measures_the_window_it_claims(agent_snapshot) -> None:
|
||||
assert usage == 10.0
|
||||
|
||||
|
||||
def test_the_host_rates_keep_their_own_cadence_s_window(agent_snapshot) -> None:
|
||||
"""Phase 23's other direction: a host cell labels ~10 s, the LLM cell's 2 s is not its shape.
|
||||
|
||||
`app.build_hub` gives the `host` jobs `RateEstimator(span=RATE_WINDOW_SECONDS)` and the `llm`
|
||||
jobs one spanned by `LLM_INTERVAL_SECONDS`; the payload shape is identical and only the number
|
||||
differs, so a pin naming one of them is how the smaller window could quietly reach both
|
||||
sources. `tests/test_llm_source.py` pins the 2 s side; this pins the 10 s side on the same
|
||||
scrape-the-agent-twice shape the page reads (`window_seconds` is what `/system`'s footnote
|
||||
prints).
|
||||
|
||||
The second half is the reason one estimator cannot serve both cadences: hand the host's own
|
||||
10 s poll the LLM's 2 s span and there is nothing inside the window to diff, so every rate
|
||||
answers `n/a` — the designed refusal, and the reason the host keeps `RATE_WINDOW_SECONDS`.
|
||||
"""
|
||||
second = advance(agent_snapshot, 10.0, rx_bytes=100_000, tx_bytes=50_000)
|
||||
|
||||
collector = HostCollector(RateEstimator(span=HOST_WINDOW))
|
||||
parse_host_snapshot("deskwork", agent_snapshot, collector, now=1_000.0)
|
||||
payload = parse_host_snapshot("deskwork", second, collector, now=1_010.0).payload
|
||||
assert_window(payload["window_seconds"], HOST_WINDOW,
|
||||
where="the host payload's window_seconds")
|
||||
assert delta_label(payload["window_seconds"]) == "Δ10.0s", \
|
||||
"the /system footnote prints this, and it is not the LLM page's Δ"
|
||||
assert payload["cpu"]["usage_pct"] is not None
|
||||
assert payload["network"]["rx_bytes_per_s"] is not None
|
||||
|
||||
starved = HostCollector(RateEstimator(span=LLM_WINDOW))
|
||||
parse_host_snapshot("deskwork", agent_snapshot, starved, now=1_000.0)
|
||||
narrow = parse_host_snapshot("deskwork", second, starved, now=1_010.0).payload
|
||||
assert narrow["window_seconds"] is None, \
|
||||
"a 2 s window over a 10 s poll has nothing to diff: n/a, never a rate"
|
||||
assert narrow["cpu"]["usage_pct"] is None
|
||||
assert narrow["network"]["rx_bytes_per_s"] is None
|
||||
|
||||
|
||||
def test_iowait_counts_as_idle_but_is_still_reported(agent_snapshot) -> None:
|
||||
_first, payload = parse_twice(agent_snapshot)
|
||||
modes = payload["cpu"]["modes_pct"]
|
||||
|
||||
@@ -198,7 +198,9 @@ async def test_engine_still_publishes_the_series_we_parse(engine: str,
|
||||
"gufo": ("tokens_generated_total", "tokens_prompt_total"),
|
||||
"vllm": ("generation_tokens_total", "prompt_tokens_total"),
|
||||
}[engine]
|
||||
rates = RateEstimator(span=settings.rate_window)
|
||||
# The span the LLM jobs really get: phase 23 spans the LLM estimator by the poll
|
||||
# interval, not by the host's `RATE_WINDOW_SECONDS`.
|
||||
rates = RateEstimator(span=settings.llm_interval)
|
||||
async with httpx.AsyncClient(timeout=settings.llm_timeout) as client:
|
||||
result = await collectors.fetch_engine(client, engine, base_url, api_key, rates,
|
||||
with_slots=engine == "gufo")
|
||||
@@ -224,7 +226,9 @@ async def test_gufo_slots_answer_the_pinned_shape(settings: Settings) -> None:
|
||||
"""
|
||||
if not settings.gufo_url:
|
||||
pytest.skip("GUFO_BASE_URL is unset — no gufo to ask for /slots")
|
||||
rates = RateEstimator(span=settings.rate_window)
|
||||
# The span the LLM jobs really get: phase 23 spans the LLM estimator by the poll
|
||||
# interval, not by the host's `RATE_WINDOW_SECONDS`.
|
||||
rates = RateEstimator(span=settings.llm_interval)
|
||||
async with httpx.AsyncClient(timeout=settings.llm_timeout) as client:
|
||||
result = await collectors.fetch_engine(client, "gufo", settings.gufo_url,
|
||||
settings.gufo_api_key, rates, with_slots=True)
|
||||
|
||||
+195
-1
@@ -6,10 +6,12 @@ the deployed servers (tests/fixtures/metrics_*.txt), not to upstream documentati
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Callable, Mapping, Sequence
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
|
||||
from dashboards.rates import RateEstimator
|
||||
from dashboards.rates import MIN_WINDOW_SECONDS, RateEstimator
|
||||
from dashboards.sources.llm import (
|
||||
engine_result,
|
||||
fleet_totals,
|
||||
@@ -18,6 +20,7 @@ from dashboards.sources.llm import (
|
||||
rate_key,
|
||||
series_names,
|
||||
)
|
||||
from rate_window_contract import LLM_WINDOW, assert_window, delta_label
|
||||
from slots_contract import load_slots_capture
|
||||
|
||||
|
||||
@@ -33,6 +36,29 @@ def bump_metric(text: str, name: str, delta: float) -> str:
|
||||
return bumped
|
||||
|
||||
|
||||
def poll_engine(metrics_text: Any, engine: str, span: float, stamps: Sequence[float],
|
||||
growth: Mapping[str, Callable[[float], float]]) -> list[dict[str, Any]]:
|
||||
"""Re-scrape one capture at `stamps` with counters growing as f(t), through one estimator.
|
||||
|
||||
The production shape in miniature: **one** `RateEstimator(span=span)` for every poll (the
|
||||
object `app.build_hub` hands that source), each poll stamped with its own `now`, the
|
||||
exposition carrying the tokens produced since the last scrape. Returns the parsed metrics per
|
||||
poll so a pin reads the very cells `/api/llm/snapshot` serves — including their `window`.
|
||||
"""
|
||||
parse = parse_gufo_metrics if engine == "gufo" else parse_vllm_metrics
|
||||
rates = RateEstimator(span=span)
|
||||
body = metrics_text(engine)
|
||||
carried = dict.fromkeys(growth, 0.0)
|
||||
scrapes: list[dict[str, Any]] = []
|
||||
for now in stamps:
|
||||
for name, produced in growth.items():
|
||||
value = produced(now)
|
||||
body = bump_metric(body, name, value - carried[name])
|
||||
carried[name] = value
|
||||
scrapes.append(parse(body, rates, now))
|
||||
return scrapes
|
||||
|
||||
|
||||
# ---- gufo / llama.cpp ----------------------------------------------------- #
|
||||
|
||||
def test_gufo_counters_come_straight_from_the_capture(metrics_text) -> None:
|
||||
@@ -77,6 +103,144 @@ def test_gufo_throughput_prefers_measured_delta_over_the_native_gauge(metrics_te
|
||||
assert metrics["decode_tokens_per_second_native"]["value"] == pytest.approx(42.621031668072597)
|
||||
|
||||
|
||||
def test_gufo_the_window_the_llm_jobs_are_spanned_by_is_one_that_fills(metrics_text) -> None:
|
||||
"""Phase 23: a 2 s span is a real window at the LLM cadence, and it kills the ramp.
|
||||
|
||||
`app.build_hub` spans the LLM estimator by `LLM_INTERVAL_SECONDS` (the wiring is pinned in
|
||||
`tests/test_api.py`); this is what that span buys. A slot that starts mid-window is spread
|
||||
over the whole span by the wider window, so the hero climbs one fifth of the true rate per
|
||||
poll — the "ramp up/ramp down" of `TODO.md:8` — while a window of one poll interval reads
|
||||
the engine's speed on the first poll after it changes and 0 again on the first after it ends.
|
||||
|
||||
The engine here is the capture plus a steady 50 tok/s that begins at t=1006, polled every
|
||||
2 s, which is the production cadence rather than a fast scrape: the estimator's
|
||||
`MIN_WINDOW_SECONDS` floor is 4x below this window, so nothing is being refused here.
|
||||
"""
|
||||
text = metrics_text("gufo")
|
||||
started = 1_006.0 # the slot starts here, at a steady 50 tok/s
|
||||
polls = [1_000.0, 1_002.0, 1_004.0, 1_006.0, 1_008.0, 1_010.0, 1_012.0]
|
||||
|
||||
def series(span: float) -> list[tuple[float | None, float | None]]:
|
||||
rates = RateEstimator(span=span)
|
||||
body = text
|
||||
cells: list[tuple[float | None, float | None]] = []
|
||||
previous = 0.0
|
||||
for now in polls:
|
||||
generated = 50.0 * max(0.0, now - started)
|
||||
body = bump_metric(body, "llamacpp:tokens_predicted_total", generated - previous)
|
||||
previous = generated
|
||||
cell = parse_gufo_metrics(body, rates, now)["decode_tokens_per_second"]
|
||||
cells.append((cell["value"], cell.get("window")))
|
||||
return cells
|
||||
|
||||
narrow = series(2.0)
|
||||
wide = series(10.0)
|
||||
# the new window: n/a on the first poll (no window yet), then the engine's real speed, and
|
||||
# every one of them labelled with the 2 s window it actually measured
|
||||
assert narrow[0] == (None, None)
|
||||
for _value, window in narrow[1:]:
|
||||
assert window == pytest.approx(2.0), "the label must name the window the value used"
|
||||
assert [v for v, _ in narrow[4:]] == [pytest.approx(50.0)] * 3, (
|
||||
f"a 2 s window reads the slot's speed at once, got {narrow}")
|
||||
# the old one: the same work, averaged across the polls the operator is watching
|
||||
assert [v for v, _ in wide[4:]] != [pytest.approx(50.0)] * 3, \
|
||||
f"the 10 s window no longer smooths, so this test's premise moved: {wide}"
|
||||
assert wide[4][0] is not None and wide[4][0] < 30.0, \
|
||||
f"expected the ramp to print well under the true rate, got {wide}"
|
||||
|
||||
|
||||
def test_gufo_rate_cells_name_the_window_the_llm_jobs_are_measured_over(metrics_text) -> None:
|
||||
"""Phase 23's payload pin: an LLM rate cell's `window` is the LLM cadence, label and all.
|
||||
|
||||
`build_hub` spans the `llm` estimator by `LLM_INTERVAL_SECONDS` (the object-level pin is
|
||||
`tests/test_api.py`'s); this is the same span where the browser reads it — the `window` beside
|
||||
the value, the `computed:Δ<series>/Δt` provenance printed with it, and the `Δ2.0s` label
|
||||
`provenance()` builds from that number. **Both** throughput cells are checked, on the
|
||||
production 2 s cadence. `tests/test_host_source.py` is the other direction of the same pin:
|
||||
the host cells keep 10 s in the same payload shape.
|
||||
"""
|
||||
started = 1_006.0
|
||||
stamps = [1_000.0, 1_002.0, 1_004.0, 1_006.0, 1_008.0, 1_010.0]
|
||||
scrapes = poll_engine(
|
||||
metrics_text, "gufo", LLM_WINDOW, stamps,
|
||||
{"llamacpp:tokens_predicted_total": lambda t: 50.0 * max(0.0, t - started),
|
||||
"llamacpp:prompt_tokens_total": lambda t: 200.0 * max(0.0, t - started)},
|
||||
)
|
||||
assert "window" not in scrapes[0]["decode_tokens_per_second"], \
|
||||
"one sample is not a window, so it must not grow a Δ label"
|
||||
for key, series, speed in (
|
||||
("decode_tokens_per_second", "llamacpp:tokens_predicted_total", 50.0),
|
||||
("prompt_tokens_per_second", "llamacpp:prompt_tokens_total", 200.0),
|
||||
):
|
||||
cells = [scrape[key] for scrape in scrapes[1:]]
|
||||
for cell in cells:
|
||||
assert cell["value"] is not None, f"{key} went n/a on a healthy cadence: {cell}"
|
||||
assert_window(cell.get("window"), LLM_WINDOW, where=f"gufo {key}")
|
||||
assert cell["derived"] is True
|
||||
assert cell["source"] == f"computed:Δ{series}/Δt", \
|
||||
"the provenance string travels with the value whatever window it used"
|
||||
assert cells[-1]["value"] == pytest.approx(speed), (
|
||||
f"a one-interval window reads the engine's speed on the next poll, got {cells}")
|
||||
|
||||
|
||||
def test_gufo_the_floor_still_refuses_a_catch_up_gap_at_the_llm_window(metrics_text) -> None:
|
||||
"""The phase-13 floor belongs to the estimator, not to the old 10 s span.
|
||||
|
||||
Shrinking the window to one poll interval must not shrink the guard: `poller.Job._loop` still
|
||||
sleeps `max(0.2, interval - elapsed)`, so a slow fetch is still followed by a catch-up poll
|
||||
~0.2 s later, and 0.2 s of a 2 s window is the same lie a 0.2 s slice of a 10 s one was (the
|
||||
3284.59 tok/s row for a 53 tok/s engine). `test_rates.py` pins the gate at both spans; this
|
||||
is the refusal through the production parse path, where the `n/a` keeps the window it refused
|
||||
**and** its `source` so the cell still says which Δ it is talking about.
|
||||
"""
|
||||
scrapes = poll_engine(
|
||||
metrics_text, "gufo", LLM_WINDOW,
|
||||
[1_000.0, 1_002.0, 1_004.0, 1_014.25, 1_014.47, 1_016.47],
|
||||
{"llamacpp:tokens_predicted_total": lambda t: 50.0 * (t - 1_000.0)},
|
||||
)
|
||||
stalled = scrapes[3]["decode_tokens_per_second"]
|
||||
assert stalled["value"] is None, \
|
||||
"a 10 s gap is past a 2 s span's ceiling (2 x span): n/a, not a diluted rate"
|
||||
assert "window" not in stalled, "the ceiling answers no window because it measured none"
|
||||
refused = scrapes[4]["decode_tokens_per_second"]
|
||||
assert refused["value"] is None, f"0.22 s is the loop's catch-up gap, not a rate: {refused}"
|
||||
assert refused["window"] == pytest.approx(0.22)
|
||||
assert refused["window"] < MIN_WINDOW_SECONDS
|
||||
assert refused["source"] == "computed:Δllamacpp:tokens_predicted_total/Δt"
|
||||
assert delta_label(refused["window"]) == "Δ0.2s", "the label names the window it refused"
|
||||
recovered = scrapes[5]["decode_tokens_per_second"]
|
||||
assert recovered["value"] == pytest.approx(50.0), (
|
||||
f"the refusal re-bases, so the next honest poll reads the engine again: {recovered}")
|
||||
assert_window(recovered.get("window"), LLM_WINDOW, where="gufo decode")
|
||||
|
||||
|
||||
def test_gufo_a_poll_gap_a_hair_wider_than_the_span_is_still_the_2s_claim(metrics_text) -> None:
|
||||
"""Live reality: a poll lands slightly after its interval, and that is still `Δ2.0s`.
|
||||
|
||||
With the span equal to the cadence (phase 23) only the current sample is strictly *inside* the
|
||||
window, so the base comes from the estimator's window-empty fallback — the previous sample,
|
||||
one poll back — and the published window is the real gap. Read off the running dev server the
|
||||
LLM cells report 2.009-2.026 s and the host payload's `window_seconds` 9.978-10.002 s, so a
|
||||
pin demanding `window == span` exactly would fail against working software. This drives the
|
||||
jittered shape and asserts what the operator sees: the same `Δ2.0s` label and the engine's real
|
||||
speed. A gap of a whole extra interval is the other story — the `2 x span` ceiling refuses it,
|
||||
which is what the test above pins.
|
||||
"""
|
||||
started = 1_006.0
|
||||
stamps = [1_000.0, 1_002.03, 1_004.07, 1_006.1, 1_008.13, 1_010.16]
|
||||
scrapes = poll_engine(
|
||||
metrics_text, "gufo", LLM_WINDOW, stamps,
|
||||
{"llamacpp:tokens_predicted_total": lambda t: 50.0 * max(0.0, t - started)},
|
||||
)
|
||||
cells = [scrape["decode_tokens_per_second"] for scrape in scrapes[1:]]
|
||||
for cell in cells:
|
||||
assert cell["value"] is not None
|
||||
assert_window(cell.get("window"), LLM_WINDOW, where="gufo decode on a jittered cadence")
|
||||
assert delta_label(cell["window"]) == "Δ2.0s", \
|
||||
"a 2.03 s gap still prints the window the panel claims"
|
||||
assert cells[-1]["value"] == pytest.approx(50.0, rel=0.02)
|
||||
|
||||
|
||||
def test_gufo_prompt_rate_excludes_cached_tokens(metrics_text) -> None:
|
||||
text = metrics_text("gufo")
|
||||
rates = RateEstimator(span=10)
|
||||
@@ -244,6 +408,36 @@ def test_vllm_measured_rate_when_the_build_has_no_throughput_gauge(metrics_text)
|
||||
assert metrics["decode_tokens_per_second"]["value"] == pytest.approx(25.0)
|
||||
|
||||
|
||||
def test_vllm_rate_cells_name_the_same_window_the_llm_jobs_are_measured_over(
|
||||
metrics_text) -> None:
|
||||
"""The pin is the **source's** span, so both engines answer it the same way.
|
||||
|
||||
The capture exports no throughput gauge, which is what makes these two cells measured (see
|
||||
`test_vllm_measured_rate_when_the_build_has_no_throughput_gauge`), and vLLM's counters are
|
||||
labelled — the estimator keys each label set separately while the cell's `source` names the
|
||||
series. Same two rules as the gufo pin: the `window` inside the LLM span, and
|
||||
`computed:Δ<series>/Δt` beside the value.
|
||||
"""
|
||||
started = 2_006.0
|
||||
scrapes = poll_engine(
|
||||
metrics_text, "vllm", LLM_WINDOW,
|
||||
[2_000.0, 2_002.0, 2_004.0, 2_006.0, 2_008.0, 2_010.0],
|
||||
{"vllm:generation_tokens_total": lambda t: 40.0 * max(0.0, t - started),
|
||||
"vllm:prompt_tokens_total": lambda t: 300.0 * max(0.0, t - started)},
|
||||
)
|
||||
for key, series, speed in (
|
||||
("decode_tokens_per_second", "vllm:generation_tokens_total", 40.0),
|
||||
("prompt_tokens_per_second", "vllm:prompt_tokens_total", 300.0),
|
||||
):
|
||||
cells = [scrape[key] for scrape in scrapes[1:]]
|
||||
for cell in cells:
|
||||
assert cell["value"] is not None, f"{key} went n/a on a healthy cadence: {cell}"
|
||||
assert_window(cell.get("window"), LLM_WINDOW, where=f"vllm {key}")
|
||||
assert cell["source"] == f"computed:Δ{series}/Δt"
|
||||
assert cells[-1]["value"] == pytest.approx(speed), (
|
||||
f"the labelled series still reads its own speed over one interval, got {cells}")
|
||||
|
||||
|
||||
def test_vllm_a_joining_label_set_does_not_lend_its_carried_counter(metrics_text) -> None:
|
||||
"""A second `engine=`/`model_name=` series appearing mid-window must not be a rate.
|
||||
|
||||
|
||||
@@ -12,6 +12,7 @@ import math
|
||||
import pytest
|
||||
|
||||
from dashboards.rates import RateEstimator
|
||||
from rate_window_contract import HOST_WINDOW, LLM_WINDOW
|
||||
|
||||
|
||||
def test_first_sample_has_no_rate() -> None:
|
||||
@@ -210,6 +211,65 @@ def test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor() -> None:
|
||||
assert published[-1] == pytest.approx(45.0), "the stall ends in the truth, not a tail"
|
||||
|
||||
|
||||
# --- phase 23: the gates are the estimator's, not the old 10 s span's --------- #
|
||||
# `app.build_hub` now spans the `llm` jobs by `LLM_INTERVAL_SECONDS` (2 s), a fifth of the host's
|
||||
# `RATE_WINDOW_SECONDS`. Everything phase 13 built has to survive the shrink or the smaller window
|
||||
# is a licence for the same 3284.59 tok/s row, so each gate runs here at both spans. The numbers
|
||||
# scale with the span: what these three pin is the *relation*, which is what "unchanged" means.
|
||||
# The fourth protection — a held jump needing a **different anchor** — is deliberately not
|
||||
# parametrized: it compares two sample timestamps and no span arithmetic, and a 2 s window holds
|
||||
# one sample per poll interval anyway, so a stale anchor ages out one poll later instead of five.
|
||||
# Its span-10 regression pair (`test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor`
|
||||
# above, `test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` in
|
||||
# `tests/test_llm_source.py`) runs unchanged, and the floor row of this section is the one that
|
||||
# could actually have been weakened by the smaller window: `MIN_WINDOW_SECONDS` is absolute.
|
||||
|
||||
@pytest.mark.parametrize("span", [LLM_WINDOW, HOST_WINDOW])
|
||||
def test_the_floor_refuses_the_catch_up_gap_at_either_window(span: float) -> None:
|
||||
"""The poll loop's ~0.2 s catch-up is a gap whatever the panel's span is.
|
||||
|
||||
0.22 s is a tenth of a 10 s window and an eighth of a 2 s one, and neither is a measurement:
|
||||
the floor is absolute on purpose (phase 13 pinned that), so the LLM's narrower claim is not a
|
||||
narrower guard.
|
||||
"""
|
||||
r = RateEstimator(span=span)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
r.rate("x", 50.0 * span, now=span) # the honest cadence: 50 tok/s
|
||||
rate, window = r.rate("x", 50.0 * span + 11.0, now=span + 0.22)
|
||||
assert rate is None, f"a 0.22 s catch-up became a rate at span={span}"
|
||||
assert window == pytest.approx(0.22), "the n/a still names the window it refused"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("span", [LLM_WINDOW, HOST_WINDOW])
|
||||
def test_the_ceiling_is_two_spans_at_either_window(span: float) -> None:
|
||||
"""`2 x span` is relative, so a stall stays a stall at either window size."""
|
||||
r = RateEstimator(span=span)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
assert r.rate("x", 100.0, now=2.5 * span) == (None, None), "past the ceiling: no rate"
|
||||
rate, window = r.rate("x", 200.0, now=3.5 * span)
|
||||
assert rate == pytest.approx(100.0 / span)
|
||||
assert window == pytest.approx(span), "the next poll re-bases on the fresh sample"
|
||||
|
||||
|
||||
@pytest.mark.parametrize("span", [LLM_WINDOW, HOST_WINDOW])
|
||||
def test_a_jump_is_held_and_confirmed_at_either_window(span: float) -> None:
|
||||
"""An 8x step holds one poll at a 2 s window exactly as at a 10 s one.
|
||||
|
||||
The ratio is dimensionless, so narrowing the window does not loosen the gate: a real burst
|
||||
outlives one poll interval of either size and prints one poll late, while the held value keeps
|
||||
the `Δ<window>` label it was published with.
|
||||
"""
|
||||
r = RateEstimator(span=span)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
r.rate("x", 10.0 * span, now=span) # 10 tok/s published
|
||||
rate, window = r.rate("x", 90.0 * span, now=2.0 * span) # 80 tok/s = 8x: held
|
||||
assert rate == pytest.approx(10.0)
|
||||
assert window == pytest.approx(span), "the held value keeps the Δ label it was published with"
|
||||
rate, window = r.rate("x", 172.0 * span, now=3.0 * span) # 82: a new anchor confirms it
|
||||
assert rate == pytest.approx(82.0)
|
||||
assert window == pytest.approx(span)
|
||||
|
||||
|
||||
def test_a_rise_from_zero_is_never_held() -> None:
|
||||
"""The first work an idle engine does must print on its own poll, not one late."""
|
||||
r = RateEstimator(span=10)
|
||||
|
||||
Reference in New Issue
Block a user