Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 ducoterra
|
1bbd868401 | ||
|
ducoterra
|
04d934dd94 |
@@ -0,0 +1,44 @@
|
||||
# Phase 15 — Protected exporter web servers, auth by certificate
|
||||
|
||||
**Source:** `TODO.md` L3 — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
**Story:** n/a
|
||||
**Context:** `agent/host_agent.py` (the stdlib-only exporter web server; today: a bearer token via `--token`/`--token-file`, refuses an off-loopback bind without one, `ThreadingHTTPServer`, the `resolve_token` idiom), `src/dashboards/config.py` (`Target`/`parse_targets` — token in URL userinfo, `https://` URLs already accepted; **no change needed**), `src/dashboards/collectors.py` (`fetch_snapshot`, the one `UpstreamError`, redacted display URLs, the one shared `httpx.AsyncClient` from the lifespan), `src/dashboards/app.py` (the `PAGES` dict with the settings-page comment, `build_hub`/`build_context`, the lifespan that builds `ctx.auth`), `src/dashboards/api.py` (`AppContext`, the gated-write + event pattern), `src/dashboards/store.py` (`get_meta`/`set_meta`/`delete_meta` — the only durable key/value state, home of `auth.*`), `deploy/install-agent.sh` (the token-file idiom `0640 root:agent-user`, the `ProtectSystem=strict` unit with no `RuntimeDirectory`, the `--status` probe), `tests/test_agent.py` (the in-process agent harness), `tests/test_collectors.py`, `tests/test_deploy_scripts.py`, `tests/test_api.py` (the route walk, the settings-page pin, the orphan-id audit), `src/dashboards/web/pages/settings.html` + `src/dashboards/web/static/page-settings.js` (the "a page, not a dashboard" pattern the new `/certs` admin page follows)
|
||||
|
||||
## Objective
|
||||
|
||||
Every exporter (host agent) web server is protected end-to-end by certificates the dashboard itself generates and manages: the dashboard holds a root CA in the `meta` table, signs its own client cert and each agent's server cert, the agent serves TLS with a per-path client-cert check (only the dashboard can read it), the dashboard dials `https://` with its client cert, and a new `/certs` admin page lists every cert with its expiry and status and triggers rotation, revocation, and bundle re-delivery. The operator's stated follow-up — firewalling upsd 3051 and talking to the agent only — is the motivation, not part of this phase. The agent's bearer-token mode stays a compatibility mode; the dashboard's own login is a separate auth domain and is untouched.
|
||||
|
||||
**Scope (operator-confirmed):** "exporters" = the host agents this repo installs. gufo / vLLM / upsd are third-party services this repo does not deploy; their TLS (e.g. upsd `STARTTLS`) is a separate operator concern.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `14_store_performance` (todo) — **shared-file, run-order note**: 14 rewrites the hot paths in `store.py`/`api.py`; 15 only *adds* meta keys and new routes/pages. Apply on top of the file state at the time it runs; never restore a line the other phase owns.
|
||||
- `11_admin_login` (complete) — the session gate (invariant 12) for the new routes, the gated-write + CSRF + event pattern for the new `POST /api/certs/*`, the `meta` helpers, and the settings-page pattern `/certs` follows.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_cert_management.md` — `src/dashboards/certs.py`: the dashboard-managed root CA, client cert, per-agent server certs, revocation and reissue, all in `meta`; the `cryptography` dependency (dashboard-side only).
|
||||
2. `02_agent_tls.md` — `host_agent.py` TLS mode: one installed `--cert-bundle`, the per-path client-cert check, the bundle revocation list, `bundle_rev` in the snapshot; token mode unchanged.
|
||||
3. `03_collector_tls.md` — the dashboard dials `https://` agent URLs with its client cert, verifies against its root, refuses a revoked agent without dialing; error sentences stay redacted.
|
||||
4. `04_cert_api.md` — the gated `/api/certs` surface: list, per-host bundle, rotate, revoke, root-rotate; fixed-text events, no keys out of `meta`.
|
||||
5. `05_certs_admin_page.md` — the `/certs` admin page (the settings pattern): every cert with its expiry and status, issued vs deployed bundle rev, rotate/revoke/root-rotate actions, bundle download.
|
||||
6. `06_deploy_cert_bundle.md` — `install-agent.sh --cert-bundle` installs `/etc/dashboards-agent.certs` (`0600 root:agent-user`), the unit gains the flag + `RuntimeDirectory`, `--status` probes over TLS; the script pins move with it.
|
||||
7. `07_live_rollout.md` — the two live agents flip to TLS, `HOST_TARGETS` goes `https://`, the live suite + health table + screenshot prove it; the `driveripper` install procedure is documented.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- Unit: `tests/test_certs.py` (new) — the whole lifecycle against a dict `MetaStore`, exactly how `tests/test_auth.py` does it.
|
||||
- Integration: `tests/test_agent.py` (the TLS handshake matrix against the in-process agent), `tests/test_collectors.py` (the `https` dial, the revoked refusal with a no-dial spy, the error sentences), `tests/test_api.py` (the `/api/certs` shapes, the gate/CSRF on the new writes, the served `/certs` page pins), `tests/test_deploy_scripts.py` (the new flags/files/permissions).
|
||||
- Live: task 07 (AGENTS.md invariant 10 — the dev server stays up; screenshots in `data/logs/`).
|
||||
- Coverage: **>90%** on new/modified code; the 94% gate holds. `uv pip list --outdated` empty (the `cryptography` dependency — invariant 8).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] a fresh database boots with a `meta` root CA, generated once and idempotent across restarts (pinned in `tests/test_certs.py`)
|
||||
- [ ] an agent run with a bundle answers `/snapshot` only to a client cert signed by its root; a wrong-CA or certless client is refused; a revoked serial gets the 403; `/health` stays probeable with the CA only
|
||||
- [ ] with `https://` `HOST_TARGETS`, the dev server's `host:`/`disks:` jobs are `ok`, and a revoked host says "certificate revoked" without any dial (spy-pinned)
|
||||
- [ ] signed-in `GET /certs` lists every cert with expiry and status; rotate/revoke/root-rotate work through the gated API; the bundle download is byte-identical to the stored bundle
|
||||
- [ ] the installer places the bundle `0600 root:agent-user`, the unit comes up in TLS mode, `--status` is green over TLS
|
||||
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green, `uv pip list --outdated` empty
|
||||
- [ ] live: deskwork + gateway in TLS mode, `https://` dials, live suite green, screenshots in `data/logs/`
|
||||
- [ ] no behaviour change in completed phases: the token-mode agent is byte-identical, the engine/UPS polling is untouched, the public surface is unchanged (the route walk)
|
||||
@@ -0,0 +1,43 @@
|
||||
# Task 01 — The dashboard owns a root CA and every cert it signs
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
|
||||
## Objective
|
||||
|
||||
The certificate lifecycle becomes a pure, testable module: the dashboard generates a root CA once (idempotent across restarts), signs its own client cert and each agent's server cert, and supports reissue and revocation — all state in the `meta` table, exactly how `auth.py` keeps the verifier and the session secret.
|
||||
|
||||
## Work
|
||||
|
||||
1. `pyproject.toml` — add `cryptography` to `dependencies` (dashboard-side only; the agent stays stdlib-only) and run `uv lock`. Invariant 8: `uv pip list --outdated` empty afterwards.
|
||||
- ASSUMPTION: `cryptography` is the only new dependency and it stays on the dashboard side; the agent (task 02) gains no third-party code — it loads the PEMs with stdlib `ssl`.
|
||||
2. `src/dashboards/certs.py` (new) — the whole lifecycle as pure functions over the three-method `MetaStore` protocol (the same protocol `auth.load_auth_state` uses, so this module imports nothing else from the package, and its tests use a dict):
|
||||
- `ensure_certs(store, *, now: float | None = None)` → a frozen `CertsState` carrying the PEM text (`root_key`, `root_cert`, `client_key`, `client_cert`) plus the root/client `serial` + `not_after`. Reads the `meta` rows first; only generates and persists when missing (first boot); an existing row wins and is **never regenerated on restart** — regenerating on every start would rotate the CA and invalidate every installed agent bundle. `now` is injected (the module never reads a clock itself — the `auth.py` rule).
|
||||
- meta keys (one concern per key, the `auth.*` precedent): `certs.root.key`, `certs.root.cert`, `certs.client.key`, `certs.client.cert`, `certs.agents` (one JSON object mapping `host name` → `{serial, not_after, status, bundle_rev, bundle, san}` — the bundle text is stored, so what the agent holds is byte-identical to what the API serves).
|
||||
- Shapes: root CA self-signed, CN `dashboards-root`, **10 years**; client cert signed by the root, CN `dashboards`, **1 year**; agent server certs signed by the root, CN = the host name, **1 year**, `SAN` = the address(es) the dashboard dials for that host.
|
||||
- ASSUMPTION: 10 y root / 1 y leaf lifetimes. No env seed for the root CA (unlike the admin password, the root key is a secret only the app needs to know, so it is auto-generated on first boot and merely persisted).
|
||||
- `issue_agent_cert(store, state, name, san: list[str])` / `reissue_agent_cert(store, state, name, san)` — a fresh serial, `bundle_rev` +1 (1 on first issue), `status: "active"`, a fresh `bundle` (below). Idempotent per call — never issued on a read.
|
||||
- `rotate_root(store, state)` — a fresh root CA + fresh client cert + every existing agent entry reissued (each `bundle_rev` +1): one action re-arms the whole fleet.
|
||||
- `revoke(store, state, name)` — `status: "revoked"` (the serial stays on the denylist until a reissue gives the host a fresh identity; there is no "unrevoke" — reissue is the fix).
|
||||
- the **bundle** (the install artifact, the TODO's "root certificate installed with each exporter"): one PEM text = the agent's server cert + the agent's key + the root CA *cert* (never the root key) + a `# BUNDLE-REV <n>` line + a `# REVOKED <serial>` line (one per revoked serial of the *current* root generation — a root rotation resets the list, since every cert is reissued anyway).
|
||||
- ASSUMPTION: one client cert for the whole dashboard (the agent verifies "signed by our root", not a per-process identity); one agent entry per `HOST_TARGETS` name.
|
||||
- invariant 6, the `auth.py` rule: no `logging` and no raise with anything interpolated — a malformed stored PEM is a fixed-text `RuntimeError` telling the operator to delete the `certs.*` meta keys (the shape of the auth reset), and nothing else.
|
||||
3. `src/dashboards/app.py` (+ the `AppContext` in `src/dashboards/api.py`) — the startup path builds `ctx.certs` (an `ensure_certs` read, the one-time bootstrap included) **before** `build_hub`, in both the owned-store (`build_context`) and the injected-store (lifespan) branches, through **one** helper so the two wiring sites cannot drift (the `build_hub` rule in AGENTS.md); the PEMs are written once to a private temp dir (mode `0600`, lives as long as the process) and the *paths* are what `AppContext.certs` carries for task 03's dialing.
|
||||
- A `certs` whose stored PEM is malformed (a truncated `meta` row) is a fail-closed startup — a dashboard that cannot present its own credential must not start the loops (the `ctx.auth` rule, the one deliberate raise).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- `tests/test_certs.py` (new) — a dict `MetaStore` (the `tests/test_auth.py` idiom):
|
||||
- the first `ensure_certs` call persists exactly the five `certs.*` keys; the second is byte-identical (no regeneration — pin the `certs.root.cert` value);
|
||||
- the parsed chain (`cryptography.x509` loaders) verifies: client → root, agent → root; the SAN list appears in the agent cert; the expiries match the assumed lifetimes (injected `now`);
|
||||
- `reissue_agent_cert` bumps `serial` and `bundle_rev` and sets `status` active; `revoke` flips the status and puts the serial on the bundle's `# REVOKED` line; a reissue after a revoke gives a fresh serial that is *not* on the list;
|
||||
- `rotate_root` reissues every agent entry (each `bundle_rev` bumped), the bundle carries the new root cert, and no stored value still contains the old root key;
|
||||
- the bundle contains the agent cert + key + root cert + `# BUNDLE-REV` + the `# REVOKED` lines and **never** the root key (assert via the PEM header and a parsed-key mismatch);
|
||||
- a malformed stored PEM → the fixed-text `RuntimeError` naming the meta keys, nothing interpolated.
|
||||
- Coverage: **>90%** on this task's new code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/test_certs.py` green; the full suite green
|
||||
- [ ] `uv pip list --outdated` empty; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||
- [ ] a fresh store boots with the `certs.*` rows present; a second boot changes no byte of `certs.root.cert`
|
||||
- [ ] no behaviour change in completed work: `auth.*` keys, the public surface, and the poller wiring are untouched
|
||||
@@ -0,0 +1,37 @@
|
||||
# Task 02 — The agent serves a protected TLS web server from one installed bundle
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
|
||||
## Objective
|
||||
|
||||
`host_agent.py` gains a TLS mode that makes the web server protected by certificate: one installed bundle holds the server cert + key, the root CA, a bundle revision, and a revocation list; the agent requires a client cert signed by the root on every path except `/health` (mirroring the token-mode `/health` exemption, so an installer probe can reach `/health` with the CA only), and it reports the bundle revision it loaded so the dashboard can tell a stale bundle from a fresh one. The token mode is byte-identical to today.
|
||||
|
||||
## Work
|
||||
|
||||
1. `agent/host_agent.py` — stdlib only (an `ssl` import; no new third-party dependency):
|
||||
- a new flag `--cert-bundle <file>` (env `AGENT_CERT_BUNDLE`): one PEM text containing the server cert, the server key, the root CA cert, a `# BUNDLE-REV <n>` line, and `# REVOKED <serial>` lines. `resolve_bundle(path) -> (context, bundle_rev, revoked: frozenset[str], error)` is a pure function (the `resolve_token` idiom): parse the PEM blocks by text, extract the three blocks, write them to a temp dir for `load_cert_chain` (into `/run/dashboards-agent` when present and writable — the unit's `RuntimeDirectory`, task 06 — else `tempfile.mkdtemp`), and build the context.
|
||||
- The TLS context: `ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)`, `load_cert_chain`, `load_verify_locations(ca)`, `verify_mode = ssl.CERT_OPTIONAL` — the handshake is optional and the **handler** enforces it: on every path except `/health`, `self.connection.getpeercert()` must carry a verified certificate (subject present), else `403 {"ok": false, "detail": "client certificate required"}`; that cert's serial must not be in the bundle's `# REVOKED` set, else `403 {"ok": false, "detail": "client certificate revoked"}`. The denylist file is re-read on mtime change per request (a small file; a redeployed bundle takes effect without a restart — the reason the `# REVOKED` lines live in the bundle at all).
|
||||
- The startup gate: off-loopback with neither a token nor a bundle → the existing exit-2 refusal (its message now names the bundle too); a bundle present → the off-loopback bind is allowed (the client cert is the protection); a token plus a bundle → the token check is skipped on the TLS path (the handshake is the auth).
|
||||
- The snapshot payload gains two **additive** top-level fields in TLS mode: `tls: true` and `bundle_rev: <n>` (absent in token mode — an additive field never changes an existing shape); the `serve()` start line says `tls (client cert required, rev <n>)`.
|
||||
- `AGENT_VERSION` 1.0.0 → 1.1.0; the docstring's deployment paragraph gains one line for the TLS mode (no flavour text — the AGENTS.md audit rule).
|
||||
- ASSUMPTION: `CERT_OPTIONAL` + the per-path check (not `CERT_REQUIRED`) is what keeps `/health` probeable with the CA only — the installer's `--status` and the systemd probe (task 06) cannot present a client cert the installer does not hold, and a probe that cannot health-check is a dead install.
|
||||
- ASSUMPTION: the agent never generates or renews certificates (stdlib cannot mint x509); the bundle is the artifact the dashboard signs and the installer places.
|
||||
2. `tests/test_agent.py` — the existing in-process harness (`serve()` in a thread on `127.0.0.1`, `urllib`/`ssl` clients), with a test CA + leafs generated by `src/dashboards/certs.py` over a dict store (no second generator):
|
||||
- a client with a matching client cert → `/snapshot` 200 with `tls: true` and `bundle_rev` equal to the bundle's;
|
||||
- a client with no cert → the handshake fails or a 403 on a non-`/health` path; `/health` still 200 with a CA-only context (the probe path);
|
||||
- a client signed by a *different* CA → refused;
|
||||
- a client whose serial is in the bundle's `# REVOKED` list → 403 "revoked"; rewrite the bundle file without the line (mtime change) → the same client is accepted on the next request (the denylist re-read);
|
||||
- a token-mode agent (no bundle): the existing tests unchanged and green (byte-identical behaviour — no `tls`/`bundle_rev` fields);
|
||||
- `resolve_bundle` error paths: a missing file, an empty file, a bundle missing the key block, a bundle missing the `# BUNDLE-REV` line (an error — a revless bundle is not one this app signed; ASSUMPTION: the `# BUNDLE-REV` line is mandatory).
|
||||
3. `deploy/` is untouched here (task 06); the non-systemd fallback (`tempfile.mkdtemp`) keeps dev and tests working.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- the `tests/test_agent.py` matrix above; the existing token-mode tests stay green unchanged.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] a TLS-mode agent on a loopback port answers the full handshake matrix (tests) and reports `bundle_rev`
|
||||
- [ ] a token-mode agent is byte-identical to today (existing tests green, no new fields present)
|
||||
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||
@@ -0,0 +1,35 @@
|
||||
# Task 03 — The dashboard dials agents over https with its own client cert
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
|
||||
## Objective
|
||||
|
||||
An `https://` `HOST_TARGETS` entry is now a certificate-authenticated dial: the dashboard presents the client cert task 01 gives it, verifies the agent against its own root CA, refuses a *revoked* agent before dialing (invariant 3's shape — a source it knows is dead says so without a network), and passes the agent's `tls`/`bundle_rev` through the `system` and `disks` payloads so task 05's page can compare the issued rev against the deployed one. Engines and token-mode agents are untouched.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/config.py` — **no change** (an `https://` URL already parses; a TLS host carries no userinfo, and the userinfo token stays available for token-mode hosts).
|
||||
2. `src/dashboards/collectors.py` —
|
||||
- a small frozen `AgentTls` dataclass: `verify: str` (the CA path), `cert: tuple[str, str]` (the client cert + key paths), `status: Callable[[], str]` — the `status` callable returns `"active"`/`"revoked"` from a **fresh** `get_meta("certs.agents")` read (a small-table `SELECT`, cheap at poll cadence, and phase 14's work keeps it off the hot paths; reading it live is what makes a runtime revocation take effect on the next poll without a restart).
|
||||
- `fetch_snapshot(client, target, *, timeout=None, tls: AgentTls | None = None)`: when `tls` is given **and** `target.url` is `https://`, the request passes `verify=<ca path>` and `cert=(cert path, key path)` to `client.get` (httpx per-request overrides — the shared client dials both plain and TLS upstreams, so a per-client `verify` would poison the engine dials).
|
||||
- Revocation before any network: `tls is not None` and `tls.status() == "revoked"` → `UpstreamError(f"{target.name}: agent certificate revoked")` **before** `client.get` — the poller's stale/last-good semantics then do the rest (invariant 4).
|
||||
- A `403` from an agent gets its own honest sentence (the existing `401` branch keeps the token-mode wording): `f"{target.name}: agent refused the client certificate ({detail})"` with the `detail` from the agent's JSON (`client certificate required` / `client certificate revoked`), the redacted `display_url` (invariant 6) like every message in this function.
|
||||
- `fetch_host_snapshot`/`fetch_disk_snapshot` pass the `tls` through; the per-host `AgentTls` is plumbed from `AppContext.certs` into the `host_jobs`/`disks_jobs` lambdas (a new optional `tls` parameter on both `*_jobs`, `build_hub` takes it — one call site per branch so the production and injected wiring cannot drift, the `build_hub` rule in AGENTS.md). `status` reads from the store the jobs already hold.
|
||||
3. `src/dashboards/app.py` (+ the `AppContext` in `api.py`) — `AppContext.certs` (task 01) exposes the temp-dir paths (`ca`, `client_cert`, `client_key`); `build_context` and the lifespan's injected branch build the per-host `AgentTls` objects **before** `build_hub`.
|
||||
4. `src/dashboards/sources/host.py` + `src/dashboards/sources/disks.py` — pass the agent's `tls`/`bundle_rev` snapshot fields through into the per-host payload (additive, the `degraded` idiom) so `/api/system/snapshot` and `/api/disks/snapshot` carry the deployed rev. **No** `severity` for a rev mismatch — a count is not a graded number (the fleet-header rule), the page colours nothing for it.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- `tests/test_collectors.py` — an in-process TLS agent (task 02's harness, in a thread) + `AgentTls` material generated by `certs.ensure_certs` over a dict store:
|
||||
- an `https` target with `tls` → the 200 body; a missing client-cert file and a wrong-CA failure still surface as `UpstreamError` with the redacted URL;
|
||||
- a revoked status (dict store with `status: "revoked"`) → the error is raised and a spy on `client.get` (the `tests/test_api.py` spy idiom) proves **no dial happened**;
|
||||
- an `http` token-mode target: the existing tests unchanged;
|
||||
- the 403 sentence: a scripted agent 403 (a `tests/conftest.py` fake) → the `detail` appears in the message, the URL redacted;
|
||||
- the payload passthrough: a snapshot with `tls: true, bundle_rev: 3` → both the `system` and the `disks` payloads carry it.
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest` green; the engine/UPS/token-host paths are byte-identical (existing pins hold)
|
||||
- [ ] a revoked host fails its `host:` and `disks:` jobs with "certificate revoked" and no network dial (spy-pinned)
|
||||
- [ ] `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||
@@ -0,0 +1,36 @@
|
||||
# Task 04 — The gated /api/certs surface: list, bundle, rotate, revoke, root-rotate
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
|
||||
## Objective
|
||||
|
||||
The API half of "the collector manages the certs": a signed-in operator can list every cert with its expiry and status, download the exact bundle an agent should hold, and trigger rotation / revocation / root rotation. The routes dial nothing (invariant 1's shape), are gated like every other write (invariant 12), and write fixed-text events — a key never leaves `meta` (invariant 6).
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/api.py` — the new routes (the middleware gates them by default; the route-walk test covers them for free):
|
||||
- `GET /api/certs` — `{root: {serial, not_after}, client: {serial, not_after}, agents: [{name, serial, not_after, status, bundle_rev, deployed_bundle_rev, san}]}`. `deployed_bundle_rev` is the `bundle_rev` from the host's last `system` job payload (the same source `alerts.py` reads job states from); a host that never answered → `null`. Reads never issue (no side effects on a read — a `GET` that mints a cert is a surprise).
|
||||
- `GET /api/certs/<name>/bundle` — the stored bundle text, `Content-Type: application/x-pem-file`, `Content-Disposition: attachment; filename="agent-<name>.bundle"` — byte-identical to what `certs.py` stored (the installer's input, task 06/07).
|
||||
- `POST /api/certs/<name>/rotate` — a reissue (`certs.reissue_agent_cert`, the SANs taken from the configured `HOST_TARGETS` URL host(s) + the name); a `cert-rotated` event (a fixed message naming the host — no PEM, no key).
|
||||
- `POST /api/certs/<name>/revoke` — `certs.revoke`; a `cert-revoked` event. The next poll refuses without dialing (task 03) — that is the enforcement; the event is the audit line.
|
||||
- `POST /api/certs/root/rotate` — `certs.rotate_root`; a `cert-root-rotated` event; the response carries `{"ok": true, "restart_required": true}` — the client cert and CA are startup material, so a new root is fully in force at the next process start (the page, task 05, prints that; the honest constraint, not a hidden one).
|
||||
- ASSUMPTION: a root rotation requires a dashboard restart to be fully in force (the client cert + CA load at startup); the API says so instead of pretending a hot-swap that does not exist.
|
||||
- A `<name>` that is not a configured `HOST_TARGETS` name → `404 {"configured": false}` **without** issuing, dialing, or touching the store (invariant 3, the `/api/disks` gate shape, spy-pinned in `tests/test_api.py`).
|
||||
- Every `POST` is a gated write: session + `Origin`/`Sec-Fetch-Site` + the cookie's `X-CSRF-Token` (the `POST /api/ups/command` test idiom — a tokenless write records nothing on the wire, with the token it does).
|
||||
- invariant 6: no `meta` value (a key PEM) may appear in an `event` row, a `/api/health` field, or an error `detail`; the events carry the host name + the action in fixed text.
|
||||
2. `tests/test_api.py` — the signed-in harness (`tests/auth_client.py::sign_in`):
|
||||
- `GET /api/certs` with zero / one / three configured hosts: the key paths pinned (the `/api/disks` pin idiom — a panel that retires can never shrink the payload);
|
||||
- an unknown `<name>`: the `404` and the spies on `httpx.AsyncClient.get` and `collectors.fetch_snapshot` (the `tests/test_api.py` spy idiom) prove no network and no store write (a `meta` row count before/after);
|
||||
- `POST rotate` → `bundle_rev` +1, a `cert-rotated` event row, the `meta` value changed, and the old serial is not on the new bundle's denylist;
|
||||
- `POST revoke` → the status, the event, and (through task 03's path) the next `fetch_snapshot` for that host refuses without a dial (the spy);
|
||||
- `POST root/rotate` → a fresh root serial, every agent reissued, `restart_required: true` in the body;
|
||||
- `GET bundle` → the body byte-identical to the stored `meta` bundle, and a reissue changes the served bytes;
|
||||
- the unsigned gate: the route-walk test (existing, automatic) plus one explicit unsigned `303`/`401` on `GET /api/certs` and a `POST` refusal (the `tests/test_auth_api.py` idiom).
|
||||
- Coverage: **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] the routes are signed-in + CSRF-gated (the wire test); the unsigned shape pinned
|
||||
- [ ] an unknown name 404s with no dial and no store write (spy-pinned)
|
||||
- [ ] rotate / revoke / root-rotate each leave a fixed-text event row and no key in the `event`/`health` tables (a test greps the stored events for a PEM header and finds none)
|
||||
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||
@@ -0,0 +1,35 @@
|
||||
# Task 05 — The /certs admin page: every cert, its expiry, rotate, revoke, re-deliver
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter." (operator addition: an admin page to see generated certs and expirations, trigger rotation, and revoke certs)
|
||||
|
||||
## Objective
|
||||
|
||||
A signed-in operator sees every cert the dashboard has generated — the root CA, its own client, and one per host — with its expiry and status, can rotate / revoke / root-rotate through task 04's API, and can download the exact bundle an agent should hold. The page follows `settings.html`'s precedent exactly: a page and not a dashboard — in `PAGES` (so it is a route), not in `.tabs`, linked from `.masthead-right`, with no `aria-current` on the tabs and no status dots.
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/web/pages/certs.html` + `src/dashboards/web/static/page-certs.js` (the `page-settings.js` idiom: the same `<head>` + masthead, `Dash.fetchJSON`, errors printed through `textContent`, a `confirm()` for destructive actions — the typed `DANGER` word is reserved for power cuts):
|
||||
- the **Root CA** block: the CN, the serial, the expiry (a local timestamp **and** the time left, the settings session-display idiom), and the **Rotate root CA** button → `POST /api/certs/root/rotate` with a confirm dialog; the response's `restart_required` prints "restart the dashboard to activate the new root" (the API's own sentence, not an invented one).
|
||||
- the **Agents** table: one row per configured `HOST_TARGETS` host (the `GET /api/certs` `agents[]`): the name, the serial, the `status` chip (`active` → `ok`, `revoked` → `fail` — a string-to-class map, the page never compares a number), the expiry (local timestamp + time left), the **issued** `bundle_rev` against the **deployed** `bundle_rev` (`n/a` when the agent never reported — a `null` verdict, uncoloured; issued > deployed → a neutral `stale` note, uncoloured — a count is not a graded number, the fleet-header rule), and the `san` list in mono.
|
||||
- per-row actions: **Rotate** (a confirm → `POST …/rotate`), **Revoke** (a confirm that says it stops the dashboard trusting that host until a reissue and a re-delivered bundle — a fixed sentence from the page, not the API), and **bundle** (a plain `<a href="/api/certs/<name>/bundle">` — a signed-in `GET` needs no CSRF).
|
||||
- a client-cert line (the serial + the expiry) — the one cert the agents check. It is **display-only**: no revoke button (a revoked client cert is a dashboard that reaches no agent — a footgun the page should not make easy); its rotation happens with the root.
|
||||
- ASSUMPTION: the client cert is display-only on the page (no revoke control); revoking it is deliberately not offered from the UI.
|
||||
2. `src/dashboards/app.py` — `PAGES` gains `"/certs": "certs.html"` (the settings-page comment at the `PAGES` dict extends by one line of the same shape); a `Certs` link (a `⚷` glyph is acceptable) in `.masthead-right` of every page including `/settings`, next to `⚙ Account` — **outside** `.tabs` (the settings precedent, `tests/test_api.py::test_every_dashboard_page_links_to_settings` pins that structure; extend the pin, do not break it).
|
||||
3. `src/dashboards/web/static/dash.css` — reuse existing vocabulary only (`.chip`, the table/`.hero`-family styles present in `settings.html`); **no** new `:root` tokens (the `tests/test_api.py` pin fails a CSS rule that reaches for an undeclared token).
|
||||
4. `tests/test_api.py` — the served-page pins (the settings-pin idiom):
|
||||
- the `Certs` masthead link on every dashboard page and on `/settings` (extend `test_every_dashboard_page_links_to_settings` into a two-link pin, or a sibling test);
|
||||
- the served `/certs` HTML: the root block, the agents table container, the three action labels; the page's own tab structure carries **no** `aria-current` on any of the five (the settings pin, copied);
|
||||
- the automatic audits cover the rest: every `id` the page keeps is referenced by `page-certs.js` (the orphan-id audit), no `Dash.severityFor` named (the existing pin), the deleted-flavour-text audit, the `no-store`/gate shape from the route walk.
|
||||
5. the live verification screenshot is task 07 (invariant 10).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- the served-HTML pins above; the JS is verified by the live screenshot (the repo's UI test policy — no JS test runner).
|
||||
- Coverage: **>90%** on the modified Python (`app.py`'s `PAGES`/route code).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] signed-in `GET /certs` serves the page; unsigned is a `303` to `/login?next=/certs` (the query included — `auth.safe_next`)
|
||||
- [ ] the masthead pin green (every page links to both settings and certs)
|
||||
- [ ] the orphan-id + flavour-text + `severityFor` audits green on the new page
|
||||
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 06 — The installer places the bundle; the unit runs TLS; --status probes over TLS
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
|
||||
## Objective
|
||||
|
||||
`deploy/install-agent.sh` can install the certificate bundle the same way it already installs the token — one file, the right owner and mode, the unit flag, a `--status` that proves the TLS path end-to-end — and `tests/test_deploy_scripts.py` moves its pins so a regression is a red test, not a dead agent.
|
||||
|
||||
## Work
|
||||
|
||||
1. `deploy/install-agent.sh` —
|
||||
- a new flag `--cert-bundle <file>` (the operator already fetched it from `GET /api/certs/<name>/bundle` — task 04's surface; the script does not dial the dashboard, per its existing no-dial rule): validate the bundle before touching anything (a `python3` one-liner: three PEM blocks — server cert, key, CA cert — and a `# BUNDLE-REV` line; a bad bundle refuses with the reason, the `resolve_bundle` idiom).
|
||||
- install to `/etc/dashboards-agent.certs` as `0600 root:$AGENT_USER` (the token-file idiom, with one line of the same comment explaining why the mode matters — the bundle holds the agent's key, so only root and the agent's own service user may read it); on re-run, replace it only if the flag is given (an existing bundle survives an unrelated re-run — the token-reuse idiom).
|
||||
- the unit: when a bundle is installed, `ExecStart` gains `--cert-bundle /etc/dashboards-agent.certs` (the `--token-file` line stays — task 02's token check is skipped in TLS mode, so the harmless dual mode eases a rollout) and `RuntimeDirectory=dashboards-agent` (the unit's `ProtectSystem=strict` leaves the agent no writable file but this — the extracted PEM blocks live there; `PrivateTmp` already exists, so a non-systemd dev run falls back to `tempfile`).
|
||||
- the `--status` probe: when the unit runs with a bundle, probe `https://127.0.0.1:$PORT/health` with a `python3 -c` `ssl` context that `load_verify_locations`-es the bundle's CA block and presents **no** client cert (task 02's `CERT_OPTIONAL` + the `/health` exemption is what makes this possible — the installer holds no client cert); a token-mode install keeps the plain `/health` probe. The probe is `python3` (the unit's own interpreter), never a new binary dependency (no `openssl` — one less thing to install on a host).
|
||||
2. `deploy/agent-sudoers` — **no** new line (the bundle travels through the same root installer invocation as the token file).
|
||||
3. `tests/test_deploy_scripts.py` — the new pins (the file's existing idiom: read the script, run it unprivileged so it exits before touching anything, and assert on the lines):
|
||||
- `--cert-bundle` in the usage + the `/etc/dashboards-agent.certs` path with the `0600` and `root:$AGENT_USER` `chown` (assert the `install -m`/`chown` line);
|
||||
- a `RuntimeDirectory=dashboards-agent` line in the unit when a bundle is installed (assert the conditional that emits it);
|
||||
- the `--status` probe branch: the `python3` `ssl` one-liner for TLS, the plain probe for token mode;
|
||||
- the existing pins unchanged (the sudoers-line identity, phase 10's no-tunnel name/port).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- the `tests/test_deploy_scripts.py` pins above; the real-host smoke is task 07 (invariant 10 keeps the dev server up).
|
||||
- Coverage: the script is shell — the pins are its tests (the file's stated policy: no shell test framework; a subprocess buys nothing more).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] an unprivileged `--cert-bundle /nonexistent` run refuses with the reason and touches nothing
|
||||
- [ ] the pins green: the bundle path/mode, the unit flag + `RuntimeDirectory`, the TLS `--status` probe
|
||||
- [ ] `uv run pytest` green; `uv run ruff check src tests agent` + `uv run pyright` clean
|
||||
@@ -0,0 +1,29 @@
|
||||
# Task 07 — The fleet runs TLS: install, flip, verify, document
|
||||
|
||||
**Phase:** `15_agent_mtls` · **Source:** `TODO.md:3` — "I want all exporters to be protected web servers. The collector should generate and manage a root certificate that needs to be installed with each exporter. Use the certs as auth to each exporter."
|
||||
|
||||
## Objective
|
||||
|
||||
The two live agents (deskwork, gateway) present the dashboard-issued server cert and require the dashboard's client cert; the dashboard dials `https://`; the health table, the live suite, and a `/certs` screenshot prove it — and the still-missing `driveripper` install is documented to carry a bundle from day one.
|
||||
|
||||
## Work
|
||||
|
||||
1. **Fleet (operator root actions, the `deploy/agent-sudoers` grant):** for `deskwork` and `gateway`: `GET /api/certs/<name>/bundle` (a signed-in `curl`), `scp` the bundle to the host, `sudo ./deploy/install-agent.sh --cert-bundle <file>`, `systemctl restart dashboards-agent`, `./deploy/install-agent.sh --status` green over TLS. Record the exact commands + output in `data/logs/NOTES-agent-install.md` (an append to the file's existing per-host record; local evidence, not tracked, the AGENTS.md rule).
|
||||
2. **The flip:** `.env.dev` — the `HOST_TARGETS` entries become `https://<same address>:9120` (no userinfo — the cert is the auth; the token goes away for a TLS host), `./scripts/dev-server.sh restart`, `./scripts/dev-server.sh status` shows `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`, `disks:gateway ok` (the two `driveripper` jobs stay `DOWN` — unchanged, the designed degradation).
|
||||
3. **Proof (invariant 10):** `uv run pytest -m live --live` — the host/disks tests over the `https` path; a `/certs` screenshot (the table showing both rows `active`, issued rev == deployed rev) **and** a `/system` screenshot (the host rows not dimmed) in `data/logs/`; a `data/logs/NOTES-verify.md`-style note: the `curl` codes (unsigned `/api/certs` refused, signed-in 200), one line for a wrong-CA client's handshake failure (the redacted URL only).
|
||||
4. **The `driveripper` procedure:** an append to `data/logs/NOTES-agent-install.md` — when the agent is finally installed (the outstanding operator action, the AGENTS.md fleet section), install the bundle in the same invocation (`sudo ./deploy/install-agent.sh --bind … --cert-bundle …`) and point `HOST_TARGETS` at `https://` from day one; a token-only install after this phase is a host the `https`-dialing dashboard cannot reach.
|
||||
5. **Docs (the AGENTS.md definition of done):** the AGENTS.md fleet section (the transport fact: the agent is now mTLS, the bundle file, the `/certs` page) + the commands section (the bundle-fetch line); a short "Protecting the agent" section in `README.md` (the operator's how-to: fetch the bundle, install, flip the URL — three sentences, no flavour).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- the live suite is read-only (the repo's rule) — no bundle mutations on the hardware; the revocation/rotation proof happens on a dev-box agent, not the fleet's.
|
||||
- invariant 10: the dev server stays up throughout; screenshots in `data/logs/`.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `./scripts/dev-server.sh status`: `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`, `disks:gateway ok` over the `https://` dials
|
||||
- [ ] `uv run pytest -m live --live` green for the host/disks paths
|
||||
- [ ] the `/certs` + `/system` screenshots in `data/logs/` show `active`, issued rev == deployed rev, the host rows not dimmed
|
||||
- [ ] the `data/logs/` note records the exact commands + the `curl` codes + the wrong-CA failure line
|
||||
- [ ] AGENTS.md + `README.md` updated (the definition of done)
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
All green. Final verification summary:
|
||||
|
||||
**Verified / implemented (phase 13 final pass)**
|
||||
- All 4 tasks already in `complete/`; this was the verification pass. Confirmed the CSS rhythm rules (`#strip`, `#banners:empty ~ #fleet`) are served and pinned, `data/logs/NOTES-decode-spike.md` names the mechanism, and the phase-07/04 surfaces still pass their pins.
|
||||
- **Found and fixed a residual defect in this phase's own code**: a stalled fetch stamps its body with the poll's *finish* time, so that sample anchors every window for the rest of the span; the jump gate "confirmed" a jump measured from that same anchor → live gufo rows 232.28 tok/s (one request at ~50) + a decaying 146/114/95 tail, repeated at 200.6. `rates.py::_publish` now requires an **independent anchor** (`_pending` stores the base sample's ts; the hold continues until it ages out). Pinned by `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` + `test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (both fail on task 03's gate).
|
||||
- Re-verified live after a dev-server restart: 10 min / 540 signed-in `/api/llm/snapshot` polls + read-only SQLite pass; **0 values > 300, 0 without a `Δ<window>` label**; 0 post-fix stored rows over the ceiling except the two pre-fix-shape rows above.
|
||||
|
||||
**Tests / lint (exact, all 100 % green)**
|
||||
- `bash .agents/validate.sh` → **EXIT=0**; `uv run pytest --cov --cov-fail-under=94` → **1195 passed, 11 skipped, coverage 98.70 %**; `ruff check src tests agent` → All checks passed; `pyright` → 0 errors; `node --check` clean; repo hygiene clean.
|
||||
- `uv run pytest -m live --live` → **11 passed / 0 failed**. `/api/health` → **all nine sources ok**.
|
||||
|
||||
**Completion criteria**
|
||||
- Fleet/strip + head/fleet gap with screenshots (`llm-fleet-spacing{,-top,-banner-case}.png`) — met (CSS unchanged this pass, pinned on served bytes).
|
||||
- Task-02 findings file names the confirmed mechanism — met.
|
||||
- `test_rates.py`/`test_llm_source.py` pin label-set, both window bounds and the always-honest `Δ` — met (+2 new pins).
|
||||
- Sustained watch under the recorded ceiling (300 tok/s) — met (worst 203.05, all labelled).
|
||||
- Full suite + validate green; no phase-07/04 behaviour change — met.
|
||||
|
||||
**Documentation**
|
||||
- README: `RATE_WINDOW_SECONDS` row now states the hold/confirm rule (only change needed; rest verified consistent).
|
||||
- AGENTS.md: `rates.py` bullet + gap 9 rewritten (residual closed *and* the remaining sub-8× skew measured, root-caused to gap 4, explicitly not fixed by inventing a new threshold).
|
||||
- **Fleet facts changed under this pass and were corrected**: operator added driveripper's bearer token → `host:driveripper`/`disks:driveripper` `ok`, live suite 9/2→11/0; but driveripper reads **identity only, zero SMART numbers** (root `smartctl` gives 55 °C/14 %; `/dev/nvme0`, `/dev/ng0n1`, `/dev/bsg/*` are root-only; its unit is `DevicePolicy=auto`, not the installer's `strict`). Evidence appended to `data/logs/NOTES-agent-install.md` + `NOTES-decode-stable.md`.
|
||||
|
||||
**Notable decisions / deviations**
|
||||
- Did **not** add a span-relative window floor: that would re-decide task 03's pinned `MIN_WINDOW_SECONDS` and its "in-bounds diluted row publishes" test. Recorded as the next phase's call, with the root cause left at gap 4 / `14_store_performance` (the ~10 s stall).
|
||||
- No live fleet change made (read-only probes + one dev-server restart). Fixtures deliberately not re-captured (no parse change, task 02 proved no shape drift). Dependency-currency advisory left untouched (no dependency moved in this phase): certifi/fastapi/starlette/websockets have newer releases.
|
||||
|
||||
**Next pending phase:** `.agents/phases/todo/14_store_performance` (then `15_agent_mtls`).
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 65%]
|
||||
....................sssssssssss......................................... [ 71%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 95%]
|
||||
...................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 348 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 73 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 188 2 99% 255-256
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2914 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.70%
|
||||
1195 passed, 11 skipped, 1 warning in 92.75s (0:01:32)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.18 wheel
|
||||
fastapi 0.143.5 0.149.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
Everything is green. Final state:
|
||||
|
||||
**Implemented**
|
||||
- `src/dashboards/web/static/dash.css` — added the two rhythm rules in the fleet-totals section: `#strip { margin-top: .85rem; }` (unconditional, so the strip keeps the gap when `#fleet` is `[hidden]`) and `#banners:empty ~ #fleet { margin-top: .85rem; }` (gap above the block only when the banner slot renders nothing), with a comment naming why each half is shaped that way. No markup/JS change; the sections were already siblings of `<main>`.
|
||||
- `tests/test_api.py::test_llm_section_stack_keeps_the_shared_vertical_rhythm` — pins both rules on the served stylesheet, pins that **exactly one** rule styles each of `#strip`/`#fleet` (an unguarded second one = the double gap), pins the `~`-sibling structure and the inline `.85rem` margins the rules are written around. Verified to fail both ways (rules deleted; extra `#fleet` rule added).
|
||||
- Live capture with **two engines answering**: `data/logs/llm-fleet-spacing.png` (full page), `-top.png`, `-banner-case.png`, and `data/logs/NOTES-llm-fleet-spacing.md` with the before/after table.
|
||||
|
||||
**Tests / lint (exact)**
|
||||
- `bash .agents/validate.sh` → **EXIT=0**: `uv run pytest --cov --cov-fail-under=94` → **1173 passed, 11 skipped, coverage 98.63 %** (baseline 1172; +1 new test, coverage unchanged); `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → 0 errors; `node --check web/static/*.js` → clean.
|
||||
- `uv run pytest -m live --live` → 8 passed / 3 failed — the three `driveripper` agent tests, i.e. the transport state AGENTS.md already records; unchanged by a CSS rule.
|
||||
- Chromium measurements (`.85rem` = 13.6px): `#fleet → #strip` **0 → 13.59px**; `#strip → panels` 13.59px; banner slot → fleet 13.59px with a banner up (`#fleet` computes `margin-top: 0px`, i.e. no double gap) and with `#fleet` hidden.
|
||||
|
||||
**Docs**: AGENTS.md (`/llm` UI-conventions paragraph: the shared `.85rem` rule, both rule shapes, the test pins) and README.md (`/llm` walkthrough clause).
|
||||
|
||||
**Decisions/deviations**
|
||||
- Measured honesty: only fleet→strip actually moves. The `:empty`-guarded rule changes no rendered distance today because the *empty* `#banners` slot carries its own inline `margin-bottom:.85rem` (the same 1rem + .85rem head spacing `/` and `/ups` already have); I kept the prescribed rule — it moves the gap from markup-that-renders-nothing onto the block and prevents the double gap — and recorded that measurement in the test docstring and the notes instead of claiming a second fix.
|
||||
- Left the `/`–`/ups` page-head rhythm and `/disks`'s analogous fleet→"Per host" adjacency alone (out of this task's scope); noted `/disks` here as the same pattern for a future pass.
|
||||
- `validate.sh` advisory (not a gate): certifi/fastapi/starlette have newer releases; this task moved no dependency.
|
||||
|
||||
**Next pending task**: `.agents/phases/todo/13_llm_page_and_decode_rate/02_pin_decode_spike_mechanism.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 66%]
|
||||
....................sssssssssss......................................... [ 72%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 97%]
|
||||
................................ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 348 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 1 97% 51
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 228
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 188 2 99% 255-256
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2843 39 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.63%
|
||||
1173 passed, 11 skipped, 1 warning in 93.51s (0:01:33)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.18 wheel
|
||||
fastapi 0.143.5 0.148.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
+15
@@ -0,0 +1,15 @@
|
||||
**Implemented**
|
||||
- `data/logs/NOTES-decode-spike.md` — mechanism **confirmed, not guessed**: `RateEstimator` has no lower bound on its window (`base = inside[0] if len(inside)>1 else samples[-2]`) **and** `poller.Job._loop`'s `max(0.2, interval-elapsed)` catch-up makes a 0.2 s window after any slow fetch, so a stalled fetch's carried tokens are diffed over 0.2 s. Candidates (a) label-set sum, (c) lineage jump and (d) engine gauge are all excluded by measurement; (a) stays a latent defect with its own red test.
|
||||
- Evidence: 277/277 stored gufo values >1000 tok/s are catch-up rows (every one with a >3 s fetch gap in the prior 15 s); no spike's numerator exceeds 1.12× what its elapsed span can produce; the 15.47 s incident totals 827 tokens = 53.5 tok/s, the engine's real rate; all sources gap at the same instant (process-wide stall, gap-4/phase-14 territory).
|
||||
- Live read-only probes (`data/logs/probe-decode-spike{,2}.py`, `/metrics`+`/slots` only): gufo counters unlabelled + monotonic (max +76 tokens/read); vLLM has one label set and **no** throughput gauge; probe 1 reproduced a 0.297 s window, probe 2 reproduced the spike live (2043.66 tok/s, `Δ0.4s`) with the same tail as the stored incident.
|
||||
- Fixture↔live shape: **no drift** (gufo 15/15, vLLM 128/128) → no re-capture needed for task 04.
|
||||
- 4 red tests: 2 in `tests/test_rates.py` (spike + the inflated tail), 2 in `tests/test_llm_source.py` (gufo incident through `parse_gufo_metrics`; vLLM joining label set).
|
||||
- Physical ceiling for task 04: spike-free max **119.9** (gufo) / **171.7** (vLLM) tok/s, engine lifetime means 30.0 / 2.8, native gauge 27.3–49.5, bandwidth bounds 256/640 GB/s → **>300 tok/s is arithmetic**, ≥1000 impossible.
|
||||
|
||||
**Tests/lint/coverage** — `uv run pytest --cov --cov-fail-under=94` → **1173 passed, 11 skipped, 4 xfailed, coverage 98.66%**; `uv run ruff check src tests agent` → All checks passed; `uv run pyright` → 0 errors; `bash .agents/validate.sh` → exit 0 (incl. `node --check`, hygiene). No deps moved.
|
||||
|
||||
**Docs** — AGENTS.md: new known gap 9 (confirmed mechanism, evidence file, the four xfails that XPASS when fixed) + the `rates.py` architecture line now says the window has no floor yet. README: no doc impact (nothing user-visible changed yet).
|
||||
|
||||
**Decisions/deviations** — (1) "Red tests" ship as `xfail(strict=True)` so the suite is 100% green now and fails loudly (XPASS) when task 03 fixes the behaviour — markers must be removed, assertions kept. (2) No live change of any kind was needed or made; the ~10–15 s stall's root cause is left to gap 4 / `14_store_performance`, with a pointer in the notes.
|
||||
|
||||
**Next pending task** — `.agents/phases/todo/13_llm_page_and_decode_rate/03_bounded_estimator_and_per_series_rates.md`.
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 42%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 54%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 66%]
|
||||
....................sssssssssss....................................x.... [ 72%]
|
||||
...........x............................................................ [ 78%]
|
||||
........................................................................ [ 84%]
|
||||
.................xx..................................................... [ 90%]
|
||||
........................................................................ [ 96%]
|
||||
.................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 348 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 105 5 95% 107, 144, 158, 161, 164
|
||||
src/dashboards/rates.py 37 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 150 1 99% 228
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 188 2 99% 255-256
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2843 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.66%
|
||||
1173 passed, 11 skipped, 4 xfailed, 1 warning in 95.47s (0:01:35)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.18 wheel
|
||||
fastapi 0.143.5 0.148.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
All green. Final state verified against the recorded fleet baseline (the 3 live failures are the documented `driveripper`-no-agent ConnectErrors, unchanged by this task).
|
||||
|
||||
## Report — task 03 `bounded_estimator_and_per_series_rates`
|
||||
|
||||
**Implemented**
|
||||
- `src/dashboards/rates.py`: effective window now bounded both sides — `MIN_WINDOW_SECONDS=0.5` floor (the poll loop's ~0.2 s catch-up answers `(None, dt)` — one `n/a` with its honest label — **and drops the stale base** so the inflated tail re-bases) and the documented `2 × span` ceiling enforced via `MAX_WINDOW_RATIO` on the window-empty fallback; `JUMP_CONFIRM_RATIO=8.0` one-poll hold (previous value + label repeat; jump publishes only when the next interval confirms, else dropped); reset and `forget()` clear the judgement state; `min_window=0.0` constructor escape for sub-cadence tests.
|
||||
- `src/dashboards/prom.py`: `values_by_series()` per-label-set accessor; `scalar` untouched.
|
||||
- `src/dashboards/sources/llm.py`: gufo + vLLM `measured()` now diff **each label set under its own key** (`rate_key()` = resolved name + sorted, quoted labels); cell = Σ per-series rates over the **max** contributing window; a baseline-less series contributes 0 (never its carried value); a refused window keeps `computed:Δ<series>/Δt` + `Δ<window>` beside `n/a`.
|
||||
- Removed all four task-02 `xfail(strict=True)` markers, assertions kept; added pins for floor, ceiling, hold/confirm/drop, per-key state, per-series joins, provenance shape, `values_by_series`.
|
||||
|
||||
**Tests / lint / coverage** — `uv run pytest`: **1193 passed, 11 skipped**; `bash .agents/validate.sh` exit 0 (coverage **98.69%** ≥ 94; rates.py 100%, llm.py 99%, prom.py 96% — misses pre-existing in `_quantile`); `ruff` + `pyright` clean; dev server restarted, `/api/health` as recorded (driveripper DOWN = missing agent), live `/api/llm/snapshot` reads e.g. `42.5 tok/s @ Δ9.9s computed:Δllamacpp:tokens_predicted_total/Δt`; `pytest -m live --live` = 8 passed / 3 failed, all three the documented `driveripper:9120` ConnectError.
|
||||
|
||||
**Docs** — `AGENTS.md` (invariant 2 keyed-by-name+label-set, `rates.py`/`prom.py`/`sources/llm.py` bullets, gap 9 rewritten as fixed) and `README.md` (`RATE_WINDOW_SECONDS` row: a stall answers one `n/a`).
|
||||
|
||||
**Decisions/deviations**
|
||||
- The task file's "accept fallback base ≤ 1.5 × span" contradicts its own pinned red test (asserts the diluted 7.47 row at a 15.25 s window, span 10). The test wins: ceiling enforced at the estimator's documented `2 × span`; the confirmed mechanism (NOTES candidate b, *lower*-bound form) is fixed by the floor + stale-base drop, as the findings file prescribes.
|
||||
- A refused window is treated as an answer (skips vLLM's weaker `running/tpot` estimate for that poll) so the stall is visible as one self-labelled `n/a`.
|
||||
- Two `test_collectors.py` real-time tests (fetches µs apart) now pass `min_window=0.0` — they pin window semantics, not the floor.
|
||||
|
||||
**Next pending task:** `13_llm_page_and_decode_rate/04_live_stable_verification.md` (sustained live watch ≤ 300 tok/s, screenshots).
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 65%]
|
||||
....................sssssssssss......................................... [ 71%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 95%]
|
||||
.................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 348 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 69 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 188 2 99% 255-256
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2910 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.69%
|
||||
1193 passed, 11 skipped, 1 warning in 97.58s (0:01:37)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.18 wheel
|
||||
fastapi 0.143.5 0.148.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
Everything is green. Final report:
|
||||
|
||||
**Implemented (task 04 — live proof; no production code needed changing)**
|
||||
- `data/logs/watch-decode-stable.py` — signed-in watcher: 11.0 min / 285 `/api/llm/snapshot` polls, logging `decode_tokens_per_second` value + `Δ<window>` + `source` per engine (no load generated).
|
||||
- `data/logs/analyze-watch.py` — judge: TSV + `mode=ro` SQLite over the identical window; ceilings, label check, jump/hold check, 1 m rollups → `VERDICT: PASS`.
|
||||
- `data/logs/NOTES-decode-stable.md` — the completion evidence (row-by-row holds/confirmations/refusals, fixtures decision, live tallies, screenshots, gap measurements).
|
||||
- Screenshots: `llm-decode-stable.png` (full page), `llm-decode-stable-top.png` (fleet block + **13.59 px** `.85rem` gaps + strip + `DECODE` heroes with `Δ8.2s`/`Δ8.0s`), `llm-decode-burst.png` (real two-slot burst: 62.1 tok/s `Δ8.5s`, fleet sum 102.4).
|
||||
|
||||
**Results** — gufo max **117.57**, vLLM max **98.00** tok/s (ceiling 300): 0 payload and 0 stored raw rows over it, 0 values printed without a `Δ` label; 3 floor refusals (`Δ0.387/0.399/0.363 s`) and 2 ≥8× jumps held-then-confirmed ran live; 1 m rollups top out at 51.9/34.1 (task 02's bad minute was 222.7). `bash .agents/validate.sh` → **rc 0** (1193 passed, 11 skipped; coverage 98.69 % ≥ 94; ruff "All checks passed!"; pyright 0 errors; `node --check` clean). `pytest -m live --live` → **9 passed / 2 failed**, both `driveripper` agent tests.
|
||||
|
||||
**Docs** — `AGENTS.md`: gap 9 rewritten as *fixed and verified live* (numbers, screenshots, fixtures-not-recaptured), stale `1172 passed`/`98.6 %` counts corrected, and the fleet facts updated for a change that happened mid-task (see below). README: no doc impact (task 03 already carried the operator-facing line).
|
||||
|
||||
**Decisions / deviations**
|
||||
- **Fixtures not re-captured**: task 02 proved no shape drift (gufo 15/15, vLLM 128/128); task 13 changed no parsing, so `capture-fixtures.sh` was deliberately not run.
|
||||
- **Fleet fact moved under the task**: the operator installed `driveripper`'s `dashboards-agent` at 13:27 (`active`, `/health` 200, `/snapshot` 401) — the two `driveripper` jobs are still DOWN but now on a **missing bearer token** (`HOST_TARGETS` has none for driveripper), which is why the live tally moved 8/3 → 9/2. I made **no live change**: reading/rotating that root-owned token is an operator action; AGENTS.md now records it and the one step that turns those rows green.
|
||||
- The ~10 s process-wide stall persists (38 catch-up polls in 11 min) — gap 4 / phase 14's problem; the estimator absorbs it, so no re-open of task 03.
|
||||
|
||||
**Next pending task:** none in phase 13 — `13_llm_page_and_decode_rate/04` was the last; next is `14_store_performance` (and `15_agent_mtls` now in `todo/`).
|
||||
+1
@@ -0,0 +1 @@
|
||||
ok
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
== uv run pytest --cov (hermetic: no network, live tests skip) ==
|
||||
........................................................................ [ 5%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 17%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 65%]
|
||||
....................sssssssssss......................................... [ 71%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 95%]
|
||||
.................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/dashboards/.venv/lib64/python3.14/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.14.4-final-0 ________________
|
||||
|
||||
Name Stmts Miss Cover Missing
|
||||
------------------------------------------------------------------
|
||||
src/dashboards/__init__.py 3 0 100%
|
||||
src/dashboards/__main__.py 14 1 93% 32
|
||||
src/dashboards/alerts.py 176 0 100%
|
||||
src/dashboards/api.py 348 0 100%
|
||||
src/dashboards/app.py 178 0 100%
|
||||
src/dashboards/auth.py 265 0 100%
|
||||
src/dashboards/collectors.py 155 5 97% 141-142, 146-147, 164
|
||||
src/dashboards/config.py 120 0 100%
|
||||
src/dashboards/debug.py 14 0 100%
|
||||
src/dashboards/poller.py 165 3 98% 192-194, 279
|
||||
src/dashboards/prom.py 112 5 96% 126, 163, 177, 180, 183
|
||||
src/dashboards/rates.py 69 0 100%
|
||||
src/dashboards/severity.py 49 0 100%
|
||||
src/dashboards/sources/__init__.py 0 0 100%
|
||||
src/dashboards/sources/disks.py 130 0 100%
|
||||
src/dashboards/sources/host.py 199 1 99% 147
|
||||
src/dashboards/sources/llm.py 178 1 99% 285
|
||||
src/dashboards/sources/nut.py 413 20 95% 153, 172, 186, 198, 204, 217, 225, 241, 319-320, 322, 579, 591, 626, 657, 661, 718-719, 724, 736
|
||||
src/dashboards/sources/nut_sim.py 134 0 100%
|
||||
src/dashboards/store.py 188 2 99% 255-256
|
||||
------------------------------------------------------------------
|
||||
TOTAL 2910 38 99%
|
||||
Required test coverage of 94% reached. Total coverage: 98.69%
|
||||
1193 passed, 11 skipped, 1 warning in 95.31s (0:01:35)
|
||||
== uv run ruff check src tests agent ==
|
||||
All checks passed!
|
||||
== uv run pyright ==
|
||||
0 errors, 0 warnings, 0 informations
|
||||
== node --check src/dashboards/web/static/*.js ==
|
||||
advisory: dependencies are out of date (AGENTS.md rule 8 — update on purpose):
|
||||
certifi 2026.7.22 2026.10.18 wheel
|
||||
fastapi 0.143.5 0.149.0 wheel
|
||||
starlette 1.8.0 1.9.0 wheel
|
||||
websockets 17.2 17.3 wheel
|
||||
@@ -56,7 +56,10 @@ page still renders.
|
||||
as `0`. Every derived value carries the `source` string it came from and `derived: true`
|
||||
so the UI can print *how* it was obtained (`computed:Δtokens/Δt`, `Δ10.0s`). Aggregating
|
||||
two different series into one rate is the failure mode this rule exists to catch — the
|
||||
rate estimator is keyed by the *resolved* upstream series name for exactly that reason.
|
||||
rate estimator is keyed by the *resolved* upstream series name **plus its label set**
|
||||
for exactly that reason (phase 13: a label set joining a summed counter mid-window
|
||||
contributes nothing until it has its own baseline, never its carried cumulative value),
|
||||
and the estimator refuses a window too short or too stale to carry the claim.
|
||||
3. **Unconfigured is not broken.** A source with no env config answers
|
||||
`404 {"configured": false}` **without dialling**, and the page says "not configured".
|
||||
A red panel for a server the operator never installed destroys the signal from the
|
||||
@@ -253,8 +256,13 @@ Responsibilities, one line each:
|
||||
counting it in `hosts_total` — the `/llm` fleet-block rule, so the page never does
|
||||
arithmetic and a dead agent never shrinks a fleet total.
|
||||
- `sources/llm.py` — gufo + vLLM field maps (`GUFO_SERIES`, `VLLM_SERIES`,
|
||||
`VLLM_HISTOGRAMS`), measured Δ rates, prefill speed as
|
||||
`mean(prompt tokens)/mean(prefill time)`, session list from gufo `/slots`.
|
||||
`VLLM_HISTOGRAMS`), measured Δ rates **diffed per label set** (`rate_key()` keys the
|
||||
estimator on the resolved name plus its labels; the cell is the Σ of the per-series
|
||||
rates over the widest window one of them actually used — phase 13, so a joining
|
||||
`{engine="1"}` series can never lend its carried counter to one interval's delta),
|
||||
prefill speed as `mean(prompt tokens)/mean(prefill time)`, session list from gufo
|
||||
`/slots`. A refused window (the estimator's floor) is still an answer: the cell keeps
|
||||
its `computed:Δ<series>/Δt` source and its honest `Δ<window>` beside a `null` value.
|
||||
- `sources/nut.py` — upsd client, `UpsReading`, `VAR_MAP`, `assess()` (explicit checks, not
|
||||
a mystery score), `SELF_TEST_OUTCOMES` + `self_test_outcome()` (the `Last self-test` verdict
|
||||
follows the *outcome* of the driver's `*.test.result` word, not its phrasing: pass words →
|
||||
@@ -287,7 +295,23 @@ Responsibilities, one line each:
|
||||
`from_env()` is the single place the `NUT_SIMULATE` truthy rule lives — `app.py` builds the
|
||||
monitor through it, so the assembly and the model cannot disagree about what is enabled.
|
||||
- `prom.py` — Prometheus text-exposition parser (in-house; only for engines we don't own).
|
||||
- `rates.py` — Δ/window estimator, keyed by resolved series name.
|
||||
`scalar(agg="sum")` stays right for **gauges**; `values_by_series()` is the per-label-set
|
||||
view **counter** deltas need, because a summed counter mixes whatever label sets exist now.
|
||||
- `rates.py` — Δ/window estimator, keyed by resolved series name + label set. Its effective
|
||||
window is `now - base[0]` **bounded on both sides** (phase 13): under
|
||||
`MIN_WINDOW_SECONDS` (the poll loop's 0.2 s catch-up gap after any slow fetch) it answers
|
||||
`(None, dt)` — one `n/a` with the window it refused — and drops that base so the polls
|
||||
*after* the stall cannot re-diff it; the window-empty fallback accepts a base no older
|
||||
than `MAX_WINDOW_RATIO × span` (the `2 × span` ceiling the docstring always claimed, now
|
||||
enforced). A rate ≥ 8× the previously published one is held for one
|
||||
poll (the previous value repeats, label unchanged) and publishes only if the next interval
|
||||
confirms it (`JUMP_CONFIRM_RATIO`) — and **confirmation means a different anchor**: a held
|
||||
jump remembers the sample its window was diffed from, and stays held while that same sample
|
||||
still anchors the window. A stalled fetch stamps its body with the time the poll *finished*,
|
||||
so that sample is the stall older than its own timestamp, and every later window off it is
|
||||
the same skewed delta re-diffed over a longer span — measured twice from one sample is one
|
||||
measurement, and the hold simply continues until the stale anchor ages out of the span.
|
||||
`min_window=0.0` exists for tests that scrape faster than any real loop.
|
||||
- `auth.py` — every authentication decision as a pure function, stdlib only (phase 11):
|
||||
`hash_password`/`verify_password`/`needs_rehash` over a self-describing
|
||||
`pbkdf2_sha256$<iterations>$<salt_hex>$<hash_hex>` verifier (so the cost can rise without a
|
||||
@@ -530,15 +554,26 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
**Fleet transport, re-verified from the laptop on 2026-10-19.** The port-selective inter-VLAN
|
||||
drop phase 10 recorded on the lab gateway `10.2.0.1` is **gone**: `gateway` (`10.1.1.27:9120`)
|
||||
and `deskwork` (`10.1.0.51:9120`) both take a plain TCP connect and answer `/snapshot`, and
|
||||
`driveripper`'s upsd answers `LIST UPS` on `10.1.0.126:3051` from this same laptop. Two hosts
|
||||
run the **installed** service — `systemctl is-active dashboards-agent` is `active` on both,
|
||||
with `/usr/local/lib/dashboards-agent/host_agent.py` and a `--token-file` unit bound to the
|
||||
address the dashboard dials. `driveripper` is the one host with no agent at all (unit
|
||||
`inactive`, no installed file, nothing listening on `:9120` → `ConnectionRefused`), and it
|
||||
renders as an `unreachable` stub row: the designed degradation, recorded rather than hidden.
|
||||
`/api/health` today reads `host:deskwork ok`, `host:gateway ok`, `disks:deskwork ok`,
|
||||
`disks:gateway ok`, `nut ok`, `llm:gufo ok`, `llm:vllm ok` with the two `driveripper` jobs
|
||||
`DOWN`, and the `live` suite is 8 passed / 3 failed with every failure that one transport.
|
||||
`driveripper`'s upsd answers `LIST UPS` on `10.1.0.126:3051` from this same laptop. All three
|
||||
hosts now run the **installed** service — `systemctl is-active dashboards-agent` is `active` on
|
||||
each, with `/usr/local/lib/dashboards-agent/host_agent.py` and a `--token-file` unit bound to
|
||||
the address the dashboard dials. **`driveripper` joined during phase 13 on 2026-10-19, in two
|
||||
operator steps.** It went from *no agent at all* (unit `inactive`, nothing listening →
|
||||
`ConnectionRefused`, an `unreachable` stub row) to **`active`** with
|
||||
`/usr/local/lib/dashboards-agent/host_agent.py` installed at 13:27, and then its bearer token
|
||||
went into `HOST_TARGETS`' URL userinfo — so `host:driveripper` and `disks:driveripper` are
|
||||
**`ok`** and the `live` suite is **11 passed / 0 failed** (it was 9/2 on the missing token that
|
||||
morning, and 8/3 on the ConnectError before that). The unit reads
|
||||
`--host 10.1.0.126 --port 9120 --token-file /etc/dashboards-agent.token` (`640
|
||||
root:dashboards-agent`, the mode `deploy/install-agent.sh` writes) with `User=dashboards-agent`,
|
||||
and `getent group disk` → `disk:x:6:dashboards-agent`.
|
||||
**Two things about that install are not what this repo writes, and both cost SMART data on
|
||||
`/disks` (see the SMART block below): `systemctl show -p DevicePolicy dashboards-agent` answers
|
||||
`auto` with no `DeviceAllow` lines** (the installer's `strict` + node allow-list is not what is
|
||||
running there), **and its `smartctl` cannot read drives as that user.**
|
||||
`/api/health` today reads **all nine sources `ok`**: `host:driveripper`, `host:deskwork`,
|
||||
`host:gateway`, `disks:driveripper`, `disks:deskwork`, `disks:gateway`, `nut`, `llm:gufo`,
|
||||
`llm:vllm`.
|
||||
Phase 10's proof that the old drop was not this repo's firewall stands as history in
|
||||
`data/logs/NOTES-verify.md` — do not re-add a tunnel on the strength of it, and re-run the
|
||||
probe rather than trusting either note.
|
||||
@@ -597,8 +632,13 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
`systemctl restart|reload nut-server` pair documented in `deploy/nut-sudoers`); `deskwork`
|
||||
grants an unrelated set (`btrfs scrub`, `snapper`, `setfacl /srv/*`, `osbuild-auto.sh`).
|
||||
Neither host grants `ops` the installer or `visudo` (re-checked 2026-10-19), so every install
|
||||
that has happened on those two went through root by the operator — including the one still
|
||||
outstanding, `driveripper`'s.
|
||||
that has happened on those two went through root by the operator — `driveripper`'s included,
|
||||
which is why its unit is not the one this repo writes (see the SMART block: `DevicePolicy=auto`,
|
||||
no `DeviceAllow`). Worth knowing before the next install: the allow-list `install-agent.sh`
|
||||
does write (`/dev/nvme*`, `/dev/sd*`, `/dev/vd*`, `/dev/sg*`) has **no `/dev/ng*n*` and no
|
||||
`/dev/bsg/*`**, the nodes smartctl 7.5 reaches for NVMe admin and SATA pass-through, so a
|
||||
`strict` unit there would deny exactly what a SMART read needs — whatever phase takes
|
||||
driveripper's SMART on has to settle the node list and the grant together.
|
||||
- **SMART: real captures, and one live host finally reads them.** `smartctl` 7.5 is installed on
|
||||
deskwork and driveripper. The `ops` account (which is who runs a hand-started dev agent) is in
|
||||
no `disk` group, driveripper grants it `(root) NOPASSWD: /usr/sbin/smartctl*` and deskwork
|
||||
@@ -607,12 +647,25 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
What changed by 2026-10-19 is that `deskwork`'s **installed** service runs as
|
||||
`dashboards-agent`, which is in the `disk` group (`getent group disk` → `disk:dashboards-agent`)
|
||||
on an agent installed after the SMART pass: `/disks` shows real numbers there —
|
||||
`smartctl_available: true`, `nvme0n1` healthy at 52 °C, 19 160 power-on hours (an odometer,
|
||||
`smartctl_available: true`, `nvme0n1` healthy at 55 °C (amber, past `disk.temp_c`'s 50 °C
|
||||
threshold), 19 166 power-on hours (an odometer,
|
||||
so that figure is a capture date, not a fact about the drive), 11 % of rated
|
||||
wear — so the wear bar and the alarm chips are drawn from live hardware, not only fixtures.
|
||||
`gateway` answers `smartctl_available: false` (no smartmontools in its VM: the *install it*
|
||||
half of the three-way flag), and `driveripper` answers nothing at all until its agent exists,
|
||||
which is where `smartctl_available: null` — "nobody told us either way" — still comes from.
|
||||
half of the three-way flag), and **`driveripper` is the fourth case the flag was built for:
|
||||
`smartctl_available: true` with almost no numbers.** Its agent answers (16 whole-disk rows,
|
||||
`nvme0n1`…`nvme7n1` + `sda`…`sdh`) and `smart` is non-null for the 8 NVMes, but every one of
|
||||
them carries only identity — `health`, `temp_c`, `power_on_hours`, `percentage_used` and the
|
||||
counters are all `null` — while the 8 spinning drives come back `smart: null` each with the
|
||||
agent's own line `smart: sda: Try an additional '-d ata' or '-d sat' argument.` Measured
|
||||
read-only on 2026-10-19: the same `smartctl -i -A -H /dev/nvme0n1` **as root** prints
|
||||
`PASSED`, 55 °C and 14 % used, so the drives answer and the *agent user* is what is locked out
|
||||
— `/dev/nvme0`, `/dev/ng0n1` and `/dev/bsg/*` (the nodes NVMe admin and SATA pass-through
|
||||
actually go through) are `crw------- root root`, and only `/dev/nvme0n1` / `/dev/sg0` carry the
|
||||
`disk` group the agent is in. So `/disks` shows driveripper's drive inventory with `n/a` values
|
||||
and a why-line, which is the designed degradation, and **lighting it up is a new phase, not a
|
||||
token**: a udev rule granting those nodes, and/or `host_agent.smart()` learning the
|
||||
`-d sat` / `/dev/ngXnY` forms smartctl needs when auto-detection fails.
|
||||
**Real data is in the repo now:** `tests/fixtures/smartctl_driveripper_<dev>.txt` is all 16
|
||||
drives (8 NVMe, 8 spinning) read through that sudo rule, and `tests/test_agent.py` parses
|
||||
every one — which is what found the parser's one real bug: this binary annotates the
|
||||
@@ -622,13 +675,15 @@ That scheme is the history API's key — treat it as a public interface.
|
||||
honest look at this fleet's drives: `nvme5n1` reports 13 media-and-data-integrity errors and
|
||||
`nvme1n1` is at 58 % of its rated wear, so the alarm chip and the amber wear bar on `/disks`
|
||||
are real features with real data behind them, not demo furniture.
|
||||
**Lighting up the rest of it is the same two operator actions that travel together:** install
|
||||
the agent on `driveripper`, and give its agent user device read access — the `disk` group
|
||||
**Lighting up the rest of it was the same two operator actions that travel together** — install
|
||||
the agent on `driveripper`, and give its agent user device read access: the `disk` group
|
||||
(`sudo usermod -aG disk <agent-user>`, which is what `deploy/install-agent.sh` does on install),
|
||||
*not* a sudo rule, because the agent must read the device itself — on an `agent/host_agent.py`
|
||||
from phase 09 or later (never redeploy without the grant: 16 permission-denied lines would age
|
||||
nothing but are pure noise, while with the grant they are 16 drives' worth of temperature,
|
||||
hours and sector counters). `gateway` needs smartmontools first and will still read no SMART
|
||||
hours and sector counters). **The agent and the `disk` group are now both in place on
|
||||
`driveripper`, and it still reads no SMART numbers** — the group only covers nodes that carry
|
||||
it (see above), so what is left is the node grant / `-d sat` work, not an install. `gateway` needs smartmontools first and will still read no SMART
|
||||
after that, being a VM with no physical disk behind `vda`. A host that cannot read its drives
|
||||
renders the designed degradation: one dimmed card per physical drive, every number `n/a`, and
|
||||
the one-line why.
|
||||
@@ -656,8 +711,8 @@ set -a; . ./.env.dev; set +a # dev dials the fleet the prod way, no
|
||||
./scripts/dev-server.sh status # per-source ok/DOWN — unauthenticated: /api/health
|
||||
./scripts/dev-server.sh logs
|
||||
|
||||
uv run pytest # hermetic: 1172 passed + 11 live-skipped, no network
|
||||
uv run pytest --cov # coverage gate (98.6%; floor 94%)
|
||||
uv run pytest # hermetic: 1195 passed + 11 live-skipped, no network
|
||||
uv run pytest --cov # coverage gate (98.7%; floor 94%)
|
||||
uv run pytest -m live --live # opt-in: real fleet / engines / upsd
|
||||
uv run ruff check src tests agent && uv run pyright
|
||||
./scripts/capture-fixtures.sh # re-record ground truth (agents + UPS + gufo /slots)
|
||||
@@ -1036,7 +1091,16 @@ carries **no severity colour anywhere**: `fleet_totals` has no `severity` to ren
|
||||
summed-KV bar is gone for that reason) and `severity.py` rates no throughput or queue depth,
|
||||
so the numbers stay ink and only the dot speaks reachability. `fleet.seams` — the API's own
|
||||
note on how the two engines count prompt tokens differently — is the prompt sum's tooltip
|
||||
rather than another line of ink. The
|
||||
rather than another line of ink. **The block, the strip and the panels keep the page's shared
|
||||
`.85rem` gap** (phase 13, `TODO.md:4`: fleet and strip rendered flush into one wall):
|
||||
`#strip { margin-top: .85rem }` is unconditional because below two engines `#fleet` is `[hidden]`
|
||||
and the strip *is* the first visible section, and `#banners:empty ~ #fleet { margin-top: .85rem }`
|
||||
gives the block its top gap only when the banner slot renders nothing, so a banner's own
|
||||
`margin-bottom:.85rem` never stacks a second gap under it. `tests/test_api.py` pins both rules on
|
||||
the served stylesheet, pins that **exactly one** rule styles each of the two ids (a second,
|
||||
unguarded one is the double gap), and pins that `#banners`/`#fleet`/`#strip` are still direct
|
||||
siblings of `<main>` — the `~` combinator resolves between siblings, so re-flowing that markup has
|
||||
to rewrite the selectors instead of losing the gap. The
|
||||
four trend charts stay but at height 110 *below* the panels, each with an `ariaLabel`. Panels
|
||||
sit at their own height (`#engines` carries `.items-start`): a two-hero gufo card next to a
|
||||
latency-row vLLM one otherwise grows a blank bottom, and a blank panel reads as "no data". An
|
||||
@@ -1216,6 +1280,72 @@ memory plot; the `cpu.usage_pct@coreN` series is still stored and still queryabl
|
||||
the operator's own machine is the threat model here, not a shared kiosk — and the reason a
|
||||
lost laptop is a password change and not a sign-out. A real revocation list would be a row per
|
||||
session with a prune rule, which is the thing this design traded away.
|
||||
9. **The `/llm` `decode tok/s` spike is fixed, and verified live** (phase 13, 2026-10-19;
|
||||
the diagnosis is in the git-ignored `data/logs/NOTES-decode-spike.md`, the live proof in
|
||||
`data/logs/NOTES-decode-stable.md`). **Task 04's sustained watch passed the ceiling test:**
|
||||
11 minutes / 285 signed-in `/api/llm/snapshot` polls against the live engines, worst value
|
||||
**117.57 tok/s** (gufo) and **98.00** (vLLM) — both in the payload and in the stored raw
|
||||
series, 2.6× under the 300 tok/s pass line, with **zero** values printed without their
|
||||
`Δ<window>` label. The stall is still there (38 catch-up gaps < 1.5 s in 11 minutes, a
|
||||
10.6 s gap between two samples) and the estimator now absorbs it: three refusals each
|
||||
labelled with the sub-0.5 s window it refused (`Δ0.387s`, `Δ0.399s`, `Δ0.363s`), two ≥ 8×
|
||||
steps held for one poll and published only when the next interval confirmed them, and — the
|
||||
point of the whole fix — **1 m rollups topping out at 51.9 / 34.1 tok/s** where task 02 found
|
||||
a 222.7 tok/s minute, so the trend chart's y-scale stops being set by arithmetic. Fixtures
|
||||
were **not** re-captured: task 02 compared the live exposure with `tests/fixtures/` series
|
||||
by series (gufo 15/15, vLLM 128/128, identical sets), task 13 changed no parsing, and the
|
||||
page still reads `15 series · 0 unparsed` / `128 series · 0 unparsed`. Screenshots:
|
||||
`data/logs/llm-decode-stable.png` (+ `-top`, the fleet-block gap and both `Δ` tags) and
|
||||
`data/logs/llm-decode-burst.png` (a real two-slot burst: 62.1 tok/s `Δ8.5s`, fleet sum 102.4).
|
||||
The confirmed mechanism was neither a counter bug nor a label-set sum: `poller.Job._loop`
|
||||
sleeps `max(0.2, interval - elapsed)`, so any fetch slower than `LLM_INTERVAL_SECONDS`
|
||||
is followed by a poll **0.2 s later**, and the estimator happily diffed a whole stall's
|
||||
tokens over that gap — the 2026-10-18 09:16 gufo row read **3284.59 tok/s for an engine
|
||||
doing 53**, with an inflated tail because the stale sample stayed `inside[0]`. Task 03
|
||||
made that impossible by construction (see the `rates.py` bullet above): the window floor
|
||||
turns the catch-up poll into one `n/a` **and drops the stale base** so the tail re-bases;
|
||||
the fallback's `2 × span` ceiling is enforced (the honest 15.25 s diluted row still
|
||||
publishes); jumps ≥ 8× hold one poll and publish only on confirmation; and
|
||||
`values_by_series()` + `rate_key()` diff every counter
|
||||
per label set, so the latent joining-series defect (candidate (a)) contributes 0 tokens
|
||||
until it has its own baseline. The four task-02 `xfail(strict=True)` tests are green with
|
||||
their markers removed and every assertion kept. Spike-free across two days of raw history
|
||||
the fleet's real maximum is **119.9 tok/s** (gufo / deskwork) and **171.7**
|
||||
(vLLM / driveripper) — anything above ~300 is arithmetic, not work, which was task 04's
|
||||
pass line and what the watch above stayed under. The `Δ<window>` label stayed honest throughout —
|
||||
both before and after the fix it names the window actually used; do not "fix" anything
|
||||
here by hiding or rewriting it. The ~10–15 s process-wide **stall** that triggers a
|
||||
refused poll is still gap 4's and `14_store_performance`'s problem; the estimator no longer
|
||||
turns a slow fetch into 0.2 s arithmetic, and one `n/a` per stall is the designed degradation
|
||||
(the residual the verification pass measured is the paragraph below).
|
||||
**The verification pass found one residual path and closed it** (same day, live rows in the
|
||||
store the fix was running): a stalled fetch stamps its body with the time the poll
|
||||
*finished*, so the sample it stores is the stall older than its own timestamp. The floor
|
||||
cannot see that (the window is honest, the value is old), the sample then anchors every
|
||||
window for the rest of the span, and the jump gate published the second one because it was
|
||||
measured from the same anchor — gufo 12:42:10 read **232.28 tok/s** with exactly one request
|
||||
running at ~50, followed by the tell-tale decaying tail 146.4 / 114.1 / 94.8 and the true
|
||||
47.5 (a 13:40:58 row repeats it at 200.6). `rates.py` now requires an **independent anchor**
|
||||
for a confirmation (`_pending` stores the base sample's timestamp and the hold continues
|
||||
while the window is still diffed from it), so a skewed sample is held until it ages out and
|
||||
the honest rate publishes — the reconstruction of both rows is pinned by
|
||||
`test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor` (estimator) and
|
||||
`test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine` (through
|
||||
`parse_gufo_metrics`), each of which fails on the pre-fix gate and prints the live shape.
|
||||
**That closes the gate's path, not the skew itself, and the verification pass measured the
|
||||
difference.** A slow reply's counter is true somewhere between the request and the reply and
|
||||
nothing in the payload says where, so when the resulting jump is *under* 8× the last published
|
||||
rate the gate never engages: the post-fix watch (10 min / 540 reads, `data/logs/
|
||||
llm-decode-verify-pass.tsv`) caught gufo at **203.05 tok/s labelled `Δ2.595s`** where two busy
|
||||
slots were really doing ~110, then 127.4 / 100.2 / 89.6 as the window grew back to the span and
|
||||
35.6 once the freed slot settled — every value labelled, **0 of 540 above the 300 pass line**,
|
||||
and no stored raw row over it since the restart either. A collapsed window is what makes a
|
||||
burst *look* like a rate, so the fix for this family is the ~10 s stall itself (gap 4 /
|
||||
`14_store_performance`), not a fresh threshold invented in the estimator: task 03 pinned
|
||||
`MIN_WINDOW_SECONDS` as an absolute floor and pinned that an in-bounds diluted row publishes
|
||||
(`test_a_fallback_inside_the_documented_ceiling_still_publishes`), so a span-relative floor
|
||||
would be a re-decision of that phase, not a bug fix. Do not "fix" the residual by hiding the
|
||||
short window or its label.
|
||||
|
||||
## Locked decisions (and why)
|
||||
|
||||
|
||||
@@ -329,7 +329,7 @@ documentation and is never loaded, and `scripts/dev-server.sh` exports `.env.dev
|
||||
| ------------------------------------- | -------------------------------------------------------------- |
|
||||
| `HOST_TARGETS` | `name=url` per host running the agent; a bearer token may ride in the URL userinfo (`http://TOKEN@host:9120`) |
|
||||
| `GUFO_BASE_URL` / `VLLM_BASE_URL` | LLM engines; unset means *not deployed here*, and its half of the page disappears |
|
||||
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured rate — tokens/sec, bytes/sec, jiffies→CPU % (default 10); each value carries the window it actually got, and `/system`'s footnote prints that, not this default |
|
||||
| `RATE_WINDOW_SECONDS` | the Δ window behind every measured rate — tokens/sec, bytes/sec, jiffies→CPU % (default 10); each value carries the window it actually got, and `/system`'s footnote prints that, not this default. A window the poll loop cannot honour — the ~0.2 s catch-up after a fetch slower than the interval, or a base older than 2× the window — answers one `n/a` instead of a spike, and a rate 8× above the last one is held until a window measured from a **different** sample confirms it (phase 13) |
|
||||
| `DASH_HOST` / `DASH_PORT` | bind address for the dashboard itself (default `127.0.0.1:8090`; the container sets `0.0.0.0`) |
|
||||
| `HOST_INTERVAL_SECONDS` / `LLM_INTERVAL_SECONDS` / `NUT_INTERVAL_SECONDS` | how often each source is polled (10 / 2 / 5 s) |
|
||||
| `DISKS_INTERVAL_SECONDS` | how often the disk-health source re-reads each agent for SMART (default 300 s — SMART is not a per-second signal). No targets of its own: the `disks:<host>` jobs exist exactly when `HOST_TARGETS` does |
|
||||
@@ -502,7 +502,8 @@ auth.py ── verifier + session key, loaded once at startup (no row and no see
|
||||
healthy — the dot is reachability, and `fleet.seams` (how the two engines count prompt
|
||||
tokens differently) is a tooltip on the prompt sum rather than another line of ink. Below
|
||||
two configured engines the block is hidden: a cross-engine sum of one engine duplicates that
|
||||
engine's panel. Below the fleet block sits the **stats strip**: one card per configured
|
||||
engine's panel. Below the fleet block — with the page's shared `.85rem` gap, as everywhere else
|
||||
in the section stack — sits the **stats strip**: one card per configured
|
||||
engine — reachability dot, name, age stamp, the **KV-cache bar** (the one rated cell: the
|
||||
fill and a breach's text take their colour from the payload's `severity` and nothing else),
|
||||
then `prompt tok/s` and `decode tok/s` with their provenance tag (`Δ10.0s`, scraped,
|
||||
|
||||
@@ -1,3 +1 @@
|
||||
# TODO
|
||||
|
||||
- [ ] For privileged actions (UPS commands) the exporter should be installed with password protection and that password must be used to register it with the collector. Effectively the exporters become protected web servers. Additionally, the collector should generate and manage a root certificate that needs to be installed with each exporter. In fact, just use the certs as auth rather than a password.
|
||||
|
||||
@@ -97,6 +97,25 @@ class Prom:
|
||||
return sum(finite) / len(finite)
|
||||
return sum(finite)
|
||||
|
||||
def values_by_series(self, names: Sequence[str],
|
||||
label_match: dict[str, str] | None = None) -> list[tuple[dict[str, str], float]]:
|
||||
"""Every label set of the resolved name as `[(labels, value), ...]`.
|
||||
|
||||
`scalar(agg="sum")` is right for a *gauge* — a sum over label sets is a real
|
||||
aggregate — but a **counter delta** may not mix series: a label set that joins
|
||||
mid-window lends its carried cumulative value to one interval's delta, and a
|
||||
counter never goes down, so the restart check never fires. Diffing each series
|
||||
individually (keyed by name + label set in `sources/llm.py`) is what keeps a
|
||||
joining series at zero until it has its own baseline.
|
||||
"""
|
||||
name = self.resolve(names)
|
||||
if name is None:
|
||||
return []
|
||||
if not label_match:
|
||||
return list(self.series[name])
|
||||
return [(labels, value) for labels, value in self.series[name]
|
||||
if all(labels.get(k) == x for k, x in label_match.items())]
|
||||
|
||||
def by_label(self, names: Sequence[str], label: str) -> dict[str, float]:
|
||||
name = self.resolve(names)
|
||||
if name is None:
|
||||
|
||||
+112
-10
@@ -6,24 +6,83 @@ because work arrives in bursts, so the rate is measured over a window (default 1
|
||||
the window that was actually used travels with the value — the UI prints `Δ10.0s` rather
|
||||
than implying a scraped gauge it never got.
|
||||
|
||||
Keyed by the *resolved* upstream series name (e.g. `llamacpp:tokens_predicted_total`, or
|
||||
`driveripper:cpu.total`), so a renamed counter starts a fresh window instead of diffing
|
||||
two different series into a fabricated rate, and two hosts can never read each other's
|
||||
samples.
|
||||
The effective window is bounded on **both** sides (phase 13, `data/logs/NOTES-decode-spike.md`):
|
||||
|
||||
* **floor** — `poller.Job._loop` sleeps `max(0.2, interval - elapsed)`, so any fetch
|
||||
slower than the interval is followed by a catch-up poll ~0.2 s later. Whatever the
|
||||
stalled fetch carried is then diffed over that gap: the tokens are real, the window is
|
||||
a lie (that was the 3284.59 tok/s row for a 53 tok/s engine). A window under
|
||||
`MIN_WINDOW_SECONDS` answers `(None, dt)` — one `n/a` that still labels the window it
|
||||
measured — and the base it was measured from is dropped, so the *next* poll re-bases
|
||||
instead of re-diffing the same stalled delta (the tail rows after a spike were as
|
||||
wrong as the spike, because the stale sample stayed `inside[0]`).
|
||||
* **ceiling** — the window-empty fallback (`base = samples[-2]`) accepts a base no older
|
||||
than `MAX_WINDOW_RATIO x span`; older than that, the window is a different claim than
|
||||
the panel makes ("a 20-second window on a 10-second panel"), so this poll answers
|
||||
`(None, None)` and the next one recovers from the fresh sample.
|
||||
|
||||
And a jump needs two polls to publish: a rate ≥ `JUMP_CONFIRM_RATIO` x the previously
|
||||
published rate (previous > 0) is held for one poll — the previous value repeats, its
|
||||
`Δ<window>` label unchanged — and publishes only if the next interval confirms it. A
|
||||
genuine step-up outlives a poll interval and appears one poll late; a single-interval
|
||||
artifact (a carried counter, a mis-timed stall) never appears at all.
|
||||
|
||||
*Confirmation means a second, different measurement*, so the held jump also remembers the
|
||||
**base sample** it was measured from and is confirmed only by a window anchored on another
|
||||
one. A stalled fetch stamps its body with the time the poll *finished*, so that body is
|
||||
older than its own timestamp by the length of the stall; the sample that carries it then
|
||||
anchors every window for the rest of the span, and each poll re-diffs the same skewed delta
|
||||
over a longer window (a decaying tail). Measured twice against the same anchor is one
|
||||
measurement, not a confirmation: the hold simply continues until the anchor ages out.
|
||||
|
||||
Keyed by the *resolved* upstream series name plus its label set (e.g.
|
||||
`llamacpp:tokens_predicted_total`, or `vllm:generation_tokens_total{engine="0",...}` —
|
||||
`sources/llm.py::rate_key`), so a renamed counter — or a label set joining a summed
|
||||
counter mid-window — starts a fresh window instead of diffing two different series into
|
||||
a fabricated rate, and two hosts can never read each other's samples.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import math
|
||||
import time
|
||||
from collections import deque
|
||||
|
||||
#: A window shorter than this is the poll loop's catch-up gap, not a measurement over
|
||||
#: the panel's span: answer `n/a`, never a rate. The spike rows were 0.2-0.4 s windows
|
||||
#: on a 10 s span; every real cadence in this app polls at ≥ 1 s.
|
||||
MIN_WINDOW_SECONDS = 0.5
|
||||
|
||||
#: The window-empty fallback may reach back for a base no older than this many spans.
|
||||
#: This is the ceiling the estimator always documented (`2 x span`) made true: the trim
|
||||
#: keeps the last two samples however old, so without this check any length of outage
|
||||
#: between two samples became a rate.
|
||||
MAX_WINDOW_RATIO = 2.0
|
||||
|
||||
#: A rate this many times above the previously published one is held for one poll and
|
||||
#: publishes only when the next interval measures it again. A real decode step persists
|
||||
#: across a poll interval; a carried-counter artifact does not.
|
||||
JUMP_CONFIRM_RATIO = 8.0
|
||||
|
||||
|
||||
class RateEstimator:
|
||||
def __init__(self, span: float = 10.0, max_samples: int = 512) -> None:
|
||||
def __init__(self, span: float = 10.0, max_samples: int = 512,
|
||||
min_window: float | None = None) -> None:
|
||||
"""`min_window` exists for tests that scrape faster than any real poll loop
|
||||
(two fetches milliseconds apart have no rate to measure); every production
|
||||
estimator answers with the `MIN_WINDOW_SECONDS` floor."""
|
||||
self.span = span
|
||||
self.max_samples = max_samples
|
||||
self.min_window = MIN_WINDOW_SECONDS if min_window is None else min_window
|
||||
self._samples: dict[str, deque[tuple[float, float]]] = {}
|
||||
# key -> the last (rate, window) actually published, and a jump held pending
|
||||
# confirmation (keyed by the timestamp of the sample it was measured from, which is
|
||||
# the anchor the confirming window must move off). Both are judgement, not
|
||||
# measurement, and a counter restart or a `forget()` drops them together with the
|
||||
# window they were made of.
|
||||
self._published: dict[str, tuple[float, float]] = {}
|
||||
self._pending: dict[str, float] = {}
|
||||
|
||||
def rate(self, key: str, value: float | None, now: float | None = None) -> tuple[float | None, float | None]:
|
||||
"""(per-second, window actually used). (None, window|None) when not computable.
|
||||
@@ -37,6 +96,10 @@ class RateEstimator:
|
||||
samples = self._samples.setdefault(key, deque())
|
||||
if samples and value < samples[-1][1]:
|
||||
samples.clear()
|
||||
# A restart is a fresh counter with no history to be judged against: the
|
||||
# previous published rate must not veto or hold the new lineage's numbers.
|
||||
self._published.pop(key, None)
|
||||
self._pending.pop(key, None)
|
||||
samples.append((now, value))
|
||||
while len(samples) > self.max_samples:
|
||||
samples.popleft()
|
||||
@@ -44,20 +107,59 @@ class RateEstimator:
|
||||
return None, None
|
||||
|
||||
inside = [point for point in samples if now - point[0] <= self.span]
|
||||
base = inside[0] if len(inside) > 1 else samples[-2]
|
||||
if len(inside) > 1:
|
||||
base = inside[0]
|
||||
else:
|
||||
fallback = samples[-2]
|
||||
if now - fallback[0] > MAX_WINDOW_RATIO * self.span:
|
||||
return None, None # a stall, not a window: one n/a, next poll recovers
|
||||
base = fallback
|
||||
dt, dv = now - base[0], value - base[1]
|
||||
cutoff = now - 2 * self.span
|
||||
cutoff = now - MAX_WINDOW_RATIO * self.span
|
||||
while samples and samples[0][0] < cutoff and len(samples) > 2:
|
||||
samples.popleft()
|
||||
if dt <= 0 or dv < 0:
|
||||
return None, dt if dt > 0 else None
|
||||
return dv / dt, dt
|
||||
if dt <= 0:
|
||||
return None, None
|
||||
if dt < self.min_window:
|
||||
# The catch-up poll's gap: refuse to turn it into a rate, and drop the base
|
||||
# so the stall's tokens cannot be re-diffed by the polls after it either.
|
||||
with contextlib.suppress(ValueError): # base is always a point of this deque
|
||||
samples.remove(base)
|
||||
return None, dt
|
||||
# dv can never be negative here: a decrease clears the window before the append,
|
||||
# so every window this function diffs is non-decreasing by construction.
|
||||
return self._publish(key, dv / dt, dt, base_ts=base[0])
|
||||
|
||||
def _publish(self, key: str, rate: float, window: float, base_ts: float) -> tuple[float | None, float | None]:
|
||||
"""The confirmation gate: a jump ≥ ratio x previous publishes one poll late.
|
||||
|
||||
`base_ts` is the timestamp of the sample this window was diffed from, and a held
|
||||
jump stores it: a poll anchored on the same sample re-measures the same delta over
|
||||
a longer window, which confirms nothing.
|
||||
"""
|
||||
last = self._published.get(key)
|
||||
held_from = self._pending.get(key)
|
||||
if held_from is not None and last is not None:
|
||||
if held_from == base_ts:
|
||||
return last # same anchor, same measurement: the hold simply continues
|
||||
del self._pending[key]
|
||||
if rate >= JUMP_CONFIRM_RATIO * last[0]:
|
||||
self._published[key] = (rate, window) # the step is real: publish it
|
||||
return rate, window
|
||||
return last # unconfirmed artifact: the cell keeps printing the held value
|
||||
if last is not None and last[0] > 0 and rate >= JUMP_CONFIRM_RATIO * last[0]:
|
||||
self._pending[key] = base_ts # hold: the previous value prints once more
|
||||
return last
|
||||
self._published[key] = (rate, window)
|
||||
return rate, window
|
||||
|
||||
def forget(self, prefix: str) -> int:
|
||||
"""Drop every window whose key starts with `prefix` (a target that went away)."""
|
||||
keys = [k for k in self._samples if k.startswith(prefix)]
|
||||
for key in keys:
|
||||
del self._samples[key]
|
||||
self._published.pop(key, None)
|
||||
self._pending.pop(key, None)
|
||||
return len(keys)
|
||||
|
||||
def keys(self) -> list[str]:
|
||||
|
||||
@@ -138,6 +138,54 @@ def _histogram_mean(out: dict[str, Any], key: str) -> float | None:
|
||||
return mean if isinstance(mean, float) and math.isfinite(mean) else None
|
||||
|
||||
|
||||
def _quote_label(value: str) -> str:
|
||||
escaped = value.replace("\\", "\\\\").replace('"', '\\"')
|
||||
return f'"{escaped}"'
|
||||
|
||||
|
||||
def rate_key(name: str, labels: dict[str, str]) -> str:
|
||||
"""The estimator key of ONE labelled series: the resolved name plus its label set.
|
||||
|
||||
Keying a summed counter by the name alone lets a label set that joins between two
|
||||
polls (`{engine="1"}` on a DP rank, a reloaded model still exporting its old series)
|
||||
lend its carried cumulative value to one interval's delta — an *increase* never
|
||||
trips the restart check. Under its own key a joining series simply has no baseline
|
||||
and contributes 0 tokens, which is the "a renamed series starts a fresh window"
|
||||
doctrine extended from names to label sets (rule 2).
|
||||
"""
|
||||
if not labels:
|
||||
return name
|
||||
inner = ",".join(f'{k}={_quote_label(labels[k])}' for k in sorted(labels))
|
||||
return f"{name}{{{inner}}}"
|
||||
|
||||
|
||||
def _series_delta(parsed: Prom, rates: RateEstimator, name: str,
|
||||
now: float | None) -> tuple[float | None, float | None]:
|
||||
"""Σ of the per-label-set Δrates of one cumulative counter.
|
||||
|
||||
Returns (sum-of-rates, widest window a contributing series actually used). A series
|
||||
with no previous sample contributes nothing this interval — never its carried value.
|
||||
When no series contributed but one refused the window (phase 13's floor), the refused
|
||||
window travels back so the cell can print `n/a` beside its honest `Δ<window>` label.
|
||||
"""
|
||||
total = 0.0
|
||||
contributions = 0
|
||||
window: float | None = None
|
||||
refusal: float | None = None
|
||||
for labels, value in parsed.values_by_series([name]):
|
||||
rate, used = rates.rate(rate_key(name, labels), value, now)
|
||||
if rate is not None:
|
||||
contributions += 1
|
||||
total += rate
|
||||
if used is not None:
|
||||
window = used if window is None else max(window, used)
|
||||
elif used is not None:
|
||||
refusal = used if refusal is None else max(refusal, used)
|
||||
if contributions:
|
||||
return total, window
|
||||
return None, refusal
|
||||
|
||||
|
||||
def parse_gufo_metrics(text: str, rates: RateEstimator | None = None,
|
||||
now: float | None = None) -> dict[str, Any]:
|
||||
"""`rates` enables the Δ-based aggregate rates; omitted, the parse stays pure."""
|
||||
@@ -147,27 +195,30 @@ def parse_gufo_metrics(text: str, rates: RateEstimator | None = None,
|
||||
out[key] = field(parsed.scalar(names, agg), parsed.resolve(names))
|
||||
|
||||
def measured(total_key: str) -> dict[str, Any] | None:
|
||||
"""Δcounter/Δt for one cumulative counter, keyed by the RESOLVED series name so an
|
||||
alias rename starts a fresh window instead of diffing two different series."""
|
||||
"""Δcounter/Δt for one cumulative counter, per label set, keyed by the RESOLVED
|
||||
series name so an alias rename starts a fresh window instead of diffing two
|
||||
different series."""
|
||||
if rates is None:
|
||||
return None
|
||||
source = out[total_key]["source"]
|
||||
if source is None:
|
||||
return None
|
||||
rate, window = rates.rate(source, out[total_key]["value"], now)
|
||||
if rate is None:
|
||||
rate, window = _series_delta(parsed, rates, source, now)
|
||||
if rate is None and window is None:
|
||||
return None
|
||||
return field(rate, f"computed:Δ{source}/Δt", derived=True, window=window)
|
||||
|
||||
# The counters add every slot; gufo's single gauge cannot. Measured therefore wins for
|
||||
# both speeds here (the inverse of vLLM, where the native gauge is the aggregate) — and
|
||||
# when no window exists yet the cell stays n/a instead of borrowing one request's speed.
|
||||
# A *refused* window (value None, window present) is also an answer: the cell prints
|
||||
# n/a with the short Δ it refused, rather than silently waiting a poll to say anything.
|
||||
for ui_key, total_key in (
|
||||
("decode_tokens_per_second", "tokens_generated_total"),
|
||||
("prompt_tokens_per_second", "tokens_prompt_total"),
|
||||
):
|
||||
rate = measured(total_key)
|
||||
out[ui_key] = rate if rate is not None else field(None, None)
|
||||
cell = measured(total_key)
|
||||
out[ui_key] = cell if cell is not None else field(None, None)
|
||||
|
||||
prompt = out["tokens_prompt_total"]["value"]
|
||||
cached = out["tokens_prompt_cached_total"]["value"] or 0.0
|
||||
@@ -214,13 +265,19 @@ def parse_vllm_metrics(text: str, rates: RateEstimator | None = None,
|
||||
per_seq = (1.0 / tpot) if tpot else None
|
||||
out["per_request_decode_tokens_per_second"] = field(per_seq, "computed:1/tpot", derived=True)
|
||||
|
||||
def measured(key: str, total_key: str) -> dict[str, Any] | None:
|
||||
def measured(total_key: str) -> dict[str, Any] | None:
|
||||
"""Δcounter/Δt per label set, keyed by the resolved series name plus its labels —
|
||||
vLLM's counters are labelled (`{engine="0",model_name=...}`), so a summed diff
|
||||
over a changing label set is exactly the carried-counter bug phase 13 pinned."""
|
||||
if rates is None:
|
||||
return None
|
||||
rate, window = rates.rate(key, out[total_key]["value"], now)
|
||||
if rate is None:
|
||||
source = out[total_key]["source"]
|
||||
if source is None:
|
||||
return None
|
||||
return field(rate, f"computed:Δ{VLLM_SERIES[total_key][0][0]}/Δt", derived=True, window=window)
|
||||
rate, window = _series_delta(parsed, rates, source, now)
|
||||
if rate is None and window is None:
|
||||
return None
|
||||
return field(rate, f"computed:Δ{source}/Δt", derived=True, window=window)
|
||||
|
||||
decode = out["decode_tokens_per_second_native"]["value"]
|
||||
running = out["num_requests_running"]["value"]
|
||||
@@ -229,8 +286,10 @@ def parse_vllm_metrics(text: str, rates: RateEstimator | None = None,
|
||||
out["decode_tokens_per_second_native"]["source"])
|
||||
else:
|
||||
# measured beats the running/tpot estimate: it counts real output tokens, the
|
||||
# estimate assumes every running request is decoding this instant
|
||||
measured_decode = measured("vllm:generation_tokens_total", "generation_tokens_total")
|
||||
# estimate assumes every running request is decoding this instant. A refused
|
||||
# window counts as an answer too: papering over a stall with the lifetime
|
||||
# estimate would hide exactly the poll the operator needs to see as n/a.
|
||||
measured_decode = measured("generation_tokens_total")
|
||||
if measured_decode is not None:
|
||||
# `or` here would treat a genuine 0 tok/s as missing and print the lifetime
|
||||
# estimate for an engine that is provably idle
|
||||
@@ -249,7 +308,7 @@ def parse_vllm_metrics(text: str, rates: RateEstimator | None = None,
|
||||
"computed:mean(request_prompt_tokens)/mean(request_prefill_time)",
|
||||
derived=True)
|
||||
if out["prompt_tokens_per_second"]["value"] is None:
|
||||
measured_prefill = measured("vllm:prompt_tokens_total", "prompt_tokens_total")
|
||||
measured_prefill = measured("prompt_tokens_total")
|
||||
if measured_prefill is not None:
|
||||
out["prompt_tokens_per_second"] = measured_prefill
|
||||
elif prefill_avg is not None:
|
||||
|
||||
@@ -416,6 +416,22 @@ main { padding: 1.15rem 1.25rem 3rem; max-width: 1680px; margin: 0 auto; }
|
||||
}
|
||||
.fleet .metrics { margin-top: .55rem; }
|
||||
|
||||
/* The `/llm` section stack's `.85rem` rhythm (`TODO.md:4`: the fleet block and the engine
|
||||
* summaries rendered flush into one wall; the section under `#strip` already carried
|
||||
* `margin-top:.85rem` inline).
|
||||
*
|
||||
* `#strip`'s gap is unconditional: below two engines `#fleet` is `[hidden]` and the strip *is*
|
||||
* the first visible section, and a summary strip whose gap depended on a block above it
|
||||
* rendering is one layout change away from flush again.
|
||||
* The `:empty` guard is the no-double-gap half of the same idea — with a banner up, that
|
||||
* banner's own inline `margin-bottom:.85rem` **is** the gap above the block and this rule must
|
||||
* not stack a second one; with the slot empty the block owns its gap instead of borrowing the
|
||||
* empty slot's margin. `tests/test_api.py` pins both rules, and pins that nothing else styles
|
||||
* either id.
|
||||
*/
|
||||
#strip { margin-top: .85rem; }
|
||||
#banners:empty ~ #fleet { margin-top: .85rem; }
|
||||
|
||||
/* ---- drive panels -----------------------------------------------------------
|
||||
* `/disks`: one `.disk` mini-panel per **physical** drive inside its host's `.card`, in the
|
||||
* `.slot` shape (bordered mini-panel, `.chip` for the drive's own word) because a drive is an
|
||||
|
||||
@@ -2421,6 +2421,96 @@ def test_llm_fleet_block_colours_no_number(client) -> None:
|
||||
"the fleet block's head row lost its styling"
|
||||
|
||||
|
||||
#: Elements the browser treats as empty, so they never open a nesting level.
|
||||
_VOID_TAGS = {"meta", "link", "br", "img", "input", "hr", "source", "track", "wbr"}
|
||||
|
||||
|
||||
def _depth_at(html: str, needle: str) -> int:
|
||||
"""How deeply the `needle` sits inside `<main>` (1 = a direct child of it).
|
||||
|
||||
The `~` in `#banners:empty ~ #fleet` only resolves between **siblings**, so the gap rule
|
||||
depends on the markup as much as on the stylesheet. Counting tags is enough here and keeps
|
||||
the check on the served bytes: no HTML parser, no dependency, and comments are already
|
||||
gone by the time `_code` hands the page over.
|
||||
"""
|
||||
body = html[html.index("<main>"):html.index(needle, html.index("<main>"))]
|
||||
depth = 0
|
||||
for closing, tag, _attrs, self_closed in re.findall(
|
||||
r"<(/?)([a-zA-Z][\w-]*)([^>]*?)(/?)>", body):
|
||||
if self_closed or tag.lower() in _VOID_TAGS:
|
||||
continue
|
||||
depth += -1 if closing else 1
|
||||
return depth
|
||||
|
||||
|
||||
def test_llm_section_stack_keeps_the_shared_vertical_rhythm(client) -> None:
|
||||
"""Fleet block → engine summaries → engine panels sit the shared `.85rem` apart.
|
||||
|
||||
`TODO.md:4`: the fleet section had no spacing between it and the engine summaries. Every
|
||||
boundary below it was already handled inline — the section under `#strip` carries
|
||||
`margin-top:.85rem` — so `#fleet` and `#strip` skipped the rhythm and the top of the page
|
||||
read as one unbroken wall. Measured in Chromium at the 16px root (`data/logs/`
|
||||
`llm-fleet-spacing*.png`), before → after:
|
||||
|
||||
| gap | before | after |
|
||||
|-----|--------|-------|
|
||||
| `#fleet` → `#strip` | **0 (flush)** | 13.59px |
|
||||
| `#strip` → engine panels | 13.59px | 13.59px |
|
||||
| banner slot → `#fleet`, no banner | 13.59px | 13.59px |
|
||||
| banner slot → `#fleet`, one banner up | 13.59px | 13.59px (`margin-top: 0px`) |
|
||||
|
||||
One rendered distance moves, and it is the one the operator named. What the *shape* of the
|
||||
two rules buys is that no other state can put it back:
|
||||
|
||||
* `#strip`'s gap is **unconditional**, because below two engines `#fleet` is `[hidden]`
|
||||
(`display:none !important`, so it contributes nothing) and the strip is then the first
|
||||
visible section: the 13.59px below the banner slot in that state is now the strip's own
|
||||
margin rather than one borrowed from a slot above it, which is what `#fleet ~ #strip`
|
||||
would have been;
|
||||
* `#fleet`'s gap is guarded by `#banners:empty`, so the block owns its top gap exactly when
|
||||
nothing occupies the slot above it, and a banner's own `margin-bottom:.85rem` stays the
|
||||
single gap under it. An unguarded `#fleet { margin-top }` is a second rule for one gap:
|
||||
margin collapsing hides it at `.85rem`/`.85rem` today, and it doubles the moment either
|
||||
margin is changed. The empty slot keeps its inline bottom margin, so head → fleet stays
|
||||
the 1rem + .85rem the other two banner pages (`/`, `/ups`) already render.
|
||||
|
||||
Pinned on the served bytes because a stylesheet "cleanup" that drops either rule is
|
||||
invisible in a diff and only shows up as a page that reads worse.
|
||||
"""
|
||||
raw = _css(client)
|
||||
section = _css_section(raw, "fleet totals")
|
||||
assert "#strip { margin-top: .85rem; }" in section, \
|
||||
"the fleet block and the engine summaries render flush again (TODO.md:4)"
|
||||
assert "#banners:empty ~ #fleet { margin-top: .85rem; }" in section, (
|
||||
"the gap above the fleet block stopped belonging to the block: a banner-less page "
|
||||
"would then have it only as long as the empty banner slot keeps its inline margin")
|
||||
|
||||
# one rule per element: a second one (or an unguarded `#fleet`) is the double gap
|
||||
css = _code(raw)
|
||||
for element, expected in (("#strip", {"#strip"}),
|
||||
("#fleet", {"#banners:empty ~ #fleet"})):
|
||||
selectors = {s.strip() for s in re.findall(rf"([^{{}}]*{element}[^{{}}]*)\{{", css)}
|
||||
assert selectors == expected, (
|
||||
f"dash.css styles {element} with {sorted(selectors)} — exactly one rule, the "
|
||||
"guarded one, or the page grows a double gap or loses the single one")
|
||||
|
||||
html = _code(client.get("/llm").text)
|
||||
for element in ("banners", "fleet", "strip"):
|
||||
assert _depth_at(html, f'id="{element}"') == 1, (
|
||||
f"`#{element}` is no longer a direct child of <main>, so the `~` gap rule cannot "
|
||||
"match: rewrite the selectors for the new structure instead of dropping the gap")
|
||||
assert html.index('id="banners"') < html.index('id="fleet"') < html.index('id="strip"') \
|
||||
< html.index('id="engines"'), "the section stack was re-ordered"
|
||||
|
||||
# the inline half of the rhythm, which the two rules are written around
|
||||
assert 'style="margin-bottom:.85rem"' in html[:html.index('id="fleet"')], (
|
||||
"#banners lost the inline bottom margin that is the banner → fleet gap, and today the "
|
||||
"only thing spacing the page head from the block")
|
||||
after_strip = html[html.index('id="strip"'):]
|
||||
assert after_strip.count('style="margin-top:.85rem"') == 3, \
|
||||
"a section below the strip lost the inline gap the strip's rule hands off to"
|
||||
|
||||
|
||||
# ---- the `/llm` gufo slot grid (phase 07 task 04) -------------------------
|
||||
#
|
||||
# `llm_monitor`'s slot cards (`index.html` ~L1179), rebuilt against the capture instead of a
|
||||
|
||||
@@ -62,7 +62,10 @@ async def test_a_real_agent_snapshot_parses() -> None:
|
||||
return httpx.Response(200, json=body)
|
||||
|
||||
target = Target("driveripper", "http://agent.test")
|
||||
collector = HostCollector(RateEstimator())
|
||||
# min_window=0: these fetches land microseconds apart, far under any real poll
|
||||
# cadence; this test pins the Δjiffies window semantics, not the phase-13 floor
|
||||
# (tests/test_rates.py pins the floor at a real cadence).
|
||||
collector = HostCollector(RateEstimator(min_window=0.0))
|
||||
async with client_for(handler) as client:
|
||||
first = (await fetch_host_snapshot(client, target, collector)).payload
|
||||
# CPU usage is Δjiffies/Δt: one scrape has no window, so it is n/a and not 0 %
|
||||
@@ -392,7 +395,9 @@ async def test_second_gufo_poll_measures_throughput_over_a_window() -> None:
|
||||
return httpx.Response(200, json=[{"index": 0, "n_ctx": 8192, "predicted": 12}])
|
||||
return httpx.Response(404)
|
||||
|
||||
rates = RateEstimator(span=10.0)
|
||||
# min_window=0: the two fetches are microseconds apart (no real loop polls this
|
||||
# fast); this test pins that the measured cell divides by the window it labels.
|
||||
rates = RateEstimator(span=10.0, min_window=0.0)
|
||||
async with client_for(handler) as client:
|
||||
await fetch_engine(client, "gufo", "http://engine.test", "", rates)
|
||||
state["text"] = bump(GUFO, "llamacpp:tokens_predicted_total", 40.0)
|
||||
|
||||
@@ -15,6 +15,7 @@ from dashboards.sources.llm import (
|
||||
fleet_totals,
|
||||
parse_gufo_metrics,
|
||||
parse_vllm_metrics,
|
||||
rate_key,
|
||||
series_names,
|
||||
)
|
||||
from slots_contract import load_slots_capture
|
||||
@@ -94,6 +95,58 @@ def test_gufo_counter_reset_produces_n_a_not_a_spike(metrics_text) -> None:
|
||||
assert metrics.get("decode_tokens_per_second", {}).get("value") is None
|
||||
|
||||
|
||||
def test_gufo_a_stalled_poll_then_the_catch_up_is_not_a_spike(metrics_text) -> None:
|
||||
"""The exact stored incident, through the production parse path.
|
||||
|
||||
2026-10-18 09:16:31 on deskwork: a 15 s gap in every source, a first result carrying 114
|
||||
tokens (its body was generated ~2 s into the stall), then the poll loop's 0.2 s catch-up
|
||||
printing **3284.59 tok/s** for an engine doing 53. `test_rates.py` holds the estimator-level
|
||||
pair; this is the same defect where the operator sees it.
|
||||
"""
|
||||
text = metrics_text("gufo")
|
||||
rates = RateEstimator(span=10.0)
|
||||
parse_gufo_metrics(text, rates, 1_000.0)
|
||||
text = bump_metric(text, "llamacpp:tokens_predicted_total", 114.0)
|
||||
parse_gufo_metrics(text, rates, 1_015.25)
|
||||
text = bump_metric(text, "llamacpp:tokens_predicted_total", 713.0)
|
||||
metrics = parse_gufo_metrics(text, rates, 1_015.47)
|
||||
cell = metrics["decode_tokens_per_second"]
|
||||
assert cell["window"] == pytest.approx(0.22)
|
||||
assert cell["value"] is None or cell["value"] < 100.0, (
|
||||
f"53 tok/s of real work printed as {cell['value']:.0f} tok/s")
|
||||
|
||||
|
||||
def test_gufo_a_stale_scrape_body_never_becomes_a_rate_above_the_engine(metrics_text) -> None:
|
||||
"""The other half of the stall: the reply is stamped when the poll *finished*, not when the
|
||||
engine rendered it, so its counter is the stall's length older than its own timestamp.
|
||||
|
||||
That sample then anchors every window for the rest of the span and each poll re-diffs the
|
||||
same skewed delta over a longer one — the live 2026-10-19 12:42 incident printed 232 tok/s
|
||||
for one request running at ~50, then a decaying 146 / 114 / 95 tail. The estimator's jump
|
||||
gate holds the first of those and now refuses to call a window anchored on the same sample a
|
||||
confirmation, so nothing above the engine's real rate reaches the payload.
|
||||
"""
|
||||
text = metrics_text("gufo")
|
||||
rates = RateEstimator(span=10.0)
|
||||
parse_gufo_metrics(text, rates, 1_000.0)
|
||||
# (stamp, tokens the engine had really produced by then): the 1011 reply is the stalled one —
|
||||
# its body was rendered ~9 s before the poll stamped it — and from there the counter advances
|
||||
# at a steady 45 tok/s, so 45 is the only rate that is true to print.
|
||||
timeline = [(1_011.0, 90.0), (1_012.0, 540.0), (1_014.0, 630.0), (1_016.0, 720.0),
|
||||
(1_018.0, 810.0), (1_022.0, 990.0), (1_026.0, 1_170.0)]
|
||||
produced = 0.0
|
||||
cells = []
|
||||
for stamp, total in timeline:
|
||||
text = bump_metric(text, "llamacpp:tokens_predicted_total", total - produced)
|
||||
produced = total
|
||||
cells.append(parse_gufo_metrics(text, rates, stamp)["decode_tokens_per_second"])
|
||||
for cell in cells:
|
||||
assert cell["value"] is None or cell["value"] <= 100.0, \
|
||||
f"a 45 tok/s engine printed {cell['value']:.0f} tok/s ({cells})"
|
||||
assert cells[-1]["value"] == pytest.approx(45.0), "the stall ends in the truth, not a tail"
|
||||
assert cells[-1]["window"] == pytest.approx(10.0)
|
||||
|
||||
|
||||
def test_absent_gufo_metrics_are_none_with_no_source() -> None:
|
||||
metrics = parse_gufo_metrics("# nothing here\n")
|
||||
assert metrics["requests_processing"]["value"] is None
|
||||
@@ -191,6 +244,79 @@ def test_vllm_measured_rate_when_the_build_has_no_throughput_gauge(metrics_text)
|
||||
assert metrics["decode_tokens_per_second"]["value"] == pytest.approx(25.0)
|
||||
|
||||
|
||||
def test_vllm_a_joining_label_set_does_not_lend_its_carried_counter(metrics_text) -> None:
|
||||
"""A second `engine=`/`model_name=` series appearing mid-window must not be a rate.
|
||||
|
||||
vLLM's counters are labelled, so `scalar(agg="sum")` added whatever label sets exist right
|
||||
now. A rank joining (or a reloaded model whose old series is still exported) carries its own
|
||||
cumulative counter, and `value < samples[-1]` never fires for an *increase*, so the carried
|
||||
value was diffed. Confirmed as a defect, not as this fleet's trigger: `generation_tokens_total`
|
||||
has exactly one label set live (`{engine="0",model_name="juggernaut"}`). Fixed in task 03 by
|
||||
diffing each label set under its own estimator key — the joining series has no baseline and
|
||||
contributes 0 tokens until it has one.
|
||||
"""
|
||||
text = metrics_text("vllm")
|
||||
rates = RateEstimator(span=10.0)
|
||||
parse_vllm_metrics(text, rates, 2_000.0)
|
||||
gained = text + 'vllm:generation_tokens_total{engine="1",model_name="juggernaut"} 500000.0\n'
|
||||
metrics = parse_vllm_metrics(gained, rates, 2_010.0)
|
||||
cell = metrics["decode_tokens_per_second"]
|
||||
assert cell["value"] is None or cell["value"] < 100.0, (
|
||||
f"a joining label set printed {cell['value']:.0f} tok/s")
|
||||
|
||||
|
||||
def test_vllm_a_label_set_that_survives_starts_contributing_its_own_delta(metrics_text) -> None:
|
||||
"""Fresh series contribute nothing on their first poll and their real Δ on the second."""
|
||||
text = metrics_text("vllm")
|
||||
rates = RateEstimator(span=10.0)
|
||||
parse_vllm_metrics(text, rates, 2_000.0)
|
||||
gained = text + 'vllm:generation_tokens_total{engine="1",model_name="juggernaut"} 500000.0\n'
|
||||
metrics = parse_vllm_metrics(gained, rates, 2_010.0)
|
||||
assert metrics["decode_tokens_per_second"]["value"] == pytest.approx(0.0), \
|
||||
"the joining series is baseline-less: 500 000 carried, 0 tokens diffed"
|
||||
grown = bump_metric(gained, "vllm:generation_tokens_total", 100.0) # bumps every label set
|
||||
metrics = parse_vllm_metrics(grown, rates, 2_020.0)
|
||||
assert metrics["decode_tokens_per_second"]["value"] == pytest.approx(20.0), \
|
||||
"two series x 100 tokens / 10 s: the rate is the sum of the per-series rates"
|
||||
assert metrics["decode_tokens_per_second"]["window"] == pytest.approx(10.0)
|
||||
|
||||
|
||||
def test_gufo_measured_cell_keeps_the_provenance_shape(metrics_text) -> None:
|
||||
"""Rule 2 after the per-series rewrite: value, `computed:Δ<series>/Δt`, `Δ<window>`."""
|
||||
text = metrics_text("gufo")
|
||||
rates = RateEstimator(span=10)
|
||||
parse_gufo_metrics(text, rates, 1_000.0)
|
||||
text = bump_metric(text, "llamacpp:tokens_predicted_total", 400.0)
|
||||
decode = parse_gufo_metrics(text, rates, 1_010.0)["decode_tokens_per_second"]
|
||||
assert decode["value"] == pytest.approx(40.0)
|
||||
assert decode["derived"] is True
|
||||
assert decode["source"] == "computed:Δllamacpp:tokens_predicted_total/Δt"
|
||||
assert decode["window"] == pytest.approx(10.0), "the widest window a contributing used"
|
||||
|
||||
|
||||
def test_gufo_a_refused_window_is_n_a_and_names_the_window_it_refused(metrics_text) -> None:
|
||||
"""The stall refusal is an answer: value None, but the series and Δ still travel."""
|
||||
text = metrics_text("gufo")
|
||||
rates = RateEstimator(span=10)
|
||||
parse_gufo_metrics(text, rates, 1_000.0)
|
||||
text = bump_metric(text, "llamacpp:tokens_predicted_total", 50.0)
|
||||
parse_gufo_metrics(text, rates, 1_010.0)
|
||||
text = bump_metric(text, "llamacpp:tokens_predicted_total", 60.0)
|
||||
decode = parse_gufo_metrics(text, rates, 1_010.3)["decode_tokens_per_second"]
|
||||
assert decode["value"] is None, "0.3 s is the loop's catch-up gap, not a rate"
|
||||
assert decode["window"] == pytest.approx(0.3)
|
||||
assert decode["source"] == "computed:Δllamacpp:tokens_predicted_total/Δt"
|
||||
|
||||
|
||||
def test_rate_key_puts_the_label_set_in_the_estimator_key() -> None:
|
||||
"""The key is the resolved name plus its labels, label order included-stable."""
|
||||
assert rate_key("llamacpp:tokens_predicted_total", {}) == "llamacpp:tokens_predicted_total"
|
||||
assert rate_key("vllm:generation_tokens_total", {"model_name": "m", "engine": "0"}) == \
|
||||
'vllm:generation_tokens_total{engine="0",model_name="m"}'
|
||||
assert rate_key("m", {"weird": 'a"b\\c'}) == 'm{weird="a\\"b\\\\c"}', \
|
||||
"a quote in a label value must not splice two keys into one"
|
||||
|
||||
|
||||
def test_vllm_waiting_requests_are_broken_down_by_reason(metrics_text) -> None:
|
||||
metrics = parse_vllm_metrics(metrics_text("vllm"))
|
||||
reasons = metrics["waiting_by_reason"]["value"]
|
||||
|
||||
@@ -33,6 +33,23 @@ def test_labels_are_parsed_and_filterable(metrics_text) -> None:
|
||||
assert by_model and all(model for model in by_model)
|
||||
|
||||
|
||||
def test_values_by_series_keeps_the_label_sets_apart() -> None:
|
||||
"""Counter deltas diff per label set; the summed view would let a joining series
|
||||
lend its carried value to one interval (phase 13). This is the accessor that view."""
|
||||
prom = Prom('x{a="1"} 5.0\nx{a="2"} 7.0\ny 1.0\n')
|
||||
assert prom.values_by_series(["x"]) == [({"a": "1"}, 5.0), ({"a": "2"}, 7.0)]
|
||||
assert prom.values_by_series(["y"]) == [({}, 1.0)]
|
||||
assert prom.values_by_series(["x"], {"a": "2"}) == [({"a": "2"}, 7.0)]
|
||||
assert prom.values_by_series(["nope"]) == []
|
||||
|
||||
|
||||
def test_values_by_series_on_a_real_labelled_capture(metrics_text) -> None:
|
||||
prom = Prom(metrics_text("vllm"))
|
||||
series = prom.values_by_series(["vllm:generation_tokens_total"])
|
||||
assert series, "vLLM exposes generation_tokens_total"
|
||||
assert all(labels.get("model_name") for labels, _v in series), "the capture lost its labels"
|
||||
|
||||
|
||||
def test_aggregations(metrics_text) -> None:
|
||||
prom = Prom(metrics_text("vllm"))
|
||||
total = prom.scalar(["vllm:prompt_tokens_total"], agg="sum")
|
||||
|
||||
@@ -52,6 +52,203 @@ def test_nan_and_inf_are_treated_as_missing() -> None:
|
||||
assert r.rate("x", bad, now=1.0) == (None, None)
|
||||
|
||||
|
||||
# --- phase 13: the live decode-tok/s spike ---------------------------------- #
|
||||
# Confirmed against the stored history and reproduced on live hardware; the findings are in
|
||||
# data/logs/NOTES-decode-spike.md. These two tests were written RED (as strict xfails) by
|
||||
# task 02 and turned GREEN by task 03's bounded window; the markers were removed and every
|
||||
# assertion kept verbatim, as the phase mandated. They are permanent pins now.
|
||||
#
|
||||
# The incident, from `sample` (source=llm, host=gufo, metric=llm.decode_tokens_per_second,
|
||||
# res=raw) on 2026-10-18, with the counter value reconstructed from the rows' own windows:
|
||||
#
|
||||
# 09:16:15.839 53.64 tok/s window ~10 s (steady work on deskwork: 53 tok/s)
|
||||
# ... 15.25 s with no sample at all: every source in the process gaps at the same instant ...
|
||||
# 09:16:31.091 7.47 tok/s window 15.25 s (114 tokens — a body generated ~2 s into it)
|
||||
# 09:16:31.308 3284.59 tok/s window 0.217 s (713 tokens — the rest of the stall)
|
||||
# 09:16:33.311 371.23 tok/s window 2.22 s (the stale sample is still base[0])
|
||||
# 09:16:35.315 222.32, 09:16:37.318 166.23, 09:16:39.316 138.37, then back to ~50
|
||||
#
|
||||
# 827 tokens across the whole 15.47 s is 53.5 tok/s — the engine's true rate. The tokens are
|
||||
# real; the window is the lie. `probe-decode-spike2.py` produced the same row pair (and nearly
|
||||
# the same tail: 379 / 232 / 174 / 140) against live gufo with the production parser.
|
||||
|
||||
GUFO_TOKENS = "llamacpp:tokens_predicted_total"
|
||||
|
||||
|
||||
def stalled_estimator(span: float = 10.0) -> RateEstimator:
|
||||
"""A window fed the steady 2 s cadence at exactly 50 tok/s."""
|
||||
r = RateEstimator(span=span)
|
||||
for i in range(6):
|
||||
r.rate(GUFO_TOKENS, 100.0 * i, now=1_000.0 + 2.0 * i)
|
||||
return r
|
||||
|
||||
|
||||
def test_a_catch_up_poll_after_a_stall_answers_na_not_a_spike() -> None:
|
||||
"""A fetch that takes 15 s and the loop's 0.2 s catch-up floor must not make a rate.
|
||||
|
||||
`poller.Job._loop` sleeps `max(0.2, interval - elapsed)`, so any poll slower than the
|
||||
interval is followed by a second one ~0.2 s later. Whatever the stalled fetch carried is
|
||||
then diffed over that 0.2 s.
|
||||
"""
|
||||
r = stalled_estimator() # steady 50 tok/s, last sample (1010.0, 500.0)
|
||||
# the stalled fetch: 15.25 s after the last sample, but its body was generated ~2 s in,
|
||||
# so it carries 114 tokens and gets stamped 13 s later than the engine measured them
|
||||
diluted, _window = r.rate(GUFO_TOKENS, 614.0, now=1_025.25)
|
||||
assert diluted == pytest.approx(7.47, abs=0.01) # the diluted row the DB actually holds
|
||||
# the catch-up: 0.22 s later, the current counter (713 tokens of stall still unaccounted)
|
||||
rate, window = r.rate(GUFO_TOKENS, 1_327.0, now=1_025.47)
|
||||
assert window == pytest.approx(0.22)
|
||||
assert rate is None, "a 0.22 s window is not a rate over the span; it must be n/a"
|
||||
|
||||
|
||||
def test_a_stalled_sample_does_not_inflate_the_polls_after_it() -> None:
|
||||
"""The tail is as wrong as the spike: base[0] is the stale sample for the next ~span."""
|
||||
r = stalled_estimator()
|
||||
r.rate(GUFO_TOKENS, 614.0, now=1_025.25)
|
||||
r.rate(GUFO_TOKENS, 1_327.0, now=1_025.47)
|
||||
# back to a normal 2 s cadence and the same honest 50 tok/s
|
||||
tail = [r.rate(GUFO_TOKENS, 1_327.0 + 100.0 * n, now=1_029.47 + 2.0 * (n - 1))[0]
|
||||
for n in range(1, 5)]
|
||||
assert all(v is None or v <= 60.0 for v in tail), f"inflated tail, real rate is 50: {tail}"
|
||||
|
||||
|
||||
# --- phase 13 task 03: both window bounds and the confirmed jump ------------- #
|
||||
|
||||
def test_window_below_the_floor_answers_na_with_its_honest_label() -> None:
|
||||
"""A 0.4 s catch-up gap is not a measurement over a 10 s span; it labels itself."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 100.0, now=0.0)
|
||||
rate, window = r.rate("x", 103.0, now=0.4)
|
||||
assert rate is None
|
||||
assert window == pytest.approx(0.4), "the n/a still names the window it refused"
|
||||
# the refused base was dropped: the next normal poll measures from the catch-up sample
|
||||
rate, window = r.rate("x", 203.0, now=4.4)
|
||||
assert rate == pytest.approx(25.0)
|
||||
assert window == pytest.approx(4.0)
|
||||
|
||||
|
||||
def test_fallback_base_older_than_two_spans_answers_na_and_recovers() -> None:
|
||||
"""A 20-second window on a 10-second panel is a different claim — one n/a, then normal."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
assert r.rate("x", 50.0, now=30.0) == (None, None)
|
||||
rate, window = r.rate("x", 100.0, now=32.0)
|
||||
assert rate == pytest.approx(25.0)
|
||||
assert window == pytest.approx(2.0), "the next poll re-bases on the fresh sample"
|
||||
|
||||
|
||||
def test_a_fallback_inside_the_documented_ceiling_still_publishes() -> None:
|
||||
"""The stall row the DB actually holds (15.25 s ≤ 2 x 10 s) keeps its diluted rate."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
rate, window = r.rate("x", 100.0, now=15.25)
|
||||
assert rate == pytest.approx(100.0 / 15.25)
|
||||
assert window == pytest.approx(15.25)
|
||||
|
||||
|
||||
def test_two_samples_at_the_same_instant_are_not_a_rate() -> None:
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 100.0, now=5.0)
|
||||
assert r.rate("x", 200.0, now=5.0) == (None, None)
|
||||
|
||||
|
||||
def test_a_jump_is_held_for_one_poll_and_publishes_when_confirmed() -> None:
|
||||
"""A real step-up survives one more interval and then prints — one poll late.
|
||||
|
||||
The confirming window is anchored on the sample 10 s back, which has moved on from the one
|
||||
the held jump was measured from — that is what makes it a second measurement. The other
|
||||
half is `test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor`.
|
||||
"""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
r.rate("x", 100.0, now=10.0) # 10 tok/s published
|
||||
rate, window = r.rate("x", 900.0, now=20.0) # 80 tok/s = 8x: held
|
||||
assert rate == pytest.approx(10.0)
|
||||
assert window == pytest.approx(10.0), "the held value keeps the Δ label it was published with"
|
||||
rate, window = r.rate("x", 1_720.0, now=30.0) # 82 tok/s: the step is real — publish
|
||||
assert rate == pytest.approx(82.0)
|
||||
assert window == pytest.approx(10.0)
|
||||
|
||||
|
||||
def test_an_unconfirmed_jump_is_dropped_and_the_held_value_keeps_printing() -> None:
|
||||
"""A one-interval artifact never publishes; the cell kept printing through it anyway."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
r.rate("x", 100.0, now=10.0) # 10 tok/s
|
||||
assert r.rate("x", 900.0, now=20.0)[0] == pytest.approx(10.0) # 80: held
|
||||
assert r.rate("x", 1_005.0, now=30.0)[0] == pytest.approx(10.0) # 10.5: artifact dropped
|
||||
rate, _ = r.rate("x", 1_105.0, now=40.0) # ordinary publishing resumes
|
||||
assert rate == pytest.approx(10.0)
|
||||
|
||||
|
||||
def test_a_held_jump_is_not_confirmed_by_a_window_on_the_same_anchor() -> None:
|
||||
"""A stalled fetch stamps its body late, and that body then anchors the whole span.
|
||||
|
||||
Measured on live gufo at 2026-10-19 12:42 (`data/logs/NOTES-decode-stable.md`'s window,
|
||||
one request running at ~50 tok/s): the process stalled 11 s, the first reply back carried
|
||||
the counter gufo had rendered ~2 s into it, and the estimator diffed the rest of the stall
|
||||
over a 3 s window — 232 tok/s — then re-diffed **the same delta** over 5, 7 and 9 s
|
||||
(146, 114, 95) because the stale sample stayed `inside[0]`. Measured twice from one sample
|
||||
is one measurement, so the confirmation gate requires a window anchored on a different
|
||||
sample; the hold simply continues until the stale anchor ages out of the span.
|
||||
"""
|
||||
r = RateEstimator(span=10.0)
|
||||
tokens = lambda t: 45.0 * t # noqa: E731 the engine: a steady 45 tok/s
|
||||
polls = [(0.0, tokens(0.0)),
|
||||
(11.0, tokens(2.2)), # the stalled reply: honest body, 9 s-old stamp
|
||||
(12.0, tokens(12.0)), # the catch-up: ≥8x the diluted row, so it is held
|
||||
(14.0, tokens(14.0)), # …and every poll still anchored on that stale sample
|
||||
(16.0, tokens(16.0)),
|
||||
(18.0, tokens(18.0)),
|
||||
(20.0, tokens(20.0)),
|
||||
(22.0, tokens(22.0)), # the anchor has aged out; one poll still denies the jump
|
||||
(24.0, tokens(24.0)),
|
||||
(26.0, tokens(26.0))]
|
||||
published = [r.rate("g", value, now=1_000.0 + t)[0] for t, value in polls]
|
||||
assert all(v is None or v <= 60.0 for v in published), \
|
||||
f"a 45 tok/s engine printed {max(v for v in published if v is not None):.0f}: {published}"
|
||||
assert published[-1] == pytest.approx(45.0), "the stall ends in the truth, not a tail"
|
||||
|
||||
|
||||
def test_a_rise_from_zero_is_never_held() -> None:
|
||||
"""The first work an idle engine does must print on its own poll, not one late."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
r.rate("x", 0.0, now=10.0) # 0 tok/s published
|
||||
assert r.rate("x", 500.0, now=20.0)[0] == pytest.approx(50.0)
|
||||
|
||||
|
||||
def test_confirmation_state_is_per_key() -> None:
|
||||
"""One key's pending jump must not hold another key's numbers."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("busy", 0.0, now=0.0)
|
||||
r.rate("busy", 100.0, now=10.0) # 10 tok/s — a jump base exists
|
||||
r.rate("idle", 5.0, now=0.0)
|
||||
r.rate("idle", 5.0, now=10.0) # 0 tok/s — previous is 0, nothing to confirm against
|
||||
assert r.rate("busy", 900.0, now=20.0)[0] == pytest.approx(10.0)
|
||||
assert r.rate("idle", 505.0, now=20.0)[0] == pytest.approx(50.0)
|
||||
|
||||
|
||||
def test_a_counter_restart_clears_the_confirmation_state() -> None:
|
||||
"""A restarted counter has no previous lineage to be judged against."""
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("x", 0.0, now=0.0)
|
||||
r.rate("x", 100.0, now=10.0) # 10 tok/s
|
||||
assert r.rate("x", 10.0, now=20.0) == (None, None) # restart drops the window
|
||||
rate, _ = r.rate("x", 910.0, now=30.0) # 90 tok/s on the new counter
|
||||
assert rate == pytest.approx(90.0)
|
||||
|
||||
|
||||
def test_forget_drops_the_judgement_state_with_the_window() -> None:
|
||||
r = RateEstimator(span=10)
|
||||
r.rate("host:x", 0.0, now=0.0)
|
||||
r.rate("host:x", 100.0, now=10.0)
|
||||
r.rate("host:x", 900.0, now=20.0) # held: one pending jump, one published rate
|
||||
assert r._published and r._pending
|
||||
assert r.forget("host:") == 1
|
||||
assert r._published == {} and r._pending == {}
|
||||
|
||||
|
||||
def test_counter_reset_yields_none_once_then_recovers() -> None:
|
||||
"""A restart must not report a huge negative — or, worse, a huge positive — rate."""
|
||||
r = RateEstimator(span=60)
|
||||
|
||||
Reference in New Issue
Block a user