chore(agent): phase roadmap from TODO.md, 2 phases
Convert the gateway-visualisation TODO item into phases 28 (gateway_agent_proxy: agent sections for haproxy/nginx configs + live sockets, 1.2.0, capture, rollout) and 29 (gateway_dashboard_page: source, wiring, API, animated-pipe page, nav, docs). Clear TODO.md. LLM-Generated: true
This commit is contained in:
@@ -0,0 +1,39 @@
|
||||
# Phase 28 — The agent reads the gateway's proxy configs and its live sockets
|
||||
|
||||
**Source:** `TODO.md` L3 — "Create a 'gateway' page that visualizes my incoming connections based on the gateway host's nginx and haproxy configs. I want it to show animated pipes with data coming in (IPs, ports, domains, etc.) and data going out (IPs, ports, domains, etc.)." (the agent half)
|
||||
**Story:** n/a
|
||||
**Context:** `agent/host_agent.py` (the stdlib-only single-file agent, `Collector._safe` best-effort sections, `snapshot()`, the SCHEMA docstring, `AGENT_VERSION` 1.1.0, the `--no-smart` opt-out and the three-way availability precedent it sets), `tests/test_agent.py` (the in-process harness over a fake `root` tree, the real-`/proc` test, the smartctl-capture fixture pins), `scripts/capture-fixtures.sh` (the ssh-capture convention: plain command first, `sudo -n` fallback, refuse to overwrite good fixtures with unreadable output), `tests/fixtures/` (the verbatim-capture ground-truth directory), the gateway host's live state verified 2026-10-21 (read-only): `nginx` and `haproxy` both `active`, `/etc/haproxy/haproxy.cfg` (6 `frontend` blocks — `bind :::22 v4v6` et cetera — each with one `default_backend`; 6 `backend` blocks, each one `server <name> <domain>:<port> check inter 5s`), `/etc/nginx/nginx.conf` (a catch-all `listen 80` / `server_name _` block plus `include /etc/nginx/conf.d/*.conf;` and `include /etc/nginx/stream.d/*.conf;`), `/etc/nginx/conf.d/http.conf` (server blocks: `listen 4443 ssl proxy_protocol` / `listen 8443 ssl`, `server_name <domain>`, `location / { proxy_pass http://<domain>:<port>; }`), `/etc/nginx/stream.d/stream.conf` (a `map $ssl_preread_server_name $targetBackend { … }` block and `proxy_pass $targetBackend` with `ssl_preread on`), all three files world-readable and readable by the `dashboards-agent` user, `/proc/net/tcp` readable the same way. The dashboard side (source, wiring, page) is phase 29; this phase makes the data exist on the wire.
|
||||
|
||||
## Objective
|
||||
|
||||
`host_agent.py` 1.2.0 serves three new **additive** snapshot sections on the gateway host — `haproxy` (frontends + binds, backends + server targets, the frontend→backend routing), `nginx` (server blocks with `listen` + `server_name`, `upstream`, `proxy_pass` including the `$var`-to-`map` join, `map` blocks) and `sockets` (LISTEN + ESTABLISHED from `/proc/net/tcp{,6}`) — each best-effort (a missing config degrades its own section, never the snapshot), `--no-proxy` opting out silently, the real configs captured verbatim as fixtures, the rollout onto the gateway host done, and the sections proven live over mTLS. Nothing changes for token-mode or pre-1.2.0 consumers: an old agent's snapshot is byte-identical in every old field.
|
||||
|
||||
## Dependencies
|
||||
- `27_mobile_tables_and_audit` (todo) — not a code dependency (different files); ordered after it so the fleet rollout in task 05 does not race the mobile-audit's browser passes on the shared dev box
|
||||
- `15_agent_mtls` / `24_dev_mtls_rollout` (complete) — the installed-bundle install path (`deploy/install-agent.sh --cert-bundle`), the `--status` CA-only probe, and the gap-10 re-stage-first rule the rollout follows
|
||||
|
||||
## Tasks
|
||||
1. `01_haproxy_section.md` — the agent reads + parses `haproxy.cfg` into the additive `haproxy` section, three-way availability, `--no-proxy` opt-out, never fails the snapshot
|
||||
2. `02_nginx_section.md` — the agent reads the main config + one level of `include <glob>` (bounded) into the additive `nginx` section, with the `proxy_pass $var` → `map` join
|
||||
3. `03_socket_section.md` — the agent parses `/proc/net/tcp` + `/proc/net/tcp6` (bounded) into the additive `sockets` section: LISTEN + ESTABLISHED with local/remote ip:port
|
||||
4. `04_version_and_capture.md` — `AGENT_VERSION` 1.2.0, the SCHEMA docstring, `scripts/capture-gateway-config.sh` (real configs → `tests/fixtures/`), the live probe that the sections answer on the gateway host
|
||||
5. `05_rollout_gateway.md` — re-stage `/home/ops/agent-install/`, install 1.2.0 on the gateway host, restart the unit, verify the sections over mTLS — read-only live verification, transcript in `data/logs/`
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/test_agent.py` — every new section over a fake `root` tree (present config → parsed shape; missing file → `available: false` + exactly one errors line; unreadable → `null` + one errors line; `--no-proxy` → sections absent and no errors line; a corrupt config degrades its own section and the snapshot still succeeds with the other sections intact)
|
||||
- Fixture: `tests/fixtures/gateway_haproxy.cfg` + `gateway_nginx_*.conf` + `gateway_proc_net_tcp{,6}.txt` are verbatim captures from the gateway host (invariant 7); the parser tests run against them, so a config drift fails a test before a dashboard does
|
||||
- Coverage: **>90%** on the new/modified code (the three collectors + the parsers)
|
||||
- Live (read-only, invariant 10): task 04's probe over the mTLS dial, and task 05's post-restart verification with its transcript
|
||||
|
||||
## Documentation & Consistency
|
||||
|
||||
- The final pass (task 05) updates `AGENTS.md`'s agent bullet (the new sections, `AGENT_VERSION` 1.2.0, the `--no-proxy` line) and the `SCHEMA` docstring is the self-documenting half on the wire. `README.md` is checked for any agent-version sentence.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest` green (the hermetic suite, no network), coverage not down
|
||||
- [ ] a 1.2.0 agent over a captured-config root tree answers all three sections; over a tree without them it answers `available: false`/`null` with one errors line each and the snapshot's other sections are intact
|
||||
- [ ] `AGENT_VERSION` is 1.2.0 and the SCHEMA docstring names the three new sections
|
||||
- [ ] the fixture files are verbatim captures (the capture script's refusal-to-overwrite-with-unreadable pin holds)
|
||||
- [ ] the gateway host runs 1.2.0 (`systemctl is-active dashboards-agent` → `active`, the file's `AGENT_VERSION` reads 1.2.0) and the sections answer over mTLS — transcript in `data/logs/`
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
- [ ] no behavior change in completed work: a 1.2.0 agent's snapshot is identical to 1.1.0's in every pre-existing field for a non-gateway host (the new keys are the only difference, and `--no-proxy` removes even those)
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 01 — The agent reads and parses the haproxy config
|
||||
|
||||
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — "Create a 'gateway' page that visualizes my incoming connections based on the gateway host's nginx and haproxy configs…"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`host_agent.py` gains a `haproxy` snapshot section: the agent reads the haproxy config (path an argument, default `/etc/haproxy/haproxy.cfg`), parses its `frontend` blocks (`name`, each `bind` address:port, `default_backend`/`use_backend` references) and `backend` blocks (`name`, each `server` line's name + target host:port), and serves them as a new additive section. The section follows the smart precedent exactly: a missing file is `available: false` plus **one** errors line for the pass, a file that cannot be read or parsed is `available: null` plus one errors line, `--no-proxy` opts out silently, and no failure of this section ever fails the snapshot or `/health`.
|
||||
|
||||
- ASSUMPTION: the parsed shape is `haproxy: {available: true|false|null, config: str (the path), frontends: [{name, binds: [{addr, port}], default_backend: str|null}], backends: [{name, servers: [{name, target, port}]}]}` — the frontend→backend routing is a name the **dashboard** joins (sources/gateway.py, phase 29), the agent only reports what the config says.
|
||||
- ASSUMPTION: `bind` lines parse the last token's `addr:port` (the captured shape is `bind :::22 v4v6` — the address token is the one that contains `:port` after the word `bind`; a `bind` line without a parsable address:port is skipped with no error, because haproxy allows non-address bind forms and the section degrades to fewer binds rather than failing).
|
||||
- ASSUMPTION: `server` lines parse `server <name> <target>[:port] <rest>` — `check inter 5s` and the like are the ignored rest; a `target` without a port (a bare upstream name) is kept with `port: null` (absent is null, never 0 — the module's own rule).
|
||||
- ASSUMPTION: `use_backend <name>` lines inside a frontend are collected as `backends: [str]` on that frontend (the captured config uses `default_backend` only; the parser covers the captured shape plus `use_backend` because haproxy answers two names for the same routing fact, the way `resolve_command()` does for NUT verbs); unknown directives are ignored, never errors.
|
||||
|
||||
## Work
|
||||
|
||||
1. `agent/host_agent.py` — module docstring: add the three-way availability + `--no-proxy` rule to the design-rules section (the "SMART is a best-effort add-on" shape, adapted: one errors line for a missing binary becomes one errors line for a missing/unreadable config).
|
||||
2. `agent/host_agent.py` — `Collector.__init__`/arguments: a `haproxy_config` path argument (default `/etc/haproxy/haproxy.cfg`) and the `--no-proxy` flag on the CLI (the `--no-smart` precedent: a flag that removes the section entirely, no errors line), threaded into `snapshot()`.
|
||||
3. `agent/host_agent.py` — `Collector.haproxy()` + a module-level `parse_haproxy_config(text) -> dict` (the `parse_smart_output` idiom: a pure function over text so tests feed it fixtures without a filesystem): the block scanner (a `frontend`/`backend` line starts a block, an indented line belongs to it, a top-level line ends it) and the `bind`/`server`/`default_backend`/`use_backend` token parsers per the assumptions above.
|
||||
4. `agent/host_agent.py` — `snapshot()`: `self._safe(self.haproxy, "haproxy", errors)` added to the returned document (additive key; a 1.1.0 consumer sees no difference, a 1.2.0 consumer with `--no-proxy` sees no `haproxy` key at all — the same absence the smart section takes).
|
||||
5. `tests/test_agent.py` — the in-process harness cases: the captured-shape fixture parses to the exact expected frontends/backends (assert the `:::22` → `addr: "::"`, `port: 22` split and the six `default_backend` names); a `root` tree without the file → `available: false` + exactly one errors line naming the missing path (the smartctl "not found" pin's shape — the line must still say the file is missing, `tests/test_agent.py` pins that wording); an unreadable file (a mode-000 file in the fake tree, where the harness allows it) → `available: null` + one errors line; a corrupt file (garbage text) → `available: null` + one errors line and the snapshot's `cpu`/`memory` sections still intact; `--no-proxy` → no `haproxy` key and no errors line.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the five harness cases above + `parse_haproxy_config` over `tests/fixtures/gateway_haproxy.cfg` (task 04 records it; write the test against the captured shape now — the real file's content is known: 6 frontends, 6 backends, the exact `bind :::NNNN v4v6` and `server <name> <domain>:<port> check inter 5s` lines)
|
||||
- Coverage: **>90%** on `parse_haproxy_config` + `Collector.haproxy`
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/test_agent.py -q` green, including the new haproxy cases and every pre-existing agent test
|
||||
- [ ] a snapshot over a tree with the config carries the `haproxy` section with all six frontends and six backends; over a tree without it, `available: false` + one errors line; the snapshot never fails because of it
|
||||
- [ ] `--no-proxy` removes the section with no errors line (pinned)
|
||||
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: every pre-existing snapshot field byte-identical on a host without haproxy (the `haproxy` key is the only addition)
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 02 — The agent reads and parses the nginx configs
|
||||
|
||||
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — "Create a 'gateway' page that visualizes my incoming connections based on the gateway host's nginx and haproxy configs…"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`host_agent.py` gains an `nginx` snapshot section: the agent reads the main nginx config (path an argument, default `/etc/nginx/nginx.conf`) **and follows its `include <path-or-glob>` lines one level** (the captured main config includes `/etc/nginx/conf.d/*.conf` and `/etc/nginx/stream.d/*.conf`), parses `server` blocks (`listen` address:port pairs, `server_name`), `upstream` blocks, `proxy_pass` directives (a literal `scheme://host:port` **and** a `proxy_pass $var` whose target the agent resolves against the file's `map` blocks), and the `map` blocks themselves. Same three-way availability and best-effort rules as task 01: missing main file → `available: false` + one errors line; unreadable/unparseable → `null` + one line; `--no-proxy` removes the section; the snapshot never fails because of it.
|
||||
|
||||
- ASSUMPTION: the parsed shape is `nginx: {available: true|false|null, config: str, files: [str (main + the included paths actually read)], servers: [{listen: [{addr, port}], server_names: [str], proxy_passes: [{location: str, scheme: str|null, target: {host, port}|null, variable: str|null}]}], upstreams: [{name, servers: [{host, port}]}], maps: [{variable: str, entries: [{key, target: {host, port}|null}]}]}` — a `proxy_pass $targetBackend` whose `$targetBackend` matches a `map` block's variable is served with `variable` set and `target: null` (the map decides per-connection at runtime; the **dashboard** joins the variable to the map, the agent reports both facts and never merges them, because a merged target would be a number the wire did not publish).
|
||||
- ASSUMPTION: include-following is **one level, bounded** — `include` lines in the main config are read (glob expanded), but `include` lines inside an included file are not followed (the captured tree is one level deep; a deeper include is a config the operator changed, and the section degrades to `files` naming only what was read rather than erroring). Bounds: at most 8 included files and 1 MB of included text combined; exceeding a bound drops the excess includes with one errors line (the smart pass-budget shape: the pass reports what it did not cover) and the section stays `available: true`.
|
||||
- ASSUMPTION: `listen` parsing takes the first token after `listen` (the captured shapes: `listen 4443 ssl proxy_protocol;`, `listen [::]:4443 ssl;`, `listen 80;`) — `host:port`, `[host]:port`, or bare `port` (→ `addr: null`); the `ssl`/`proxy_protocol`/`http2` flags are not parsed (they change how the bytes travel, not what the page's pipes show).
|
||||
- ASSUMPTION: `server_name _` (the catch-all) is kept as the literal `_` entry — the dashboard renders it as "any domain", the agent does not interpret it.
|
||||
|
||||
## Work
|
||||
|
||||
1. `agent/host_agent.py` — `Collector.__init__`/arguments: a `nginx_config` path argument (default `/etc/nginx/nginx.conf`), threaded through like `haproxy_config`; `--no-proxy` covers this section too (it already exists from task 01).
|
||||
2. `agent/host_agent.py` — a module-level `parse_nginx_config(main_text, included: dict[str, str]) -> dict` (pure over text, the `parse_haproxy_config` idiom) with: a brace-block scanner (nginx directives nest by `{}`, unlike haproxy's indent — the scanner is the task's core); `server`/`upstream`/`map` block extraction; the `listen`/`server_name`/`proxy_pass`/`include`/`server`-inside-upstream token parsers per the assumptions above.
|
||||
3. `agent/host_agent.py` — `Collector.nginx()`: read the main config through the existing `read_text` (which returns `None` on unreadable — the section's `null` arm), expand each `include` line (a glob through `glob.glob`, the literal path otherwise), apply the 8-file/1 MB bounds, call the parser, and return the section dict with `files` naming the main + each included path actually read.
|
||||
4. `agent/host_agent.py` — `snapshot()`: `self._safe(self.nginx, "nginx", errors)` added to the returned document (additive, task 01's shape).
|
||||
5. `tests/test_agent.py` — the in-process harness cases: the captured main + `conf.d/http.conf` + `stream.d/stream.conf` (as a fake `root` tree under `etc/nginx/`) parse to the expected servers (assert the `4443`/`8443` listen ports, the `server_name` list, the `proxy_pass` targets including the `127.0.0.1:8081` literal), the `stream.conf` `map` block (assert the variable name `$targetBackend` and a representative entry), and the `proxy_pass $targetBackend` served with `variable: "targetBackend"` + `target: null`; a tree without the main file → `available: false` + one errors line; a corrupt main file → `null` + one line, snapshot intact; the include bound (a main config including 9+ files) → `available: true`, `files` the first 8, one errors line naming the dropped count.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the five harness cases above + `parse_nginx_config` unit cases for the token parsers (`listen [::]:4443` → `addr: "::"`, bare `80` → `addr: null`, a `proxy_pass http://host:port` with a scheme, an `upstream` block's `server` lines)
|
||||
- Coverage: **>90%** on `parse_nginx_config` + `Collector.nginx`
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/test_agent.py -q` green, including the new nginx cases and every pre-existing agent test
|
||||
- [ ] a snapshot over the captured-config tree carries the `nginx` section with every `server` block, the `map`, and the variable-form `proxy_pass` served unmerged
|
||||
- [ ] the one-level include bound is pinned (9th+ include dropped with one errors line, section still `available: true`)
|
||||
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: every pre-existing snapshot field byte-identical on a host without nginx
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 03 — The agent reads the live sockets
|
||||
|
||||
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — "I want it to show animated pipes with data coming in (IPs, ports, domains, etc.) and data going out (IPs, ports, domains, etc.)." (the live half — the configs say *where* the pipes go; this task says *what is flowing now*)
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`host_agent.py` gains a `sockets` snapshot section: the agent parses `/proc/net/tcp` and `/proc/net/tcp6` (both readable by the agent user, verified on the gateway host) into the LISTEN sockets (the gateway's open frontends, with their local address:port) and the ESTABLISHED sockets (local and remote address:port — the live end-to-end pairs the dashboard joins to the config). Same best-effort rules: an unreadable `/proc/net/tcp` degrades the section to `null` with one errors line, `--no-proxy` removes it, and the snapshot never fails because of it. The section is a raw fact list — **no** config join happens in the agent (a socket does not know its domain; the dashboard's `sources/gateway.py` does the join, phase 29).
|
||||
|
||||
- ASSUMPTION: the parsed shape is `sockets: {available: true|false|null, listens: [{family: "v4"|"v6", addr: str|null, port: int}], established: [{family, local: {addr, port}, remote: {addr, port}}]}` — `addr` is the dotted/colon form (v4) or the expanded `x:x:…` form (v6, the `/proc` 4-byte-group little-endian spelling unrolled); a `0.0.0.0`/`::` listener keeps its literal wildcard address (it is the fact the dashboard's inbound join keys on — the port, with the wildcard addr marking "any address").
|
||||
- ASSUMPTION: only the two states the page needs are reported — `0A` (LISTEN) into `listens`, `01` (ESTABLISHED) into `established` — every other state is skipped (a TIME_WAIT ocean is noise the pipes do not show, and skipping it keeps the section small).
|
||||
- ASSUMPTION: the read is bounded at 4096 lines per file (the header line excluded from the count); a file longer than the bound reports what it read and one errors line naming the truncation (the smart pass-budget shape), `available` stays `true`.
|
||||
- ASSUMPTION: the hex→address conversion is a pure module-level `parse_proc_net_tcp(text, family) -> (listens, established)` (the `parse_smart_output` idiom), so the captured-`/proc` fixtures test the parser without a live filesystem.
|
||||
|
||||
## Work
|
||||
|
||||
1. `agent/host_agent.py` — module docstring: the `sockets` design line (two states only, the 4096-line bound, the best-effort rule) beside the smart one.
|
||||
2. `agent/host_agent.py` — module-level `parse_proc_net_tcp(text: str, family: str) -> tuple[list[dict], list[dict]]`: the column split (the `/proc/net/tcp` columns: `sl local_address rem_address st …`; `local_address`/`rem_address` are `HEXADDR:HEXPORT`, v4 a 32-char hex little-endian word, v6 four 32-char groups in the same order), the `st` filter (`01`/`0A` only), and the address unrolling per the assumptions above.
|
||||
3. `agent/host_agent.py` — `Collector.sockets()`: read `/proc/net/tcp` and `/proc/net/tcp6` through `read_lines` (an unreadable file is `None` → the `null` arm; both unreadable → `null` + one combined errors line), apply the line bound, call the parser, and return the section dict.
|
||||
4. `agent/host_agent.py` — `snapshot()`: `self._safe(self.sockets, "sockets", errors)` added to the returned document (additive, tasks 01/02's shape).
|
||||
5. `tests/test_agent.py` — the parser unit cases over a captured `/proc/net/tcp` fragment (a LISTEN on `:::443` and one on `0.0.0.0:80`, an ESTABLISHED v4 pair, an ESTABLISHED v6 pair, a TIME_WAIT line asserted **absent** from both lists) + the harness cases: a fake tree whose `proc/net/tcp{,6}` carry the fragment → the section's exact lists; a tree without the files → `null` + one errors line, snapshot intact; the 4096-line bound (a 5000-line file) → `available: true`, the first 4095 data rows, one errors line naming the truncation.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the parser unit cases + the three harness cases above
|
||||
- Coverage: **>90%** on `parse_proc_net_tcp` + `Collector.sockets`
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/test_agent.py -q` green, including the new socket cases and every pre-existing agent test
|
||||
- [ ] a snapshot over a tree with the captured `/proc` files carries the `sockets` section with the exact LISTEN/ESTABLISHED lists; the TIME_WAIT line is absent (pinned)
|
||||
- [ ] the 4096-line bound is pinned (truncation line, section still `available: true`)
|
||||
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: every pre-existing snapshot field byte-identical on a host where `/proc/net/tcp` is unreadable (the section is `null`, nothing else moved)
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 04 — Agent 1.2.0, the SCHEMA, and the gateway-config captures
|
||||
|
||||
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — the ground-truth half: invariant 7 (verify against the deployed upstream; never guess a metric name) applied to the three new sections
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The agent's version and self-documentation catch up with tasks 01–03 (`AGENT_VERSION` 1.2.0, the SCHEMA docstring names the three sections), and the capture script records the gateway host's real `haproxy.cfg`, nginx configs, and `/proc/net/tcp{,6}` as verbatim fixtures — so the parser tests run against the deployed shape, not a memory of it, and a config drift fails a test before a dashboard does.
|
||||
|
||||
- ASSUMPTION: the fixtures **keep the real service domain names** (e.g. `nextcloud.reeseapps.com`) — unlike the smartctl capture's Serial/WWN scrub, the domains are the data the `/gateway` page exists to show (inbound domain, outbound target) and the page is session-gated; a scrubbed fixture would be a fixture that tests a config the gateway does not run. The capture script therefore scrubs nothing; what it refuses is the other direction (overwriting a good fixture with unreadable output, the `capture-fixtures.sh` rule).
|
||||
- ASSUMPTION: `AGENT_VERSION` goes 1.1.0 → 1.2.0 (a new additive section is a minor bump, the 1.0.0 → 1.1.0 TLS bump's shape); the bundle's `# BUNDLE-REV` is untouched (the agent file is not the bundle — no CA fact moved).
|
||||
|
||||
## Work
|
||||
|
||||
1. `agent/host_agent.py` — `AGENT_VERSION = "1.2.0"` and the SCHEMA docstring: the three new section shapes (the `haproxy`/`nginx`/`sockets` blocks, one line each, the way `smart` is documented) plus the three-way availability + `--no-proxy` note.
|
||||
2. `scripts/capture-gateway-config.sh` — a new capture script (the `capture-fixtures.sh` convention, ssh-alias driven: `GATEWAY_SSH` default `gateway-ops`, `ssh` alias `gateway-ops`): scp/capture `/etc/haproxy/haproxy.cfg` → `tests/fixtures/gateway_haproxy.cfg`; `/etc/nginx/nginx.conf`, `/etc/nginx/conf.d/*.conf`, `/etc/nginx/stream.d/*.conf` → `tests/fixtures/gateway_nginx_main.conf` / `gateway_nginx_conf_d_*.conf` / `gateway_nginx_stream_d_*.conf`; `cat /proc/net/tcp /proc/net/tcp6` → `tests/fixtures/gateway_proc_net_tcp.txt` / `gateway_proc_net_tcp6.txt`. The `capture-fixtures.sh` refusal rule: a capture that comes back empty or with an ssh/permission error is **not** written over an existing fixture (one line naming the failure, exit 1). No sudo (all five reads are world-readable by the agent user — verified 2026-10-21).
|
||||
3. Run the capture once (the operator's box reaches the gateway host) and commit the fixtures; `tests/test_agent.py`'s task 01–03 cases already assert against the captured shape.
|
||||
4. `tests/test_deploy_scripts.py` — pin the script the way `capture-fixtures.sh` is pinned: it dials the ssh alias (not a hardcoded IP), scrubs nothing (a domain grep over its own source asserts no scrub step exists), and the refusal-to-overwrite branch is named.
|
||||
5. Live probe (read-only, invariant 10): `ssh gateway-ops` — `systemctl is-active dashboards-agent`, then a `python3` one-liner (the `--status` probe idiom, CA-only over the installed bundle's CA) that dials the **current** 1.1.0 agent and confirms the old sections still answer (the no-behavior-change control), and confirms the new sections are **absent** from the 1.1.0 snapshot (the three-way `predates` arm's ground truth). The probe writes `data/logs/28-task04-probe.out`.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/test_deploy_scripts.py` pins for the capture script; the existing fixture-pinned parser tests (tasks 01–03) run against the real captures
|
||||
- Coverage: **>90%** on the new script's Python-side branches (the script is bash; its pins are the `test_deploy_scripts.py` shape)
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `AGENT_VERSION` reads `1.2.0` and the SCHEMA docstring names all three new sections (pinned by a `tests/test_agent.py` assert on the constant)
|
||||
- [ ] `bash scripts/capture-gateway-config.sh` (or the operator's run) leaves the seven fixture files in `tests/fixtures/`, verbatim, with the refusal rule holding
|
||||
- [ ] `tests/test_agent.py` green against the real captures (a config drift in the fixture fails a test)
|
||||
- [ ] `data/logs/28-task04-probe.out` records: the gateway's installed agent is 1.1.0, the old sections answer, the three new sections are absent
|
||||
- [ ] `uv run ruff check scripts` (if the script has a Python leg) / `bash -n scripts/capture-gateway-config.sh` clean, `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: the 1.1.0 agent's served snapshot is unchanged (the probe proves it)
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 05 — Roll 1.2.0 onto the gateway host, verify over mTLS
|
||||
|
||||
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — the deployment half: the sections only exist on the wire once the gateway host runs 1.2.0 (the phase-24 rollout's per-host safety order, gap-10's re-stage-first rule)
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The gateway host runs `host_agent.py` 1.2.0 with the three new sections live over mTLS, installed by the same granted-path procedure the fleet uses — re-stage `/home/ops/agent-install/` first (the gap-10 rule: a path grant is only as current as the file it points at), run the installer, restart the unit, and verify the sections answer. The install is **read-only on the dashboard's side** (no config change, no `HOST_TARGETS` change — the gateway is already in `HOST_TARGETS` and dials fine on 1.1.0), and the verification transcript lands in `data/logs/`.
|
||||
|
||||
- ASSUMPTION: the gateway host's unit is **unchanged by this rollout** — `--no-proxy` is not passed (the sections are on by default), so the installer's re-run guard sees an identical unit render and stays silent (the phase-17 task-03 silent path); no `--force` needed.
|
||||
- ASSUMPTION: the restart is the `systemctl restart dashboards-agent` the grant's `systemctl restart *` line allows (the gateway's `ops` has `(ALL) NOPASSWD: ALL` — the fleet record's `deploy/agent-sudoers` shape).
|
||||
|
||||
## Work
|
||||
|
||||
1. Re-stage: `ssh gateway-ops` — copy the repo's `agent/host_agent.py` (1.2.0, md5 recorded) and `deploy/install-agent.sh` (the repo's current sha256, recorded) into `/home/ops/agent-install/` (the directory is `ops`-owned, no root needed — the phase-24 task-03 procedure), leaving any existing snapshot beside it as `<name>-<date>-backup` (a file sudo will not run).
|
||||
2. Install: run the granted path `sudo /home/ops/agent-install/deploy/install-agent.sh --bind 10.1.1.27 --port 9120` (the gateway's existing unit arguments — read them first with `systemctl cat dashboards-agent` and reproduce them exactly, plus `--cert-bundle` if the unit has the line) — the re-run guard prints nothing (identical unit) and the agent file is updated in place.
|
||||
3. Restart: `sudo systemctl restart dashboards-agent`; `systemctl is-active dashboards-agent` → `active`.
|
||||
4. Verify (read-only): the `--status` CA-only probe (the `python3` `ssl` one-liner over the installed bundle's CA, `--bind 10.1.1.27 --port 9120`) answers; then a `python3` one-liner that dials over mTLS (the dashboard's client cert, the stored CA — the `tests/test_live_fleet.py` dial idiom) and asserts: `agent_version` is `1.2.0`, the three new keys are present, `haproxy.available` is `true` with 6 frontends, `nginx.available` is `true` with ≥ 9 server blocks and the `$targetBackend` map, `sockets.available` is `true` with a non-empty `listens` (port 443 among them), and the pre-existing sections (`cpu`, `memory`, `network`) still answer.
|
||||
5. `data/logs/28-task05-rollout.md` — the transcript: the md5/sha256 of what was staged where, the `systemctl cat` before/after (diff empty), the probe output, the one-liner's assertions, and the `haproxy`/`nginx`/`sockets` section summaries (counts only, no config text — invariant 6: no config body in the note).
|
||||
|
||||
## Testing & Quality
|
||||
- Live (read-only, invariant 10): the verification one-liner is the phase's proof; no dashboard process is touched, no `HOST_TARGETS` line moves
|
||||
- The hermetic suite is untouched by this task (no code change) — `uv run pytest` is re-run as the gate, not for this task's logic
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the gateway host's installed `host_agent.py` is the repo's 1.2.0 (md5 matches) and `systemctl is-active dashboards-agent` → `active`
|
||||
- [ ] the `--status` CA-only probe answers; the mTLS one-liner asserts all five facts (version, three sections present + available, pre-existing sections intact)
|
||||
- [ ] `data/logs/28-task05-rollout.md` exists with the transcript and no config text
|
||||
- [ ] the dashboard's `host:gateway` / `disks:gateway` jobs still read `ok` in `./scripts/dev-server.sh status` (the 1.2.0 snapshot's new keys cost the old sources nothing)
|
||||
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||
- [ ] no behavior change in completed work: the gateway's pre-existing snapshot fields are unchanged (the one-liner's control assertions prove it)
|
||||
@@ -0,0 +1,38 @@
|
||||
# Phase 29 — The gateway dashboard page: animated pipes from config + live sockets
|
||||
|
||||
**Source:** `TODO.md` L3 — "Create a 'gateway' page that visualizes my incoming connections based on the gateway host's nginx and haproxy configs. I want it to show animated pipes with data coming in (IPs, ports, domains, etc.) and data going out (IPs, ports, domains, etc.)." (the dashboard half)
|
||||
**Story:** n/a
|
||||
**Context:** phase 28 (the `haproxy`/`nginx`/`sockets` agent sections, `AGENT_VERSION` 1.2.0, the gateway host rolled out), `src/dashboards/config.py` (`Settings`, `_env`, the `nut_enabled`/`actions_allowed` gate properties), `src/dashboards/sources/disks.py` (the "a source of its own reading the same agents" precedent — the `disks:<host>` job dials the same `/snapshot`), `src/dashboards/poller.py` (job loop, health records, stale/last-good), `src/dashboards/app.py` (`build_hub` — the job list wired once for production and the injected-store test path; the `agent_tls_map` the `host:`/`disks:` jobs share per host), `src/dashboards/api.py` (the snapshot/history route pairs, the `404 {"configured": false}` gate that runs before any dial, `_history_batch`, the `/api/summary` additive blocks), `src/dashboards/alerts.py` (the "an unconfigured source never alerts" rule), `src/dashboards/store.py` (the `sample(source, host, metric, res, ts)` + rollup/retention story — invariant 9), `src/dashboards/web/pages/` + `static/` (the five dashboards, the five-tab masthead pinned by `tests/test_api.py`, `app.js`'s `initChrome` badges, `Dash.hero`/`DashCharts` — no framework, no CDN, invariant 5), `tests/test_api.py` (the served-bytes pins), `tests/test_config.py`, `tests/test_live_fleet.py` (the read-only live suite).
|
||||
|
||||
## Objective
|
||||
|
||||
The fifth dashboard page — `/gateway` — renders the gateway host's connection topology as animated pipes: for every configured frontend (haproxy `bind`, nginx `listen`, the stream `443`) an **inbound** pipe (client → gateway: the port, the `server_name` domains, the live connection count and client addresses) and, fanning out from the gateway, one **outbound** pipe per configured target (haproxy `server`, nginx `proxy_pass`, the stream `map` entries: the target host:port, the live count). The page is a **source of its own** (`gateway`, its own poller job on the same `/snapshot` the `host:` job dials, its own interval/timeout/health/events), its counts are stored as gauges (invariant 9's rollup/retention story) and graphed from `/api/gateway/history`, and it keeps every invariant the fleet already keeps: unconfigured is `404` without a dial (invariant 3), a dead source ages instead of zeroing (invariant 4), no count is a rate (rule 2 — no estimator, no `Δ<window>`), no count carries a verdict (like `net.*`), and an old agent (1.1.0, no sections) reads as "predates gateway support" — dimmed, `n/a`, never red.
|
||||
|
||||
## Dependencies
|
||||
- `28_gateway_agent_proxy` (todo) — the three agent sections on the wire and the gateway host running 1.2.0 (the live half of this phase's verification dials what 28 installed)
|
||||
- `27_mobile_tables_and_audit` (todo) — the page's 360 px behaviour is audited by 27's script shape (the table, if any, scrolls inside its card)
|
||||
|
||||
## Tasks
|
||||
1. `01_config_and_source.md` — `GATEWAY_HOST` in `Settings` + the gate properties; `sources/gateway.py`: the config + sockets join into inbound/outbound pipes, gauges only, the three-way agent support
|
||||
2. `02_wiring.md` — the `gateway` poller job in `build_hub` (30 s), health, the alerts rules, the `/api/summary` block, the `/api/config` entry
|
||||
3. `03_api_routes.md` — `/api/gateway/snapshot` + `/api/gateway/history`: the configured gate before any dial (spy-pinned), history through `_history_batch`, unknown names as empty series
|
||||
4. `04_page.md` — `/gateway` + `page-gateway.js`: the animated pipes (hand-rolled SVG + CSS, no CDN, `prefers-reduced-motion`, `aria-label`), the live counts, the three degradation arms
|
||||
5. `05_nav_and_summary.md` — the sixth tab on every signed-in page (the pinned masthead pins updated + browser-measured), the `/api/summary` pin extended
|
||||
6. `06_docs_and_verify.md` — README/AGENTS (the page row, the `GATEWAY_HOST` row, the fleet fact), the browser screenshot pass against live data, the read-only live verification
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/test_gateway_source.py` (new — the join over fixture snapshots: an inbound join by listen port, the domain-target port-only join labelled "by port", the raw-IP exact join, the `$var`→`map` join, the 1.1.0-predates arm, the never-answered stub arm); `tests/test_config.py` (`GATEWAY_HOST` default, the gate property); `tests/test_api.py` (the route pins, the summary block, the nav pins)
|
||||
- Coverage: **>90%** on `sources/gateway.py` + the new config field
|
||||
- Real-browser check (invariant 10): task 05's measurement + task 06's screenshots against the live dev server
|
||||
|
||||
## Documentation & Consistency
|
||||
|
||||
- Task 06 updates `AGENTS.md`'s page table (`/gateway` row), the `config.py` bullet (`GATEWAY_HOST`), and the metric-naming paragraph (`gateway.*` gauges); `README.md` gains the page's how-to row. The invariant 9 story: the `gateway.*` series roll up raw → 1 m → 1 h like every other gauge.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest` green (the hermetic suite), coverage not down; `uv run pytest --cov` holds the 94% floor
|
||||
- [ ] `GATEWAY_HOST` unset (or a name not in `HOST_TARGETS`) → `/api/gateway/snapshot` answers `404 {"configured": false}` with no dial (spy-pinned) and the page says "not configured"
|
||||
- [ ] a 1.1.0-style snapshot (no sections) → the page's "agent predates gateway support" arm, dimmed, every number `n/a`, the job still `ok` (the invariant-3 shape, pinned)
|
||||
- [ ] the gateway host (1.2.0, post-phase-28) → the page renders the six haproxy frontends, the nginx server blocks and the stream map, with live counts that match a direct `/proc/net/tcp` read (task 06's verification)
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
- [ ] no behavior change in completed work: the nine existing sources read `ok` in `./scripts/dev-server.sh status`, the five existing dashboards render as before, the `/api/summary` pre-existing blocks are byte-identical in shape (the `gateway` block is additive)
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 01 — `GATEWAY_HOST` and the gateway source
|
||||
|
||||
**Phase:** `29_gateway_dashboard_page` · **Source:** `TODO.md:3` — "…visualizes my incoming connections based on the gateway host's nginx and haproxy configs… data coming in (IPs, ports, domains, etc.) and data going out (IPs, ports, domains, etc.)."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`Settings` names which `HOST_TARGETS` entry is the gateway (`GATEWAY_HOST`, default `gateway`), and `sources/gateway.py` projects one agent snapshot into the page's data: the **inbound** pipes (each configured frontend — a haproxy `bind` port, a nginx `listen` port, the stream `443` — with its `server_name` domains and the live ESTABLISHED count + client addresses keyed on the listen port) and the **outbound** pipes (each configured target — a haproxy `server`, a nginx `proxy_pass`, a stream `map` entry — with the live ESTABLISHED count). The counts are gauges: **no rate** (no `RateEstimator`, no `window_seconds`, no `Δ<window>` — a count is a count, rule 2), **no verdict** (no `severity` dict on any count — `severity.py` rates no connection count, the `net.*` precedent), and the three-way agent support degrades a pre-1.2.0 agent to "predates gateway support" (invariant 3), never red.
|
||||
|
||||
- ASSUMPTION: **inbound join**: an ESTABLISHED socket belongs to frontend `P` when its **local port** is `P`'s listen port; the client is the socket's **remote** address. A socket whose local port matches no configured frontend is `unmatched` (reported, not dropped — the page shows "N other connections", the operator's fact that the gateway talks to something the configs don't name).
|
||||
- ASSUMPTION: **outbound join**: an ESTABLISHED socket belongs to target `T` when its **remote** address matches — a raw-IP target on an exact (ip, port) match; a **domain target on a port-only match, labelled `by_port: true`** in the payload (the agent never does DNS — invariant 7's "never guess" applied to the join: a port match against a domain is a real socket fact plus a config fact, and the label says which half is which). A socket whose remote matches no target is `unmatched` too.
|
||||
- ASSUMPTION: the payload shape is `{configured: true, supported: true, host: str, haproxy: {frontends: [{name, port, backend: str|null}]}, nginx: {servers: [{port, server_names: [str]}], map_targets: [{key, host, port}]}, inbound: [{port, source: "haproxy"|"nginx", domains: [str], connections: int, clients: [str]}, outbound: [{target: {host, port, name: str|null, by_port: bool}, connections: int, source: str|null}], unmatched: {inbound: int, outbound: int, other: int}, sockets: {listens: int, established: int}}` — `supported: false` (a 1.1.0-style snapshot) carries the same keys as empty/nulls plus `supported: false`, and a never-answered host is the stub the `/api/system` stubs use (`unreachable: true`, every count `null`, invariant 4).
|
||||
- ASSUMPTION: the `$var`→`map` join: a `proxy_pass $targetBackend` whose variable matches a `map` block contributes the map's entries as `map_targets` (key = the domain, the target = the entry's host:port) and **no** literal outbound pipe (the per-connection target is the map's job; the page renders the map entries as the fan-out — the 17-entry `stream.conf` map is exactly the "data going out (domains)" the TODO names).
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/config.py` — `Settings.gateway_host: str = _env("GATEWAY_HOST", "gateway")`, `gateway_interval_seconds` (default `30.0`, the `DISKS_INTERVAL_SECONDS` idiom) and `gateway_timeout_seconds` (default `10.0`), and the `gateway_enabled` property (`True` iff `gateway_host` is among the `HOST_TARGETS` names — the `nut_enabled` shape); the `any_source_enabled` property includes it.
|
||||
2. `src/dashboards/sources/gateway.py` — new: `parse(snapshot: dict, host: str) -> dict` (the `sources/disks.py` projection idiom — pure over the snapshot the poller fetched, no I/O of its own): the section readers (an absent section → the `supported: false` arm; `available: false`/`null` → a `why` line in the payload, the invariant-3 arms), the inbound/outbound joins per the assumptions above, the `unmatched` tallies, and the `samples` list: one gauge per distinct listen port (`gateway.in_conns@<port>`) and one per distinct target host (`gateway.out_conns@<host>`; a `by_port` domain target's series is keyed on the host, the port is in the payload not the name — the `@stream` scheme, invariant 9's naming).
|
||||
3. `tests/test_config.py` — `GATEWAY_HOST` default `gateway`, an explicit value winning, `gateway_enabled` true/false over the target list (the `nut_enabled` test shape).
|
||||
4. `tests/test_gateway_source.py` — new: the join over fixture-shaped snapshots (reuse the phase-28 fixtures' parsed shape — the test builds the snapshot dict the agent serves): the inbound join (a socket on local `:443` lands in the stream frontend's pipe with its client address; a socket on an unconfigured port is `unmatched.inbound`); the outbound join (a raw-IP target's exact match; a domain target's port-only match with `by_port: true`; no DNS call anywhere — a `socket.getaddrinfo` spy must record nothing); the `$var`→`map` join (the 17-entry map → 17 `map_targets`, no literal pipe); the `supported: false` arm (a 1.1.0-style snapshot without the sections); the never-answered arm is the API's job (task 03), not the source's.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/test_config.py` + `tests/test_gateway_source.py` as above
|
||||
- Coverage: **>90%** on `sources/gateway.py` + the new config fields
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `GATEWAY_HOST` defaults to `gateway` and `gateway_enabled` is true only for a configured name (pinned)
|
||||
- [ ] `parse()` over a 1.2.0 fixture snapshot yields the exact inbound/outbound lists (counts, clients, `by_port`, the map fan-out) — a rename in the agent's section keys fails a test
|
||||
- [ ] the `getaddrinfo` spy records **nothing** across the whole source (pinned — the join never dials DNS)
|
||||
- [ ] the `supported: false` arm renders the payload's `why` and null counts, never a red verdict (pinned)
|
||||
- [ ] `uv run pytest` green, `uv run ruff check src tests` and `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: the pre-existing `Settings` fields and the `host`/`disks`/`llm`/`nut` sources are byte-identical in behaviour
|
||||
@@ -0,0 +1,30 @@
|
||||
# Task 02 — The gateway job: poller, health, alerts, summary, config
|
||||
|
||||
**Phase:** `29_gateway_dashboard_page` · **Source:** `TODO.md:3` — the source is a **job of its own** (the `disks:<host>` precedent): its own interval, timeout, health record, and events, so a wedged config read ages the pipes instead of blinking `/system`
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The `gateway` source runs as its own poller job (the `disks:` shape — one job dials the same `/snapshot` the `host:` job of the gateway host reads, with the gateway's own interval/timeout), so the page ages on its own cadence and a failed poll keeps the last good pipes dimmed instead of zeroing them (invariant 4). The alerts rules keep an unconfigured source silent (invariant 3) and a dead source to exactly one line (the `disks` echo rule — the gateway job dials the same agent, so its failure carries no new information while the `host:` job is the voice; a gateway job failing *while its host answers* — the sections wedged — is the only voice that fault has). `/api/summary` gains the additive `gateway` block, `/api/config` gains the gateway row.
|
||||
|
||||
- ASSUMPTION: the job is `gateway` (one job, the `nut` shape — a single source, not per-host: the page is about the one gateway host), `DISKS`-interval defaults from task 01 (30 s / 10 s), events under the dashboard name `gateway`.
|
||||
- ASSUMPTION: the **no-verdict alert rule**: the source's payload carries no `severity`, so `alerts.py` gains no threshold for `gateway` — the only alerts a `gateway` job raises are reachability (stale > 3× its 30 s interval → `warn`, stopped answering → `fail`, the `disks`-echo suppression applied: a dead gateway job while its `host:` job is also dead adds no line).
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/app.py` — `build_hub`: the `gateway` job (one, not per host) wired the `disks:` job is wired — the same `AgentTls` entry the gateway host's `host:`/`disks:` jobs use (so a revocation fails all three with one sentence, the task-03 `agent_tls_map` shape), its `parse` the `sources.gateway.parse` + `samples` pass-through, no `RateEstimator` (the source takes none — task 01). The job exists **only** when `settings.gateway_enabled` (the `disks` job's configured gate: no job at all when the source is unconfigured, invariant 3).
|
||||
2. `src/dashboards/alerts.py` — the `gateway` block: the reachability rules per the assumption above; the `HREF` table gains `/gateway`; **no** verdict rule (asserted: a `gateway` payload with a fabricated `severity` dict still produces no verdict alert — the module may not have a threshold of its own, the phase-09 disk rule's shape).
|
||||
3. `src/dashboards/api.py` — `/api/summary`: the additive `gateway` block (`{configured, supported, host, inbound: int (Σ connections), outbound: int, map_targets: int, unmatched: int, states: {gateway: str}}` — a configured job with no payload yet is a stub, the `system.hosts` stub shape); `/api/config`: the gateway row (`GATEWAY_HOST`'s name + the `Target.display_url` of the entry it names, redacted — invariant 6's boundary).
|
||||
4. `tests/test_api.py` — the `build_hub` wiring pins (the job present iff `gateway_enabled`; the shared `AgentTls` entry — a revocation written into `meta` fails the `gateway` job **before any dial**, the `test_a_revoked_agent_fails_both_of_its_sources_without_a_dial` shape extended to three sources); the `/api/summary` `gateway` block pin (with and without the source configured — unconfigured: the block is `configured: false` and **absent no dial happens**, the spy-pinned shape); the alerts pins (unconfigured → no `gateway` alert; dead job + dead host → one line, not two; a wedged-sections failure while the host answers → one line naming `gateway`).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the wiring/alerts/summary pins above
|
||||
- Coverage: **>90%** on the modified `app.py`/`alerts.py`/`api.py` branches
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a configured build carries exactly one `gateway` job on the gateway host's shared TLS entry; an unconfigured build carries none (pinned)
|
||||
- [ ] a revocation in `meta` fails the `gateway` job before any dial (spy-pinned, the `host:`/`disks:` pair's shape)
|
||||
- [ ] `/api/summary`'s `gateway` block is pinned with and without configuration; the pre-existing blocks are unchanged
|
||||
- [ ] the alerts pins hold: unconfigured silent, dead-source one line, no verdict rule
|
||||
- [ ] `uv run pytest` green, `uv run ruff check src tests` and `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: the pre-existing jobs, alerts, and summary blocks behave exactly as before
|
||||
@@ -0,0 +1,28 @@
|
||||
# Task 03 — `/api/gateway/snapshot` and `/api/gateway/history`
|
||||
|
||||
**Phase:** `29_gateway_dashboard_page` · **Source:** `TODO.md:3` — the page's data routes, in the app's established shape: the configured gate before any dial (invariant 3), history through the one batched door (invariant 9 / phase-14 task 02)
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`/api/gateway/snapshot` answers the source's payload (task 01's shape: the config, the joined inbound/outbound pipes, the unmatched tallies, `supported`) for the `GATEWAY_HOST` entry — with a configured-but-never-answered host as the `unreachable` stub (every count `null`, invariant 4's page-side half) and `GATEWAY_HOST` unset/unknown as `404 {"configured": false}` **before anything that could reach the network** (no client, no fetch, no job — spy-pinned both ways, the `tests/test_api.py` disks-pair shape). `/api/gateway/history` serves the `gateway.in_conns@<port>` / `gateway.out_conns@<host>` series through the same `_history_batch()` → `store.history_many()` door the other four history routes use (phase-14 task 02 — one batched decision, the same `{source, host, metric, range_seconds, points}` document, unknown names as empty series).
|
||||
|
||||
- ASSUMPTION: the route pair is read-only — a fourth data page grows no write (invariant 1's shape: the app's only POST that dials an upstream stays `/api/ups/command`, and the `tests/test_api.py` route-verb enumeration gains the two GETs to the read-only list).
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/api.py` — the `gateway` router: `GET /api/gateway/snapshot` (the configured gate first — `settings.gateway_enabled` false → `404 {"configured": false}` with **no** client construction and **no** `fetch_snapshot` call, spy-pinned; configured → the job's last payload through `sources.gateway.parse`'s never-answered-stub arm) and `GET /api/gateway/history` (the `hosts=`/`metrics=`/`range=` query shape the other three history routes take, answered through `_history_batch`).
|
||||
2. `src/dashboards/api.py` — the route-verb enumeration test's read-only list gains the two paths (the task-01 assumption's pin).
|
||||
3. `tests/test_api.py` — the pins, the disks pair's shape: (a) `GATEWAY_HOST` unset → `404 {"configured": false}`, the `httpx.AsyncClient.get` spy and the `collectors.fetch_snapshot` spy record **nothing**; (b) `GATEWAY_HOST` a name not in `HOST_TARGETS` → the same refusal, the same no-dial proof; (c) configured, the job's payload present → the snapshot's exact keys (the task-01 shape — a rename fails a test); (d) configured, no payload yet → the `unreachable` stub (every count `null`); (e) the history route answered through `_history_batch` — points read straight out of the store with `query()` (a third path, in neither the loop nor the batch, fails the test — the phase-14 task-02 pin shape) for `gateway.in_conns@443` and a `gateway.out_conns@<host>` series, with an unknown name answering as an empty series.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the five pins above
|
||||
- Coverage: **>90%** on the new routes
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] unconfigured (unset or unknown name) → `404 {"configured": false}` with the two spies recording nothing (pinned both arms)
|
||||
- [ ] the never-answered stub is pinned (`unreachable: true`, every count `null`)
|
||||
- [ ] the history route's points equal the store's rows for the `gateway.*` series; unknown names → empty series (pinned)
|
||||
- [ ] the two GETs are in the read-only enumeration; no new POST exists (pinned)
|
||||
- [ ] `uv run pytest` green, `uv run ruff check src tests` and `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: the pre-existing snapshot/history routes answer byte-identical shapes
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 04 — The `/gateway` page: animated pipes
|
||||
|
||||
**Phase:** `29_gateway_dashboard_page` · **Source:** `TODO.md:3` — "I want it to show animated pipes with data coming in (IPs, ports, domains, etc.) and data going out (IPs, ports, domains, etc.)."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The `/gateway` page renders the joined payload (task 01) as animated pipes: one **inbound** pipe per configured frontend — `client → gateway` — labelled with the listen port, the `server_name` domains (haproxy's L4 frontends label their port only), the live connection count, and up to a few client addresses; and, fanning out from the gateway node, one **outbound** pipe per configured target — `gateway → target` — labelled with the target host:port (and `· by port` on a `by_port` match), its live count, and its source (the haproxy backend / nginx location / stream-map key it came through). The animation is **decorative at a fixed speed** — moving dashes whose direction is the data flow — because a count is not a rate (rule 2): the page never claims a tok/s-shaped number from it. No count is coloured by a verdict (the `net.*` precedent: the stylesheet section carries no `ok`/`warn`/`fail` token), and the three degradation arms render their own sentence: unconfigured ("not configured — set `GATEWAY_HOST`"), a 1.1.0 agent ("agent predates gateway support — install 1.2.0", dimmed, `n/a`), a never-answered host (dimmed `.stale`, `n/a`, the row kept, invariant 4).
|
||||
|
||||
- ASSUMPTION: the render is **one card per source family** (haproxy / nginx-https / nginx-stream / unmatched), each card a hand-rolled SVG: a left column of inbound pipe stubs, the gateway node mid-card (one rounded rect, its label the host name), a right column of outbound pipe stubs — the pipe is a `<path>` whose `stroke-dasharray`/`stroke-dashoffset` CSS animation runs only while the pipe has ≥ 1 live connection (a zero-connection pipe is a static faint line + the count `0`, because animating a pipe nobody is using would be a shape claiming a flow the count denies — the sparkline's "a shape is not a verdict" rule applied to motion); `prefers-reduced-motion: reduce` renders every pipe static (no animation at all, the repo's standing rule).
|
||||
- ASSUMPTION: the labels are `textContent`-only (no `innerHTML` for anything upstream-provided — a config domain is upstream text; the `DashCharts.escapeHtml` idiom via the page's own escape), and every SVG pipe carries `role="img"` + an `aria-label` built from its inputs (the battery-glyph convention: the pipe carries information the markup must say in words) — e.g. `"inbound pipe, port 4443, nextcloud.reeseapps.com, 3 connections"`.
|
||||
- ASSUMPTION: the page is the app's standing chrome — the masthead's sixth tab (task 05), the four `#nav-status` badges (the `gateway` job is the fifth dot, `app.js`'s one-dot-per-job rule), the theme toggle, the window chips **absent** (the page's history is the count trend — one `LineChart` per card, the `Charts` disclosure's `/system` shape: collapsed, lazy-built, one `/api/gateway/history` request naming every series the card could draw, `ariaLabel` on the plot), and the page's single `faint` footnote: the `by port` join sentence (the only why-line the operator needs to read a count).
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/web/pages/gateway.html` — the page: the shared `<head>`/masthead (task 05's tab slot), the four badge container, the card grid (the four cards' skeletons with their empty-state sentences), the `#nav-status`, and no `id` the script does not reference (the orphan-`id` audit's shape).
|
||||
2. `src/dashboards/web/static/page-gateway.js` — the page script (the `page-disks.js` idiom): `state` keyed per card; `renderPipes(card, pipes, direction)` — the SVG builder (the `DashCharts` escape discipline, the `aria-label` per the assumption, the zero-connection static arm, the `by port` sub-label); the degradation arms (the unconfigured / predates / never-answered sentences, the `.stale` dim, `n/a` for every count); the `Charts` disclosure's one-history-request rule (the `/system` shape: collapsed default, lazy build, a cached response from another window re-fetched); the `unmatched` card (the "N other connections" sentence — the operator's fact that the gateway talks to something the configs do not name).
|
||||
3. `src/dashboards/web/static/dash.css` — the pipe rules: `.pipe` (the `<path>` stroke, the `stroke-dasharray`), `@keyframes pipe-flow` (the dashoffset drift), `.pipe.still` (the zero-connection arm), and the `@media (prefers-reduced-motion: reduce)` block naming **both** the animating and the still states (the phase-18 reduced-motion shape — naming only one leaves the other animating).
|
||||
4. `tests/test_api.py` — the served-bytes pins: the pipe `@keyframes` + the reduced-motion rule (both states, inside the file's one such block); the zero-connection `.still` arm; the page's absence of any `ok|warn|fail` token in the gateway card section (the no-verdict pin, the phase-06 GPU shape); the `aria-label` on every `LineChart` the page builds (the standing chart-audit pin, extended); the orphan-`id` audit over the served page; the CDN-reference absence (invariant 5).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the served-bytes pins above (the repo has no JS test runner — a `.cjs` browser pass is the behavioural proof, task 06)
|
||||
- Coverage: **>90%** on the new Python-side branches (the pins); the JS is pinned on served bytes + the browser pass
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the page renders the four cards with their pipes, labels (port / domains / host:port / `· by port` / counts / clients), and the `unmatched` card — against a fixture-driven dev store
|
||||
- [ ] a zero-connection pipe is static (`.still`), a ≥ 1-connection pipe animates, `prefers-reduced-motion` renders all static (pinned + measured by task 06's browser pass)
|
||||
- [ ] the three degradation arms print their own sentence, dimmed where the invariant says, every count `n/a` (pinned)
|
||||
- [ ] no verdict token in the gateway card section; no CDN reference; every chart labelled (pinned)
|
||||
- [ ] `uv run pytest` green, `uv run ruff check src tests` and `uv run pyright` clean, `node --check` on the new script
|
||||
- [ ] no behavior change in completed work: the pre-existing pages render byte-identical
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 05 — The sixth tab, and the `/api/summary` pin
|
||||
|
||||
**Phase:** `29_gateway_dashboard_page` · **Source:** `TODO.md:3` — the page is one of the dashboards: the masthead's tab row gains `Gateway` on every signed-in page, and the overview's fleet probe names the new source
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The masthead's tab row — five tabs on all seven signed-in pages (`/`, `/system`, `/disks`, `/llm`, `/ups` + `/settings`/`/certs`'s no-`aria-current` five) — gains the sixth, `Gateway`, in one order on every page, and the pinned markup audits move from a five-tab assertion to a six-tab one (the pins are the contract; they move on purpose, with the browser measurement as the proof). `app.js`'s one-dot-per-job rule makes the `gateway` job the fifth `#nav-status` badge with no JS change (the container already exists; the job is what the badge reads). The `/api/summary` pin (task 02's additive block) is completed here with the page-side half: the overview renders the `gateway` block's one line (the "N inbound · M outbound" count, linked to `/gateway` via the block's own href), and the no-behavior-change control is the five pre-existing dashboards rendering as before.
|
||||
|
||||
- ASSUMPTION: the tab order is `Overview · System · Disks · LLM · UPS · Gateway` (the new tab last — a sixth tab inserted mid-row would move the `aria-current` targets the five pinned pages assert, and a moved tab is a broken-button read on every page at once).
|
||||
- ASSUMPTION: the tab is `aria-current="page"` on `/gateway` only (the standing rule: the five dashboards carry it on their own page, `/settings`/`/certs` carry the five tabs with **no** `aria-current` — the sixth tab inherits the same shape on the two admin pages).
|
||||
|
||||
## Work
|
||||
|
||||
1. `src/dashboards/web/pages/*.html` — the six tab pages (`index`, `system`, `disks`, `llm`, `ups`, `gateway`) gain the `Gateway` tab (the `aria-current` on `gateway.html` only); `settings.html` and `certs.html` gain the same sixth tab with no `aria-current` (the two admin pages' shape — the phase-20 task-04 "both links on both admin pages" rule extended to the tab row).
|
||||
2. `tests/test_api.py` — the tab pins move: the five-tab assertion becomes the six-tab one (the `MARKUP_PAGES` loop's tuple gains `Gateway`, the order pin per the assumption, the `aria-current` pin per the assumption); the `test_every_page_links_to_both_admin_pages`-shape loop stays one loop over the pages (the phase-25 task-03 pin's shape — one order, one loop).
|
||||
3. `src/dashboards/web/static/page-index.js` — the overview's `gateway` line: the block's one line in the fleet panel's shape (count + the `/gateway` link via the block's `href`), hidden-when-unconfigured (the `disks` block's invariant-3 shape — an unconfigured source never renders a row, never a red one).
|
||||
4. `tests/test_api.py` — the `/api/summary` pin completed: the `gateway` block's keys pinned (task 02) **and** the overview line's rendering pinned on the served `page-index.js` (the line reads the block's keys — a rename fails a test, the `SLOT_FIELDS` device shape).
|
||||
5. `data/logs/29-task05-tabs.cjs` — the browser measurement (the `25-task03-verify.cjs` shape): eight signed-in pages × two widths (1280 / 360) — six tabs present in order, the `aria-current` on `/gateway` only, the five pre-existing tabs' `aria-current` unchanged, the sixth `#nav-status` badge present and populated on `/gateway`, the 360 px dropdown lists all six, no document-level sideways scroll (phase 18's guarantee), screenshots `data/logs/masthead-29-*.png`.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the served-markup pins (the moved tab audit, the `aria-current` pin, the overview-line pin)
|
||||
- Real-browser check (invariant 10): the task-05 script is the proof — eight pages × two widths
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] six tabs on all eight signed-in pages in the pinned order, `aria-current` on `/gateway` only (pinned + measured)
|
||||
- [ ] the `gateway` job is the fifth `#nav-status` badge (measured on `/gateway`, populated from `/api/health`)
|
||||
- [ ] the overview renders the `gateway` line when configured and hides it when unconfigured (pinned)
|
||||
- [ ] the 360 px dropdown lists all six tabs; no document-level sideways scroll (measured)
|
||||
- [ ] `uv run pytest` green, `node --check` on the touched scripts, screenshots in `data/logs/`
|
||||
- [ ] no behavior change in completed work: the five pre-existing dashboards' tab row is byte-identical apart from the added tab (the served-bytes pin proves it)
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 06 — Docs, and the live verification of the rendered pipes
|
||||
|
||||
**Phase:** `29_gateway_dashboard_page` · **Source:** `TODO.md:3` — the definition of done: UI changes verified in a real browser with a screenshot against **live** data (invariant 10), and the docs updated when a page, a config row, or a fleet fact changed
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The docs carry the new page (the `AGENTS.md` table row, the `config.py` bullet's `GATEWAY_HOST`, the metric-naming paragraph's `gateway.*` gauges, the `README.md` how-to row), and the phase's proof is the read-only live pass: the dev server (1.2.0 gateway host, post-phase-28) renders `/gateway` with the real topology — the six haproxy frontends, the nginx server blocks, the stream map — and the page's live counts **match a direct `/proc/net/tcp` read** on the gateway host (the join is proven end-to-end, not just against fixtures). The pass writes its transcript and screenshots to `data/logs/`.
|
||||
|
||||
- ASSUMPTION: the verification is **read-only** (the `tests/test_live_fleet.py` rule): the pass dials the dashboard's public routes over the session, reads `/proc/net/tcp` on the gateway host via the ssh alias, and compares — it starts no workload, sends no command, writes nothing to the fleet.
|
||||
- ASSUMPTION: the count comparison is **port-bucketed, not exact-timestamped**: the pass reads the page's inbound count for port `P` and the host's ESTABLISHED-on-`P` count within one poll window, and asserts they are equal **or** differ by ≤ 1 (a connection can close between the two reads — the one-socket tolerance is the honest bound; a divergence > 1 fails the pass).
|
||||
|
||||
## Work
|
||||
|
||||
1. `AGENTS.md` — the page table row (`/gateway` — "which domains are coming in on which port, and where the traffic goes — the gateway host's nginx + haproxy configs joined to its live sockets" — source: "the same agents, `haproxy`/`nginx`/`sockets` sections of `GET /snapshot` (agent 1.2.0)"); the `config.py` bullet's `GATEWAY_HOST` line (default `gateway`, the gate property); the metric-naming paragraph's `gateway.in_conns@<port>` / `gateway.out_conns@<host>` gauge line (no rate, no verdict — the `net.*` sentence applied).
|
||||
2. `README.md` — the page's how-to row (the `GATEWAY_HOST` setup, the not-configured arm, the 1.2.0 agent requirement) in the standing page-row shape.
|
||||
3. `data/logs/29-task06-live.cjs` — the read-only live pass (the `24-task04-verify.cjs` shape): sign in, `GET /api/gateway/snapshot` (assert `configured: true`, `supported: true`, the six haproxy frontends, the nginx server blocks, the stream map's 17 entries), `GET /gateway` (the six tabs' page, the four cards rendered, the pipes' `aria-label`s present), then `ssh gateway-ops` → `cat /proc/net/tcp /proc/net/tcp6` → the port-bucketed count comparison per the assumption (inbound on each configured listen port; the `unmatched.inbound` tally against the port complement), and the `by_port` spot-check (one domain target's count equals the ESTABLISHED-on-that-port count).
|
||||
4. `data/logs/29-task06-live.out` + screenshots `data/logs/29-task06-{gateway,card-haproxy,card-stream,charts-expanded}.png` — the transcript (the assertion table, the count comparison, **no** config body or token in it — invariant 6) and the screenshots (the phase's proof, invariant 10).
|
||||
5. The 1.1.0 arm's live proof is the **dev-store** one (not a fleet rollback — a fleet arm is the operator's call): the dev store pointed at a 1.1.0-style fixture snapshot renders the "predates" arm, screenshot `data/logs/29-task06-predates.png`.
|
||||
|
||||
## Testing & Quality
|
||||
- Live (read-only, invariant 10): the task-06 pass is the phase's proof; the 1.1.0 arm is dev-store-driven
|
||||
- The hermetic suite is re-run as the gate (`uv run pytest --cov`), not for this task's logic
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `AGENTS.md`'s page table, the `config.py` bullet, and the metric-naming paragraph name `/gateway` + `GATEWAY_HOST` + the `gateway.*` gauges (a grep over both files finds each)
|
||||
- [ ] `README.md` has the page's how-to row
|
||||
- [ ] the live pass's transcript records: the six frontends, the nginx blocks, the 17 map entries, the port-bucketed count comparison (divergence ≤ 1 on every compared port), the `by_port` spot-check
|
||||
- [ ] the four screenshots + the predates-arm screenshot are in `data/logs/`, and the transcript carries no config body and no token (invariant 6, grepped)
|
||||
- [ ] `uv run pytest --cov` holds the 94% floor, `bash .agents/validate.sh` green
|
||||
- [ ] no behavior change in completed work: the nine pre-existing sources read `ok` in `./scripts/dev-server.sh status` and the five pre-existing dashboards render as before (the pass's control leg)
|
||||
Reference in New Issue
Block a user