phase: 28_gateway_agent_proxy
**Phase 28 final pass — complete, 100 % green.** All five tasks were already `complete/`; this pass verified the criteria, re-ran the live half, and did the docs pass. - Gate `bash .agents/validate.sh` **rc=0**: `1650 passed, 15 skipped`, coverage **98.97 %** (floor 94), `ruff` clean, `pyright` **0 errors**, `node --check` clean, hygiene clean (`/tmp/validate-done.log`). - Live: `pytest -m live --live` **11 passed**; `./scripts/dev-server.sh status` nine sources `ok`; mTLS re-read of `gateway` → agent **1.2.0**, all three sections `available: true`, 6/6 frontends routed, 11 nginx servers + `$targetBackend`, **19** listeners, no core errors (`data/logs/28-finalpass-live.out`); installed agent file byte-identical to this tree (`b0e9ebf1b718…`). - Criteria: 1 ✅ pytest/coverage · 2 ✅ three sections over a captured tree, `false`/`null` + one `errors` line each elsewhere with `cpu`/`mem`/`disk_io`/`system` intact · 3 ✅ `AGENT_VERSION` 1.2.0 + SCHEMA names all three · 4 ✅ six fixtures verbatim, capture refuses a bad read, 116 socket address columns hold no globally routable endpoint · 5 ✅ gateway runs 1.2.0 and answers over mTLS (transcript `data/logs/28-task05-rollout.md`, `28-finalpass-live.out`) · 6 ✅ validate · 7 ✅ additive: HEAD's 1.1.0 and this 1.2.0 `collect()`ed over one `--root` are key-for-key equal plus the three new sections, `--no-proxy` equal exactly (`data/logs/verify-28-final.py` → 41 PASS, `28-final-verify.out`). - Docs fixed (README/AGENTS/Containerfile, all re-green): README's opening said "Four dashboards" above a five-row table and said the new sections are "stored in the snapshot" (they are in the **agent's** document — no `/api` payload, no `sample` row, nothing charts them until phase 29); `Containerfile`'s header still said "three dashboards"; AGENTS's coverage figure (98.94 → **98.97 %**), invariant 6 gained the captured-socket-table identifier rule, the agent bullet gained "phase 28 stops at the wire", and the fleet record names the stash-and-compare control plus the volatile listener count. Full record: `data/logs/NOTES-28-final.md`. - Deviations: none functional; no git index/history touched, phase files untouched (`PLAN.md` and `complete/` unchanged). `uv pip list --outdated` still lists 4 packages (fastapi 0.153→0.161 etc.) — advisory only, dependencies untouched by this phase, left for a deliberate pass. - Next pending phase: `.agents/phases/todo/29_gateway_dashboard_page/` (the page that consumes these three sections).
This commit is contained in:
@@ -0,0 +1,39 @@
|
||||
# Phase 28 — The agent reads the gateway's proxy configs and its live sockets
|
||||
|
||||
**Source:** `TODO.md` L3 — "Create a 'gateway' page that visualizes my incoming connections based on the gateway host's nginx and haproxy configs. I want it to show animated pipes with data coming in (IPs, ports, domains, etc.) and data going out (IPs, ports, domains, etc.)." (the agent half)
|
||||
**Story:** n/a
|
||||
**Context:** `agent/host_agent.py` (the stdlib-only single-file agent, `Collector._safe` best-effort sections, `snapshot()`, the SCHEMA docstring, `AGENT_VERSION` 1.1.0, the `--no-smart` opt-out and the three-way availability precedent it sets), `tests/test_agent.py` (the in-process harness over a fake `root` tree, the real-`/proc` test, the smartctl-capture fixture pins), `scripts/capture-fixtures.sh` (the ssh-capture convention: plain command first, `sudo -n` fallback, refuse to overwrite good fixtures with unreadable output), `tests/fixtures/` (the verbatim-capture ground-truth directory), the gateway host's live state verified 2026-10-21 (read-only): `nginx` and `haproxy` both `active`, `/etc/haproxy/haproxy.cfg` (6 `frontend` blocks — `bind :::22 v4v6` et cetera — each with one `default_backend`; 6 `backend` blocks, each one `server <name> <domain>:<port> check inter 5s`), `/etc/nginx/nginx.conf` (a catch-all `listen 80` / `server_name _` block plus `include /etc/nginx/conf.d/*.conf;` and `include /etc/nginx/stream.d/*.conf;`), `/etc/nginx/conf.d/http.conf` (server blocks: `listen 4443 ssl proxy_protocol` / `listen 8443 ssl`, `server_name <domain>`, `location / { proxy_pass http://<domain>:<port>; }`), `/etc/nginx/stream.d/stream.conf` (a `map $ssl_preread_server_name $targetBackend { … }` block and `proxy_pass $targetBackend` with `ssl_preread on`), all three files world-readable and readable by the `dashboards-agent` user, `/proc/net/tcp` readable the same way. The dashboard side (source, wiring, page) is phase 29; this phase makes the data exist on the wire.
|
||||
|
||||
## Objective
|
||||
|
||||
`host_agent.py` 1.2.0 serves three new **additive** snapshot sections on the gateway host — `haproxy` (frontends + binds, backends + server targets, the frontend→backend routing), `nginx` (server blocks with `listen` + `server_name`, `upstream`, `proxy_pass` including the `$var`-to-`map` join, `map` blocks) and `sockets` (LISTEN + ESTABLISHED from `/proc/net/tcp{,6}`) — each best-effort (a missing config degrades its own section, never the snapshot), `--no-proxy` opting out silently, the real configs captured verbatim as fixtures, the rollout onto the gateway host done, and the sections proven live over mTLS. Nothing changes for token-mode or pre-1.2.0 consumers: an old agent's snapshot is byte-identical in every old field.
|
||||
|
||||
## Dependencies
|
||||
- `27_mobile_tables_and_audit` (todo) — not a code dependency (different files); ordered after it so the fleet rollout in task 05 does not race the mobile-audit's browser passes on the shared dev box
|
||||
- `15_agent_mtls` / `24_dev_mtls_rollout` (complete) — the installed-bundle install path (`deploy/install-agent.sh --cert-bundle`), the `--status` CA-only probe, and the gap-10 re-stage-first rule the rollout follows
|
||||
|
||||
## Tasks
|
||||
1. `01_haproxy_section.md` — the agent reads + parses `haproxy.cfg` into the additive `haproxy` section, three-way availability, `--no-proxy` opt-out, never fails the snapshot
|
||||
2. `02_nginx_section.md` — the agent reads the main config + one level of `include <glob>` (bounded) into the additive `nginx` section, with the `proxy_pass $var` → `map` join
|
||||
3. `03_socket_section.md` — the agent parses `/proc/net/tcp` + `/proc/net/tcp6` (bounded) into the additive `sockets` section: LISTEN + ESTABLISHED with local/remote ip:port
|
||||
4. `04_version_and_capture.md` — `AGENT_VERSION` 1.2.0, the SCHEMA docstring, `scripts/capture-gateway-config.sh` (real configs → `tests/fixtures/`), the live probe that the sections answer on the gateway host
|
||||
5. `05_rollout_gateway.md` — re-stage `/home/ops/agent-install/`, install 1.2.0 on the gateway host, restart the unit, verify the sections over mTLS — read-only live verification, transcript in `data/logs/`
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/test_agent.py` — every new section over a fake `root` tree (present config → parsed shape; missing file → `available: false` + exactly one errors line; unreadable → `null` + one errors line; `--no-proxy` → sections absent and no errors line; a corrupt config degrades its own section and the snapshot still succeeds with the other sections intact)
|
||||
- Fixture: `tests/fixtures/gateway_haproxy.cfg` + `gateway_nginx_*.conf` + `gateway_proc_net_tcp{,6}.txt` are verbatim captures from the gateway host (invariant 7); the parser tests run against them, so a config drift fails a test before a dashboard does
|
||||
- Coverage: **>90%** on the new/modified code (the three collectors + the parsers)
|
||||
- Live (read-only, invariant 10): task 04's probe over the mTLS dial, and task 05's post-restart verification with its transcript
|
||||
|
||||
## Documentation & Consistency
|
||||
|
||||
- The final pass (task 05) updates `AGENTS.md`'s agent bullet (the new sections, `AGENT_VERSION` 1.2.0, the `--no-proxy` line) and the `SCHEMA` docstring is the self-documenting half on the wire. `README.md` is checked for any agent-version sentence.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest` green (the hermetic suite, no network), coverage not down
|
||||
- [ ] a 1.2.0 agent over a captured-config root tree answers all three sections; over a tree without them it answers `available: false`/`null` with one errors line each and the snapshot's other sections are intact
|
||||
- [ ] `AGENT_VERSION` is 1.2.0 and the SCHEMA docstring names the three new sections
|
||||
- [ ] the fixture files are verbatim captures (the capture script's refusal-to-overwrite-with-unreadable pin holds)
|
||||
- [ ] the gateway host runs 1.2.0 (`systemctl is-active dashboards-agent` → `active`, the file's `AGENT_VERSION` reads 1.2.0) and the sections answer over mTLS — transcript in `data/logs/`
|
||||
- [ ] `bash .agents/validate.sh` green
|
||||
- [ ] no behavior change in completed work: a 1.2.0 agent's snapshot is identical to 1.1.0's in every pre-existing field for a non-gateway host (the new keys are the only difference, and `--no-proxy` removes even those)
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 01 — The agent reads and parses the haproxy config
|
||||
|
||||
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — "Create a 'gateway' page that visualizes my incoming connections based on the gateway host's nginx and haproxy configs…"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`host_agent.py` gains a `haproxy` snapshot section: the agent reads the haproxy config (path an argument, default `/etc/haproxy/haproxy.cfg`), parses its `frontend` blocks (`name`, each `bind` address:port, `default_backend`/`use_backend` references) and `backend` blocks (`name`, each `server` line's name + target host:port), and serves them as a new additive section. The section follows the smart precedent exactly: a missing file is `available: false` plus **one** errors line for the pass, a file that cannot be read or parsed is `available: null` plus one errors line, `--no-proxy` opts out silently, and no failure of this section ever fails the snapshot or `/health`.
|
||||
|
||||
- ASSUMPTION: the parsed shape is `haproxy: {available: true|false|null, config: str (the path), frontends: [{name, binds: [{addr, port}], default_backend: str|null}], backends: [{name, servers: [{name, target, port}]}]}` — the frontend→backend routing is a name the **dashboard** joins (sources/gateway.py, phase 29), the agent only reports what the config says.
|
||||
- ASSUMPTION: `bind` lines parse the last token's `addr:port` (the captured shape is `bind :::22 v4v6` — the address token is the one that contains `:port` after the word `bind`; a `bind` line without a parsable address:port is skipped with no error, because haproxy allows non-address bind forms and the section degrades to fewer binds rather than failing).
|
||||
- ASSUMPTION: `server` lines parse `server <name> <target>[:port] <rest>` — `check inter 5s` and the like are the ignored rest; a `target` without a port (a bare upstream name) is kept with `port: null` (absent is null, never 0 — the module's own rule).
|
||||
- ASSUMPTION: `use_backend <name>` lines inside a frontend are collected as `backends: [str]` on that frontend (the captured config uses `default_backend` only; the parser covers the captured shape plus `use_backend` because haproxy answers two names for the same routing fact, the way `resolve_command()` does for NUT verbs); unknown directives are ignored, never errors.
|
||||
|
||||
## Work
|
||||
|
||||
1. `agent/host_agent.py` — module docstring: add the three-way availability + `--no-proxy` rule to the design-rules section (the "SMART is a best-effort add-on" shape, adapted: one errors line for a missing binary becomes one errors line for a missing/unreadable config).
|
||||
2. `agent/host_agent.py` — `Collector.__init__`/arguments: a `haproxy_config` path argument (default `/etc/haproxy/haproxy.cfg`) and the `--no-proxy` flag on the CLI (the `--no-smart` precedent: a flag that removes the section entirely, no errors line), threaded into `snapshot()`.
|
||||
3. `agent/host_agent.py` — `Collector.haproxy()` + a module-level `parse_haproxy_config(text) -> dict` (the `parse_smart_output` idiom: a pure function over text so tests feed it fixtures without a filesystem): the block scanner (a `frontend`/`backend` line starts a block, an indented line belongs to it, a top-level line ends it) and the `bind`/`server`/`default_backend`/`use_backend` token parsers per the assumptions above.
|
||||
4. `agent/host_agent.py` — `snapshot()`: `self._safe(self.haproxy, "haproxy", errors)` added to the returned document (additive key; a 1.1.0 consumer sees no difference, a 1.2.0 consumer with `--no-proxy` sees no `haproxy` key at all — the same absence the smart section takes).
|
||||
5. `tests/test_agent.py` — the in-process harness cases: the captured-shape fixture parses to the exact expected frontends/backends (assert the `:::22` → `addr: "::"`, `port: 22` split and the six `default_backend` names); a `root` tree without the file → `available: false` + exactly one errors line naming the missing path (the smartctl "not found" pin's shape — the line must still say the file is missing, `tests/test_agent.py` pins that wording); an unreadable file (a mode-000 file in the fake tree, where the harness allows it) → `available: null` + one errors line; a corrupt file (garbage text) → `available: null` + one errors line and the snapshot's `cpu`/`memory` sections still intact; `--no-proxy` → no `haproxy` key and no errors line.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the five harness cases above + `parse_haproxy_config` over `tests/fixtures/gateway_haproxy.cfg` (task 04 records it; write the test against the captured shape now — the real file's content is known: 6 frontends, 6 backends, the exact `bind :::NNNN v4v6` and `server <name> <domain>:<port> check inter 5s` lines)
|
||||
- Coverage: **>90%** on `parse_haproxy_config` + `Collector.haproxy`
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/test_agent.py -q` green, including the new haproxy cases and every pre-existing agent test
|
||||
- [ ] a snapshot over a tree with the config carries the `haproxy` section with all six frontends and six backends; over a tree without it, `available: false` + one errors line; the snapshot never fails because of it
|
||||
- [ ] `--no-proxy` removes the section with no errors line (pinned)
|
||||
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: every pre-existing snapshot field byte-identical on a host without haproxy (the `haproxy` key is the only addition)
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 02 — The agent reads and parses the nginx configs
|
||||
|
||||
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — "Create a 'gateway' page that visualizes my incoming connections based on the gateway host's nginx and haproxy configs…"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`host_agent.py` gains an `nginx` snapshot section: the agent reads the main nginx config (path an argument, default `/etc/nginx/nginx.conf`) **and follows its `include <path-or-glob>` lines one level** (the captured main config includes `/etc/nginx/conf.d/*.conf` and `/etc/nginx/stream.d/*.conf`), parses `server` blocks (`listen` address:port pairs, `server_name`), `upstream` blocks, `proxy_pass` directives (a literal `scheme://host:port` **and** a `proxy_pass $var` whose target the agent resolves against the file's `map` blocks), and the `map` blocks themselves. Same three-way availability and best-effort rules as task 01: missing main file → `available: false` + one errors line; unreadable/unparseable → `null` + one line; `--no-proxy` removes the section; the snapshot never fails because of it.
|
||||
|
||||
- ASSUMPTION: the parsed shape is `nginx: {available: true|false|null, config: str, files: [str (main + the included paths actually read)], servers: [{listen: [{addr, port}], server_names: [str], proxy_passes: [{location: str, scheme: str|null, target: {host, port}|null, variable: str|null}]}], upstreams: [{name, servers: [{host, port}]}], maps: [{variable: str, entries: [{key, target: {host, port}|null}]}]}` — a `proxy_pass $targetBackend` whose `$targetBackend` matches a `map` block's variable is served with `variable` set and `target: null` (the map decides per-connection at runtime; the **dashboard** joins the variable to the map, the agent reports both facts and never merges them, because a merged target would be a number the wire did not publish).
|
||||
- ASSUMPTION: include-following is **one level, bounded** — `include` lines in the main config are read (glob expanded), but `include` lines inside an included file are not followed (the captured tree is one level deep; a deeper include is a config the operator changed, and the section degrades to `files` naming only what was read rather than erroring). Bounds: at most 8 included files and 1 MB of included text combined; exceeding a bound drops the excess includes with one errors line (the smart pass-budget shape: the pass reports what it did not cover) and the section stays `available: true`.
|
||||
- ASSUMPTION: `listen` parsing takes the first token after `listen` (the captured shapes: `listen 4443 ssl proxy_protocol;`, `listen [::]:4443 ssl;`, `listen 80;`) — `host:port`, `[host]:port`, or bare `port` (→ `addr: null`); the `ssl`/`proxy_protocol`/`http2` flags are not parsed (they change how the bytes travel, not what the page's pipes show).
|
||||
- ASSUMPTION: `server_name _` (the catch-all) is kept as the literal `_` entry — the dashboard renders it as "any domain", the agent does not interpret it.
|
||||
|
||||
## Work
|
||||
|
||||
1. `agent/host_agent.py` — `Collector.__init__`/arguments: a `nginx_config` path argument (default `/etc/nginx/nginx.conf`), threaded through like `haproxy_config`; `--no-proxy` covers this section too (it already exists from task 01).
|
||||
2. `agent/host_agent.py` — a module-level `parse_nginx_config(main_text, included: dict[str, str]) -> dict` (pure over text, the `parse_haproxy_config` idiom) with: a brace-block scanner (nginx directives nest by `{}`, unlike haproxy's indent — the scanner is the task's core); `server`/`upstream`/`map` block extraction; the `listen`/`server_name`/`proxy_pass`/`include`/`server`-inside-upstream token parsers per the assumptions above.
|
||||
3. `agent/host_agent.py` — `Collector.nginx()`: read the main config through the existing `read_text` (which returns `None` on unreadable — the section's `null` arm), expand each `include` line (a glob through `glob.glob`, the literal path otherwise), apply the 8-file/1 MB bounds, call the parser, and return the section dict with `files` naming the main + each included path actually read.
|
||||
4. `agent/host_agent.py` — `snapshot()`: `self._safe(self.nginx, "nginx", errors)` added to the returned document (additive, task 01's shape).
|
||||
5. `tests/test_agent.py` — the in-process harness cases: the captured main + `conf.d/http.conf` + `stream.d/stream.conf` (as a fake `root` tree under `etc/nginx/`) parse to the expected servers (assert the `4443`/`8443` listen ports, the `server_name` list, the `proxy_pass` targets including the `127.0.0.1:8081` literal), the `stream.conf` `map` block (assert the variable name `$targetBackend` and a representative entry), and the `proxy_pass $targetBackend` served with `variable: "targetBackend"` + `target: null`; a tree without the main file → `available: false` + one errors line; a corrupt main file → `null` + one line, snapshot intact; the include bound (a main config including 9+ files) → `available: true`, `files` the first 8, one errors line naming the dropped count.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the five harness cases above + `parse_nginx_config` unit cases for the token parsers (`listen [::]:4443` → `addr: "::"`, bare `80` → `addr: null`, a `proxy_pass http://host:port` with a scheme, an `upstream` block's `server` lines)
|
||||
- Coverage: **>90%** on `parse_nginx_config` + `Collector.nginx`
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/test_agent.py -q` green, including the new nginx cases and every pre-existing agent test
|
||||
- [ ] a snapshot over the captured-config tree carries the `nginx` section with every `server` block, the `map`, and the variable-form `proxy_pass` served unmerged
|
||||
- [ ] the one-level include bound is pinned (9th+ include dropped with one errors line, section still `available: true`)
|
||||
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: every pre-existing snapshot field byte-identical on a host without nginx
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 03 — The agent reads the live sockets
|
||||
|
||||
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — "I want it to show animated pipes with data coming in (IPs, ports, domains, etc.) and data going out (IPs, ports, domains, etc.)." (the live half — the configs say *where* the pipes go; this task says *what is flowing now*)
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
`host_agent.py` gains a `sockets` snapshot section: the agent parses `/proc/net/tcp` and `/proc/net/tcp6` (both readable by the agent user, verified on the gateway host) into the LISTEN sockets (the gateway's open frontends, with their local address:port) and the ESTABLISHED sockets (local and remote address:port — the live end-to-end pairs the dashboard joins to the config). Same best-effort rules: an unreadable `/proc/net/tcp` degrades the section to `null` with one errors line, `--no-proxy` removes it, and the snapshot never fails because of it. The section is a raw fact list — **no** config join happens in the agent (a socket does not know its domain; the dashboard's `sources/gateway.py` does the join, phase 29).
|
||||
|
||||
- ASSUMPTION: the parsed shape is `sockets: {available: true|false|null, listens: [{family: "v4"|"v6", addr: str|null, port: int}], established: [{family, local: {addr, port}, remote: {addr, port}}]}` — `addr` is the dotted/colon form (v4) or the expanded `x:x:…` form (v6, the `/proc` 4-byte-group little-endian spelling unrolled); a `0.0.0.0`/`::` listener keeps its literal wildcard address (it is the fact the dashboard's inbound join keys on — the port, with the wildcard addr marking "any address").
|
||||
- ASSUMPTION: only the two states the page needs are reported — `0A` (LISTEN) into `listens`, `01` (ESTABLISHED) into `established` — every other state is skipped (a TIME_WAIT ocean is noise the pipes do not show, and skipping it keeps the section small).
|
||||
- ASSUMPTION: the read is bounded at 4096 lines per file (the header line excluded from the count); a file longer than the bound reports what it read and one errors line naming the truncation (the smart pass-budget shape), `available` stays `true`.
|
||||
- ASSUMPTION: the hex→address conversion is a pure module-level `parse_proc_net_tcp(text, family) -> (listens, established)` (the `parse_smart_output` idiom), so the captured-`/proc` fixtures test the parser without a live filesystem.
|
||||
|
||||
## Work
|
||||
|
||||
1. `agent/host_agent.py` — module docstring: the `sockets` design line (two states only, the 4096-line bound, the best-effort rule) beside the smart one.
|
||||
2. `agent/host_agent.py` — module-level `parse_proc_net_tcp(text: str, family: str) -> tuple[list[dict], list[dict]]`: the column split (the `/proc/net/tcp` columns: `sl local_address rem_address st …`; `local_address`/`rem_address` are `HEXADDR:HEXPORT`, v4 a 32-char hex little-endian word, v6 four 32-char groups in the same order), the `st` filter (`01`/`0A` only), and the address unrolling per the assumptions above.
|
||||
3. `agent/host_agent.py` — `Collector.sockets()`: read `/proc/net/tcp` and `/proc/net/tcp6` through `read_lines` (an unreadable file is `None` → the `null` arm; both unreadable → `null` + one combined errors line), apply the line bound, call the parser, and return the section dict.
|
||||
4. `agent/host_agent.py` — `snapshot()`: `self._safe(self.sockets, "sockets", errors)` added to the returned document (additive, tasks 01/02's shape).
|
||||
5. `tests/test_agent.py` — the parser unit cases over a captured `/proc/net/tcp` fragment (a LISTEN on `:::443` and one on `0.0.0.0:80`, an ESTABLISHED v4 pair, an ESTABLISHED v6 pair, a TIME_WAIT line asserted **absent** from both lists) + the harness cases: a fake tree whose `proc/net/tcp{,6}` carry the fragment → the section's exact lists; a tree without the files → `null` + one errors line, snapshot intact; the 4096-line bound (a 5000-line file) → `available: true`, the first 4095 data rows, one errors line naming the truncation.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the parser unit cases + the three harness cases above
|
||||
- Coverage: **>90%** on `parse_proc_net_tcp` + `Collector.sockets`
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `uv run pytest tests/test_agent.py -q` green, including the new socket cases and every pre-existing agent test
|
||||
- [ ] a snapshot over a tree with the captured `/proc` files carries the `sockets` section with the exact LISTEN/ESTABLISHED lists; the TIME_WAIT line is absent (pinned)
|
||||
- [ ] the 4096-line bound is pinned (truncation line, section still `available: true`)
|
||||
- [ ] `uv run ruff check agent tests` and `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: every pre-existing snapshot field byte-identical on a host where `/proc/net/tcp` is unreadable (the section is `null`, nothing else moved)
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 04 — Agent 1.2.0, the SCHEMA, and the gateway-config captures
|
||||
|
||||
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — the ground-truth half: invariant 7 (verify against the deployed upstream; never guess a metric name) applied to the three new sections
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The agent's version and self-documentation catch up with tasks 01–03 (`AGENT_VERSION` 1.2.0, the SCHEMA docstring names the three sections), and the capture script records the gateway host's real `haproxy.cfg`, nginx configs, and `/proc/net/tcp{,6}` as verbatim fixtures — so the parser tests run against the deployed shape, not a memory of it, and a config drift fails a test before a dashboard does.
|
||||
|
||||
- ASSUMPTION: the fixtures **keep the real service domain names** (e.g. `nextcloud.reeseapps.com`) — unlike the smartctl capture's Serial/WWN scrub, the domains are the data the `/gateway` page exists to show (inbound domain, outbound target) and the page is session-gated; a scrubbed fixture would be a fixture that tests a config the gateway does not run. The capture script therefore scrubs nothing; what it refuses is the other direction (overwriting a good fixture with unreadable output, the `capture-fixtures.sh` rule).
|
||||
- ASSUMPTION: `AGENT_VERSION` goes 1.1.0 → 1.2.0 (a new additive section is a minor bump, the 1.0.0 → 1.1.0 TLS bump's shape); the bundle's `# BUNDLE-REV` is untouched (the agent file is not the bundle — no CA fact moved).
|
||||
|
||||
## Work
|
||||
|
||||
1. `agent/host_agent.py` — `AGENT_VERSION = "1.2.0"` and the SCHEMA docstring: the three new section shapes (the `haproxy`/`nginx`/`sockets` blocks, one line each, the way `smart` is documented) plus the three-way availability + `--no-proxy` note.
|
||||
2. `scripts/capture-gateway-config.sh` — a new capture script (the `capture-fixtures.sh` convention, ssh-alias driven: `GATEWAY_SSH` default `gateway-ops`, `ssh` alias `gateway-ops`): scp/capture `/etc/haproxy/haproxy.cfg` → `tests/fixtures/gateway_haproxy.cfg`; `/etc/nginx/nginx.conf`, `/etc/nginx/conf.d/*.conf`, `/etc/nginx/stream.d/*.conf` → `tests/fixtures/gateway_nginx_main.conf` / `gateway_nginx_conf_d_*.conf` / `gateway_nginx_stream_d_*.conf`; `cat /proc/net/tcp /proc/net/tcp6` → `tests/fixtures/gateway_proc_net_tcp.txt` / `gateway_proc_net_tcp6.txt`. The `capture-fixtures.sh` refusal rule: a capture that comes back empty or with an ssh/permission error is **not** written over an existing fixture (one line naming the failure, exit 1). No sudo (all five reads are world-readable by the agent user — verified 2026-10-21).
|
||||
3. Run the capture once (the operator's box reaches the gateway host) and commit the fixtures; `tests/test_agent.py`'s task 01–03 cases already assert against the captured shape.
|
||||
4. `tests/test_deploy_scripts.py` — pin the script the way `capture-fixtures.sh` is pinned: it dials the ssh alias (not a hardcoded IP), scrubs nothing (a domain grep over its own source asserts no scrub step exists), and the refusal-to-overwrite branch is named.
|
||||
5. Live probe (read-only, invariant 10): `ssh gateway-ops` — `systemctl is-active dashboards-agent`, then a `python3` one-liner (the `--status` probe idiom, CA-only over the installed bundle's CA) that dials the **current** 1.1.0 agent and confirms the old sections still answer (the no-behavior-change control), and confirms the new sections are **absent** from the 1.1.0 snapshot (the three-way `predates` arm's ground truth). The probe writes `data/logs/28-task04-probe.out`.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: `tests/test_deploy_scripts.py` pins for the capture script; the existing fixture-pinned parser tests (tasks 01–03) run against the real captures
|
||||
- Coverage: **>90%** on the new script's Python-side branches (the script is bash; its pins are the `test_deploy_scripts.py` shape)
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `AGENT_VERSION` reads `1.2.0` and the SCHEMA docstring names all three new sections (pinned by a `tests/test_agent.py` assert on the constant)
|
||||
- [ ] `bash scripts/capture-gateway-config.sh` (or the operator's run) leaves the seven fixture files in `tests/fixtures/`, verbatim, with the refusal rule holding
|
||||
- [ ] `tests/test_agent.py` green against the real captures (a config drift in the fixture fails a test)
|
||||
- [ ] `data/logs/28-task04-probe.out` records: the gateway's installed agent is 1.1.0, the old sections answer, the three new sections are absent
|
||||
- [ ] `uv run ruff check scripts` (if the script has a Python leg) / `bash -n scripts/capture-gateway-config.sh` clean, `uv run pyright` clean
|
||||
- [ ] no behavior change in completed work: the 1.1.0 agent's served snapshot is unchanged (the probe proves it)
|
||||
@@ -0,0 +1,31 @@
|
||||
# Task 05 — Roll 1.2.0 onto the gateway host, verify over mTLS
|
||||
|
||||
**Phase:** `28_gateway_agent_proxy` · **Source:** `TODO.md:3` — the deployment half: the sections only exist on the wire once the gateway host runs 1.2.0 (the phase-24 rollout's per-host safety order, gap-10's re-stage-first rule)
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
|
||||
The gateway host runs `host_agent.py` 1.2.0 with the three new sections live over mTLS, installed by the same granted-path procedure the fleet uses — re-stage `/home/ops/agent-install/` first (the gap-10 rule: a path grant is only as current as the file it points at), run the installer, restart the unit, and verify the sections answer. The install is **read-only on the dashboard's side** (no config change, no `HOST_TARGETS` change — the gateway is already in `HOST_TARGETS` and dials fine on 1.1.0), and the verification transcript lands in `data/logs/`.
|
||||
|
||||
- ASSUMPTION: the gateway host's unit is **unchanged by this rollout** — `--no-proxy` is not passed (the sections are on by default), so the installer's re-run guard sees an identical unit render and stays silent (the phase-17 task-03 silent path); no `--force` needed.
|
||||
- ASSUMPTION: the restart is the `systemctl restart dashboards-agent` the grant's `systemctl restart *` line allows (the gateway's `ops` has `(ALL) NOPASSWD: ALL` — the fleet record's `deploy/agent-sudoers` shape).
|
||||
|
||||
## Work
|
||||
|
||||
1. Re-stage: `ssh gateway-ops` — copy the repo's `agent/host_agent.py` (1.2.0, md5 recorded) and `deploy/install-agent.sh` (the repo's current sha256, recorded) into `/home/ops/agent-install/` (the directory is `ops`-owned, no root needed — the phase-24 task-03 procedure), leaving any existing snapshot beside it as `<name>-<date>-backup` (a file sudo will not run).
|
||||
2. Install: run the granted path `sudo /home/ops/agent-install/deploy/install-agent.sh --bind 10.1.1.27 --port 9120` (the gateway's existing unit arguments — read them first with `systemctl cat dashboards-agent` and reproduce them exactly, plus `--cert-bundle` if the unit has the line) — the re-run guard prints nothing (identical unit) and the agent file is updated in place.
|
||||
3. Restart: `sudo systemctl restart dashboards-agent`; `systemctl is-active dashboards-agent` → `active`.
|
||||
4. Verify (read-only): the `--status` CA-only probe (the `python3` `ssl` one-liner over the installed bundle's CA, `--bind 10.1.1.27 --port 9120`) answers; then a `python3` one-liner that dials over mTLS (the dashboard's client cert, the stored CA — the `tests/test_live_fleet.py` dial idiom) and asserts: `agent_version` is `1.2.0`, the three new keys are present, `haproxy.available` is `true` with 6 frontends, `nginx.available` is `true` with ≥ 9 server blocks and the `$targetBackend` map, `sockets.available` is `true` with a non-empty `listens` (port 443 among them), and the pre-existing sections (`cpu`, `memory`, `network`) still answer.
|
||||
5. `data/logs/28-task05-rollout.md` — the transcript: the md5/sha256 of what was staged where, the `systemctl cat` before/after (diff empty), the probe output, the one-liner's assertions, and the `haproxy`/`nginx`/`sockets` section summaries (counts only, no config text — invariant 6: no config body in the note).
|
||||
|
||||
## Testing & Quality
|
||||
- Live (read-only, invariant 10): the verification one-liner is the phase's proof; no dashboard process is touched, no `HOST_TARGETS` line moves
|
||||
- The hermetic suite is untouched by this task (no code change) — `uv run pytest` is re-run as the gate, not for this task's logic
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the gateway host's installed `host_agent.py` is the repo's 1.2.0 (md5 matches) and `systemctl is-active dashboards-agent` → `active`
|
||||
- [ ] the `--status` CA-only probe answers; the mTLS one-liner asserts all five facts (version, three sections present + available, pre-existing sections intact)
|
||||
- [ ] `data/logs/28-task05-rollout.md` exists with the transcript and no config text
|
||||
- [ ] the dashboard's `host:gateway` / `disks:gateway` jobs still read `ok` in `./scripts/dev-server.sh status` (the 1.2.0 snapshot's new keys cost the old sources nothing)
|
||||
- [ ] `uv run pytest` green, `bash .agents/validate.sh` green
|
||||
- [ ] no behavior change in completed work: the gateway's pre-existing snapshot fields are unchanged (the one-liner's control assertions prove it)
|
||||
Reference in New Issue
Block a user