chore(agent): phase roadmap from TODO.md, 2 phases

Convert the gateway-visualisation TODO item into phases 28 (gateway_agent_proxy: agent sections for haproxy/nginx configs + live sockets, 1.2.0, capture, rollout) and 29 (gateway_dashboard_page: source, wiring, API, animated-pipe page, nav, docs). Clear TODO.md. LLM-Generated: true
2026-10-21 16:24:01 -04:00
parent 89d9fb9f89
commit 558a83d0be
13 changed files with 418 additions and 0 deletions
@@ -0,0 +1,39 @@
+# Phase 28 — The agent reads the gateway's proxy configs and its live sockets
+
+**Source:** `TODO.md` L3 — "Create a 'gateway' page that visualizes my incoming connections based on the gateway host's nginx and haproxy configs. I want it to show animated pipes with data coming in (IPs, ports, domains, etc.) and data going out (IPs, ports, domains, etc.)." (the agent half)
+**Story:** n/a
+**Context:** `agent/host_agent.py` (the stdlib-only single-file agent, `Collector._safe` best-effort sections, `snapshot()`, the SCHEMA docstring, `AGENT_VERSION` 1.1.0, the `--no-smart` opt-out and the three-way availability precedent it sets), `tests/test_agent.py` (the in-process harness over a fake `root` tree, the real-`/proc` test, the smartctl-capture fixture pins), `scripts/capture-fixtures.sh` (the ssh-capture convention: plain command first, `sudo -n` fallback, refuse to overwrite good fixtures with unreadable output), `tests/fixtures/` (the verbatim-capture ground-truth directory), the gateway host's live state verified 2026-10-21 (read-only): `nginx` and `haproxy` both `active`, `/etc/haproxy/haproxy.cfg` (6 `frontend` blocks — `bind :::22 v4v6` et cetera — each with one `default_backend`; 6 `backend` blocks, each one `server <name> <domain>:<port> check inter 5s`), `/etc/nginx/nginx.conf` (a catch-all `listen 80` / `server_name _` block plus `include /etc/nginx/conf.d/*.conf;` and `include /etc/nginx/stream.d/*.conf;`), `/etc/nginx/conf.d/http.conf` (server blocks: `listen 4443 ssl proxy_protocol` / `listen 8443 ssl`, `server_name <domain>`, `location / { proxy_pass http://<domain>:<port>; }`), `/etc/nginx/stream.d/stream.conf` (a `map $ssl_preread_server_name $targetBackend { … }` block and `proxy_pass $targetBackend` with `ssl_preread on`), all three files world-readable and readable by the `dashboards-agent` user, `/proc/net/tcp` readable the same way. The dashboard side (source, wiring, page) is phase 29; this phase makes the data exist on the wire.
+
+## Objective
+
+`host_agent.py` 1.2.0 serves three new **additive** snapshot sections on the gateway host — `haproxy` (frontends + binds, backends + server targets, the frontend→backend routing), `nginx` (server blocks with `listen` + `server_name`, `upstream`, `proxy_pass` including the `$var`-to-`map` join, `map` blocks) and `sockets` (LISTEN + ESTABLISHED from `/proc/net/tcp{,6}`) — each best-effort (a missing config degrades its own section, never the snapshot), `--no-proxy` opting out silently, the real configs captured verbatim as fixtures, the rollout onto the gateway host done, and the sections proven live over mTLS. Nothing changes for token-mode or pre-1.2.0 consumers: an old agent's snapshot is byte-identical in every old field.
+
+## Dependencies
+- `27_mobile_tables_and_audit` (todo) — not a code dependency (different files); ordered after it so the fleet rollout in task 05 does not race the mobile-audit's browser passes on the shared dev box
+- `15_agent_mtls` / `24_dev_mtls_rollout` (complete) — the installed-bundle install path (`deploy/install-agent.sh --cert-bundle`), the `--status` CA-only probe, and the gap-10 re-stage-first rule the rollout follows
+
+## Tasks
+1. `01_haproxy_section.md` — the agent reads + parses `haproxy.cfg` into the additive `haproxy` section, three-way availability, `--no-proxy` opt-out, never fails the snapshot
+2. `02_nginx_section.md` — the agent reads the main config + one level of `include <glob>` (bounded) into the additive `nginx` section, with the `proxy_pass $var` → `map` join
+3. `03_socket_section.md` — the agent parses `/proc/net/tcp` + `/proc/net/tcp6` (bounded) into the additive `sockets` section: LISTEN + ESTABLISHED with local/remote ip:port
+4. `04_version_and_capture.md` — `AGENT_VERSION` 1.2.0, the SCHEMA docstring, `scripts/capture-gateway-config.sh` (real configs → `tests/fixtures/`), the live probe that the sections answer on the gateway host
+5. `05_rollout_gateway.md` — re-stage `/home/ops/agent-install/`, install 1.2.0 on the gateway host, restart the unit, verify the sections over mTLS — read-only live verification, transcript in `data/logs/`
+
+## Testing & Quality
+- Unit: `tests/test_agent.py` — every new section over a fake `root` tree (present config → parsed shape; missing file → `available: false` + exactly one errors line; unreadable → `null` + one errors line; `--no-proxy` → sections absent and no errors line; a corrupt config degrades its own section and the snapshot still succeeds with the other sections intact)
+- Fixture: `tests/fixtures/gateway_haproxy.cfg` + `gateway_nginx_*.conf` + `gateway_proc_net_tcp{,6}.txt` are verbatim captures from the gateway host (invariant 7); the parser tests run against them, so a config drift fails a test before a dashboard does
+- Coverage: **>90%** on the new/modified code (the three collectors + the parsers)
+- Live (read-only, invariant 10): task 04's probe over the mTLS dial, and task 05's post-restart verification with its transcript
+
+## Documentation & Consistency
+
+- The final pass (task 05) updates `AGENTS.md`'s agent bullet (the new sections, `AGENT_VERSION` 1.2.0, the `--no-proxy` line) and the `SCHEMA` docstring is the self-documenting half on the wire. `README.md` is checked for any agent-version sentence.
+
+## Completion Criteria
+- [ ] `uv run pytest` green (the hermetic suite, no network), coverage not down
+- [ ] a 1.2.0 agent over a captured-config root tree answers all three sections; over a tree without them it answers `available: false`/`null` with one errors line each and the snapshot's other sections are intact
+- [ ] `AGENT_VERSION` is 1.2.0 and the SCHEMA docstring names the three new sections
+- [ ] the fixture files are verbatim captures (the capture script's refusal-to-overwrite-with-unreadable pin holds)
+- [ ] the gateway host runs 1.2.0 (`systemctl is-active dashboards-agent` → `active`, the file's `AGENT_VERSION` reads 1.2.0) and the sections answer over mTLS — transcript in `data/logs/`
+- [ ] `bash .agents/validate.sh` green
+- [ ] no behavior change in completed work: a 1.2.0 agent's snapshot is identical to 1.1.0's in every pre-existing field for a non-gateway host (the new keys are the only difference, and `--no-proxy` removes even those)