services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 32_gateway_resolved_target_liveness

Browse Source 
Fixture and live stream.conf are byte-identical (modulo header comment). Let me test the parser directly:
This commit is contained in:
ducoterra
2026-10-26 14:03:42 -04:00
parent 25fd74b7c5
commit 51772ffef2
58 changed files with 5272 additions and 478 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent/host_agent.py
+523 -19
View File
@@ -67,6 +67,28 @@ Design rules
answering is `available: null` with one `sockets:` line and an otherwise intact snapshot,
and `--no-proxy` removes it with the rest of the family, because a box with no proxy has
nothing to watch from either side.
* **Names become addresses where they are actually dialled.** The `resolver` section takes every
`(name, port)` the two parsed configs write as a target — haproxy `server` lines, nginx
`upstream` members, `proxy_pass` targets and `map` entry targets, and **nothing else, ever**
(not a name from `/proc`, not a client address) — and asks libc what each one is:
`getaddrinfo` over `SOCK_STREAM`, which is the resolution haproxy's own libc path performs *on
this box*. That is the point — the dashboard resolving the same name through its own
`resolv.conf` would be a different vantage answering a different question — so `via: "libc"`
and the nginx config's own `resolver` line (`config_directive`) are both published and the
dashboard resolves nothing. An address literal in a config is `status: "literal"` with no
lookup (it is already an address, and dialling it to be told so is a round trip); a name that
does not answer is `nxdomain`/`error` with `addresses: []`, never a stale guess and never a 0.
The pass is SMART's shape for SMART's reason: at most `RESOLVE_MAX_NAMES` lookups inside
`RESOLVE_PASS_BUDGET_SECONDS` of wall clock, cached `RESOLVE_CACHE_SECONDS`, the **first** pass
inline and every later expiry refreshed on one single-flight background thread while the
request is served the previous result — a black-holing resolver must never blow the dashboard's
`HOST_TIMEOUT_SECONDS` and blink the *system* page over an add-on it does not draw. Each entry
carries the time **it** was answered, so a cache-served entry cannot pretend to be current.
`available` is the family's three-way answer (`true` a pass ran, `false` nothing to resolve
because no proxy config was read — one `resolver:` line, saying **not found** when the files
are simply absent — `null` a pass that could not run), and `--no-resolve` removes the key in
silence the way `--no-smart` does; `--no-proxy` removes it too, because a box that collects no
proxy has no declared names to look up.
* **The state** is a small pid->cpu-time cache so `processes.top_cpu` can answer a
question /proc cannot (CPU needs two samples), bounded and cleared when stale, plus
the short-TTL SMART result cache above.
@@ -158,6 +180,31 @@ SCHEMA (GET /snapshot)
# --no-proxy already reports "did not look" by removing the key. No
# config join: a socket does not know its domain, so the gateway page
# joins these ports to `haproxy` and `nginx` itself.
resolver {available: true|false|null, via: "libc", config_directive: str|null,
ts: float|null, entries: [{name, port, status: ok|nxdomain|error|literal,
addresses: [str], ts: float}], errors: [str]}
# one libc lookup per (name, port) the two sections above write as a target,
# performed here because this is the box the proxy dials from; additive
# (phase 32) and absent entirely with --no-resolve or --no-proxy. `via` is
# how the addresses were obtained (getaddrinfo — haproxy's libc path) and
# `config_directive` is the nginx config's own `resolver` line kept verbatim
# (`"10.1.0.1"` here), so the two vantages read side by side and a
# divergence is the operator's finding rather than a second resolver here.
# `ts` is the wall clock of the pass that produced the entries while each
# entry carries the time *it* was answered, because an expired cache is
# served as it was while a background thread refreshes it: never two passes
# per snapshot, never a request that waits on DNS past the first one.
# `addresses` is the unique answer sorted v4-ascending then v6-ascending so
# two polls of a healthy gateway are byte-identical; `status: literal` means
# the config already wrote an address and no lookup was made, and
# `nxdomain`/`error` mean "no address right now" with `addresses: []` — the
# page draws those routes with no count, never with a 0. `available` is
# false only when there was nothing to resolve because no proxy config was
# read (one `resolver:` line, saying **not found** when the files are simply
# absent), and null for a pass that could not run; configs that were read
# and name no target are true with `entries: []`. The lookups are bounded
# (RESOLVE_MAX_NAMES, RESOLVE_PASS_BUDGET_SECONDS) and the excess says so in
# one line, so no resolver however broken holds /snapshot past its timeout.
errors [str] reads that failed; non-empty means the snapshot is partial
Deployment: see deploy/install-agent.sh (systemd unit, binds 127.0.0.1 unless you pass
@@ -185,12 +232,12 @@ import subprocess
import tempfile
import threading
import time
from collections.abc import Callable
from collections.abc import Callable, Iterable, Mapping, Sequence
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from typing import Any, cast
AGENT_VERSION = "1.2.0"
AGENT_VERSION = "1.3.0"
SCHEMA_VERSION = 1
PAGE_SIZE = 0 # resolved from sysconf at runtime
@@ -335,6 +382,46 @@ PROC_NET_MAX_LINES = 4096
PROC_NET_STATE_LISTEN = "0A"
PROC_NET_STATE_ESTABLISHED = "01"
# --------------------------------------------------------------------------- #
# name resolution (the additive `resolver` section)
# --------------------------------------------------------------------------- #
#: How many declared targets one pass looks up. A gateway's config-declared targets are a few
#: dozen with room to grow, and the bound is what stops a config that grew into the hundreds (or
#: a map written by someone who liked wildcards) from turning a snapshot into a DNS storm. The
#: excess is reported in one `errors` line naming the count — the nginx include-bound and SMART
#: pass-budget shape: say what was not covered instead of serving a shorter list silently.
RESOLVE_MAX_NAMES = 64
#: How long a resolved address set stays fresh. A LAN target moves when the operator moves it,
#: not every poll, and this section exists to be *cheap*: the dashboard polls the agent every
#: 10 s, and a lookup per name per poll would make the agent the noisiest DNS client on the
#: network. Anything older than this is refreshed on a background thread, never on the request.
RESOLVE_CACHE_SECONDS = 300.0
#: The wall-clock ceiling on one inline pass. `socket.getaddrinfo` takes no timeout argument, so
#: the only protection against a black-holing resolver is stopping between lookups — and the
#: reason that matters is the same one SMART learned: blowing `HOST_TIMEOUT_SECONDS` here turns
#: the **system** page red over an add-on it never draws (AGENTS.md invariant 3/4). The first
#: pass runs inline (a fresh agent must answer with real addresses), every later one runs on a
#: background thread, so this bound is the worst cost of a single request.
RESOLVE_PASS_BUDGET_SECONDS = 3.0
#: How the addresses were obtained, published rather than implied: `libc` means
#: `getaddrinfo()`, which is the resolution haproxy's own `init-addr last,libc,none` performs on
#: this box. It is not the `resolver` directive's answer (nginx queries that server directly),
#: and the payload says both so an operator can see a divergence — this module does not hand-roll
#: UDP DNS queries to name it a second time.
RESOLVE_VIA = "libc"
#: The `getaddrinfo` return codes that mean "this name has no address right now" as opposed to
#: "the lookup could not be performed". `EAI_NODATA` is what glibc calls a no-data answer and
#: CPython does not export it (it is deprecated upstream), so the value is spelled out beside a
#: `getattr` rather than guessed at: a temporary DNS failure (`EAI_AGAIN`) is deliberately **not**
#: in this set, because "no answer" and "no server" are different operator problems.
_EAI_NODATA = getattr(socket, "EAI_NODATA", -5) # CPython does not export it
RESOLVE_NXDOMAIN_CODES = frozenset({socket.EAI_NONAME, _EAI_NODATA})
# pci slot -> human model, filled on first look (lspci output cannot change at runtime).
_GPU_MODEL_CACHE: dict[str, str | None] = {}
@@ -404,7 +491,7 @@ class Collector:
def __init__(self, root: str | os.PathLike[str] = "/", *, top_processes: int = 5,
net_all: bool = False, smartctl: str = "smartctl",
smart: bool = True, proxy: bool = True,
smart: bool = True, proxy: bool = True, resolve: bool = True,
haproxy_config: str = HAPROXY_CONFIG_PATH,
nginx_config: str = NGINX_CONFIG_PATH) -> None:
self.root = Path(root)
@@ -420,6 +507,22 @@ class Collector:
self.proxy_enabled = proxy
self.haproxy_config = haproxy_config
self.nginx_config = nginx_config
# Name resolution: its own flag, because it is a different kind of work from reading a
# file — it leaves the host. It is meaningless without the proxy sections (they are the
# only source of names, decision 3), so `--no-proxy` removes it as well as `--no-resolve`.
self.resolve_enabled = resolve
self.resolve_max_names = RESOLVE_MAX_NAMES
self.resolve_cache_seconds = RESOLVE_CACHE_SECONDS
self.resolve_pass_budget_seconds = RESOLVE_PASS_BUDGET_SECONDS
self._resolver_lock = threading.Lock() # a pass may run while a handler serves
# Entries keyed by `resolver_key(name, port)`; the timestamp is the monotonic age of the
# pass that wrote them, `_resolver_pass_ts` the wall clock the section publishes (an entry
# keeps the time *it* was answered, so a served-from-cache entry cannot look current).
self._resolver_cache: dict[str, dict[str, Any]] = {}
self._resolver_cache_ts: float | None = None
self._resolver_pass_ts: float | None = None
self._resolver_errors: list[str] = []
self._resolver_refreshing = False # at most one background refresh at a time
self.smart_timeout_seconds = SMARTCTL_TIMEOUT_SECONDS
self.smart_budget_seconds = SMARTCTL_BUDGET_SECONDS
self.smart_cache_seconds = SMARTCTL_CACHE_SECONDS
@@ -917,8 +1020,19 @@ class Collector:
# -- proxy configs (nginx) ------------------------------------------------ #
def nginx(self, errors: list[str]) -> dict[str, Any]:
"""The `nginx` section on its own — `nginx_document()`'s first half."""
return self.nginx_document(errors)[0]
def nginx_document(self, errors: list[str]) -> tuple[dict[str, Any], str | None]:
"""Read and parse the host's nginx configs — never raises, never fails the snapshot.
Returns `(section, resolver directive)`. The second half is the only reason this method
is not the section itself: the `resolver` section prints nginx's own `resolver` line next
to `via: "libc"` so an operator can see the two vantages disagree, and taking it from the
text this pass already read is what keeps that free (no second file read per snapshot).
It deliberately does **not** go in the section: the `nginx` document stays byte-stable,
which is what makes this phase's additivity control a strict top-level key comparison.
The same three-way answer as `haproxy()`, for the other half of the gateway's ingress:
`true` the main config was read and understood, `false` there is no config at this path
(this host runs no nginx — one `nginx: ` line saying the file is **not found**), `null`
@@ -944,10 +1058,10 @@ class Collector:
section["available"] = False
errors.append(f"nginx: {config} not found — nginx is not installed here, or its "
"config lives elsewhere (--nginx-config); --no-proxy opts out")
return section
return section, None
except (OSError, ValueError) as exc:
errors.append(f"nginx: {config} is there but could not be read: {exc}")
return section
return section, None
# The two extra return values are already spoken about in `errors`; the section itself
# carries no counters, because "which files were read" is what `files` is for.
@@ -959,11 +1073,11 @@ class Collector:
f"parsed: {parsed['unparsed'][0]}")
if parsed["error"] is not None:
errors.append(f"nginx: {config} is there but could not be parsed: {parsed['error']}")
return section
return section, None
if not parsed["recognised"]:
errors.append(f"nginx: {config} holds no nginx context (http, stream, server, "
"upstream, map) — not an nginx config")
return section
return section, None
section["available"] = True
# The main config first: `files` is the answer to "where do these servers come from",
# and an operator comparing it with `nginx -t` should see the file nginx was started on.
@@ -971,7 +1085,7 @@ class Collector:
section["servers"] = parsed["servers"]
section["upstreams"] = parsed["upstreams"]
section["maps"] = parsed["maps"]
return section
return section, cast("str | None", parsed["resolver"])
def nginx_include_files(self, include_args: list[str],
errors: list[str]) -> tuple[dict[str, str], int, list[str]]:
@@ -1107,6 +1221,210 @@ class Collector:
section["available"] = True if answered else None
return section
# -- name resolution ------------------------------------------------------ #
def resolver(self, haproxy: Mapping[str, Any], nginx: Mapping[str, Any],
config_directive: str | None, errors: list[str]) -> dict[str, Any]:
"""Resolve the names the two proxy configs declare. Never raises, never fails a snapshot.
The point is that a route written as `server mc1 minecraft.reeselink.com:25565` ends at an
address, and the kernel's socket table speaks addresses: without this section the
dashboard can only count the connections whose target the config wrote *as an address*,
which on this ingress is 2 routes out of 33. The lookup happens here, on the box that
opens the connections, because that is the vantage the proxy itself resolves from; the
dashboard resolving the name through its own `resolv.conf` would answer a different
question (and `sources/gateway.py` is spied to fail if it ever tries).
`available` is the family's three-way answer, with **one** `resolver:` line per non-`true`
arm: `true` a pass ran, `false` there was nothing to resolve because neither proxy section
was read (the line says *not found* when the files simply are not here, which is the
wording the dashboard reads), `null` a pass that could not run. Configs that were read and
name no target are `true` with `entries: []` — that is absence, not a fault.
The cache is SMART's idiom, and for the SMART reason: the first pass runs inline so a
freshly started agent answers with real addresses, and every later expiry is refreshed on
a single-flight background thread while the request is served the previous result. A
resolver that has started black-holing must not cost the *system* poll its timeout over an
add-on that page never draws (AGENTS.md invariant 3/4).
"""
targets = target_names(haproxy, nginx)
if not targets and not _section_answered(haproxy) and not _section_answered(nginx):
# Nothing was read, so there is nothing to resolve. The two config sections have
# already said why in their own lines; this one says the resolver's own state, once.
line = resolver_not_found_line(haproxy, nginx)
errors.append(line)
return {"available": False, "via": RESOLVE_VIA, "config_directive": config_directive,
"ts": None, "entries": [], "errors": [line]}
now = time.monotonic()
with self._resolver_lock:
stamp = self._resolver_cache_ts
if stamp is not None:
if now - stamp >= self.resolve_cache_seconds and not self._resolver_refreshing:
self._resolver_refreshing = True
threading.Thread(target=self._refresh_resolver, args=(list(targets),),
name="resolver-refresh", daemon=True).start()
# Served from cache either way — the fresh-cache case and the "a refresh is
# already running" case — because waiting here is what this cache exists to avoid.
return self._resolver_section(self._resolver_cache, self._resolver_pass_ts,
self._resolver_errors, config_directive, errors)
entries, pass_errors = self.resolver_pass(targets)
return self._resolver_section(entries, self._resolver_pass_ts, pass_errors,
config_directive, errors)
def _resolver_section(self, entries: Mapping[str, dict[str, Any]], pass_ts: float | None,
pass_errors: Sequence[str], config_directive: str | None,
errors: list[str]) -> dict[str, Any]:
"""The section out of one pass's (or one cache's) result — the shape, in one place.
The lines go to **both** readers, because there are two: the snapshot's shared `errors`
list, which is what `collectors.ADDON_ERROR_PREFIXES` filters to decide the *system* page
is not degraded, and the section's own `errors`, which is the gateway page's sentence
beside the entries they belong to. Same strings, one append each.
"""
section: dict[str, Any] = {"available": True, "via": RESOLVE_VIA,
"config_directive": config_directive, "ts": pass_ts,
"entries": [dict(entry) for entry in entries.values()],
"errors": []}
for line in pass_errors:
errors.append(line)
section["errors"].append(line)
return section
def _refresh_resolver(self, targets: list[tuple[str, int | None]]) -> None:
"""The expired-cache pass, off the request path (see `resolver()`).
A failure keeps the previous entries **and** re-stamps the cache: the snapshot keeps
answering with the last truth it had, and the retry comes on the next expiry instead of
every poll against a resolver that just failed. The line is replaced rather than stacked,
because a refresh that fails every `resolve_cache_seconds` for a week is one fact, not
2000 — the repeated-failure rule the poller applies to its own events.
"""
try:
self.resolver_pass(targets)
except Exception as exc: # noqa: BLE001 - the thread must always clear its own flag
with self._resolver_lock:
line = (f"resolver: background refresh failed: {type(exc).__name__}: {exc}")
self._resolver_errors = [x for x in self._resolver_errors
if not x.startswith("resolver: background refresh failed")]
self._resolver_errors = [*self._resolver_errors, line]
self._resolver_cache_ts = time.monotonic()
finally:
with self._resolver_lock:
self._resolver_refreshing = False
def resolver_pass(self, targets: Sequence[tuple[str, int | None]]
) -> tuple[dict[str, dict[str, Any]], list[str]]:
"""One bounded libc lookup per declared target. Never raises out of this function.
Entries come back keyed by `resolver_key`, in target order, merged over the previous
cache so a pass that stopped at the budget keeps the entries it did not reach (each with
**its own** `ts`, so a stale address is visibly stale rather than freshly wrong). A name
that never got looked up at all and has no previous answer simply has no entry: the line
says how many were reached, and inventing an `error` status for a lookup that was never
attempted would be rule 2 with a status field on it.
"""
errors: list[str] = []
fresh: dict[str, dict[str, Any]] = {}
deadline = time.monotonic() + self.resolve_pass_budget_seconds
need_lookup = [pair for pair in targets if address_family(pair[0]) is None]
over_bound = len(need_lookup) - self.resolve_max_names
looked_up = 0
stopped = False
for name, port in targets:
# Literals first, and never counted against either bound: reporting one is a fact and
# dialling it would be a round trip to be told what the config already wrote, so a
# truncated pass must not be the reason an address the config spells is missing.
if address_family(name) is None:
continue
fresh[resolver_key(name, port)] = {"name": name, "port": port, "status": "literal",
"addresses": [name], "ts": time.time()}
for name, port in need_lookup:
if looked_up >= self.resolve_max_names:
continue # one line for the whole excess, after the loop
fresh[resolver_key(name, port)] = self._resolve_target(name, port)
looked_up += 1
if time.monotonic() >= deadline:
stopped = True
break
if stopped:
errors.append(f"resolver: pass budget {self.resolve_pass_budget_seconds:.1f}s "
f"exhausted, {looked_up} of {len(need_lookup)} targets looked up")
if over_bound > 0:
errors.append(f"resolver: {over_bound} of {len(need_lookup)} targets are beyond the "
f"{self.resolve_max_names}-name bound and were not looked up — the "
"section covers what it could resolve")
entries = self._store_resolver_cache(targets, fresh, errors)
return entries, errors
def _store_resolver_cache(self, targets: Sequence[tuple[str, int | None]],
fresh: Mapping[str, dict[str, Any]], errors: Sequence[str]
) -> dict[str, dict[str, Any]]:
"""Merge this pass into the cache in target order, and stamp it. Returns the entries.
Order is not cosmetics: `entries` is a list a page renders, so the same config must draw
in the same order on every poll. Names the config dropped leave the cache with it — they
are no longer declared targets, not targets that stopped resolving.
"""
merged: dict[str, dict[str, Any]] = {}
with self._resolver_lock:
previous = self._resolver_cache
for name, port in targets:
key = resolver_key(name, port)
entry = fresh.get(key) or previous.get(key)
if entry is not None:
merged[key] = entry
self._resolver_cache = merged
self._resolver_cache_ts = time.monotonic()
self._resolver_pass_ts = time.time()
# The pass's own lines replace the previous ones: a cache replay has to say the same
# thing the pass that produced it said (SMART's cache does the same), and a line from
# an older pass — a budget hit that has since stopped happening — would be a stale
# complaint about a reading that is now complete.
self._resolver_errors = list(errors)
return dict(merged)
def _resolve_target(self, name: str, port: int | None) -> dict[str, Any]:
"""One `getaddrinfo`, and the entry that says what it answered.
Both families are kept (`getaddrinfo` answers v4 and v6 when the host has both) because
the gateway connects over whichever one routing picks and the socket table speaks either.
A `gaierror` naming no-name/no-data is `nxdomain` — the name is answered and has no
address — and anything else is `error`, because "the server did not answer" and "the name
has no record" are different things to fix. Both carry `addresses: []`, never a stale
guess and never a `0`. The `ts` is taken here rather than passed in, so an entry says when
**it** was answered even in a pass that then stopped at the budget.
The keyword is `type=`, and that is load-bearing in a way only a real libc can teach:
`proto` is the **protocol number** (`IPPROTO_TCP` = 6) while `SOCK_STREAM` = 1 is a socket
*type*, so `proto=SOCK_STREAM` asks libc for a stream socket over ICMP and it answers
`EAI_SERVICE` for **every** name — a resolver that reports `error` on a healthy gateway,
with every route on the page uncounted, in a way no stubbed test can see because a stub
takes any keyword. What we want is libc's answer to "where do I connect a TCP socket",
which is exactly haproxy's `init-addr last,libc,none` question. Pinned by
`tests/test_agent.py` with no stub in sight, against an address libc parses by itself.
"""
try:
infos = socket.getaddrinfo(name, port, type=socket.SOCK_STREAM)
except socket.gaierror as exc:
code = exc.errno if isinstance(exc.errno, int) else 0
return {"name": name, "port": port,
"status": "nxdomain" if code in RESOLVE_NXDOMAIN_CODES else "error",
"addresses": [], "ts": time.time()}
except OSError as exc: # a resolver that fails as an I/O error, not as a DNS one
return {"name": name, "port": port, "status": "error", "addresses": [],
"detail": str(exc), "ts": time.time()}
except Exception as exc: # noqa: BLE001 - the promise is that this function never raises
# `getaddrinfo` is the one call in this agent that leaves the host, and whatever it
# does — a platform-specific `ValueError` for a name the C library cannot take, a
# stub in a test — the answer is one entry that says "error" beside the targets that
# answered, never a half-snapshot and never the *system* page going red over an
# add-on it does not draw (AGENTS.md invariant 4).
return {"name": name, "port": port, "status": "error", "addresses": [],
"detail": f"{type(exc).__name__}: {exc}", "ts": time.time()}
addresses = sort_resolver_addresses(str(info[4][0]) for info in infos)
# An empty answer list is libc saying the name resolved to nothing: no-data, not a fault.
return {"name": name, "port": port, "status": "ok" if addresses else "nxdomain",
"addresses": addresses, "ts": time.time()}
# -- network ------------------------------------------------------------- #
def network(self) -> dict[str, Any]:
interfaces: list[dict[str, Any]] = []
@@ -1479,11 +1797,24 @@ class Collector:
document["haproxy"] = self._safe(lambda: self.haproxy(errors), "haproxy", errors) \
or {"available": None, "config": self.haproxy_config,
"frontends": [], "backends": []}
document["nginx"] = self._safe(lambda: self.nginx(errors), "nginx", errors) \
or {"available": None, "config": self.nginx_config, "files": [],
"servers": [], "upstreams": [], "maps": []}
nginx_pair = self._safe(lambda: self.nginx_document(errors), "nginx", errors)
nginx_section, nginx_resolver = nginx_pair or (
{"available": None, "config": self.nginx_config, "files": [],
"servers": [], "upstreams": [], "maps": []}, None)
document["nginx"] = nginx_section
document["sockets"] = self._safe(lambda: self.sockets(errors), "sockets", errors) \
or {"available": None, "listens": [], "established": []}
# The resolver rides with the family and is removed by both flags: `--no-resolve` is
# the operator turning off the one add-on that leaves the host, and `--no-proxy` means
# there are no declared names to resolve in the first place (decision 3). Absent, not
# empty — "not collected" stays a different fact from "collected and empty".
if self.resolve_enabled:
document["resolver"] = self._safe(
lambda: self.resolver(cast("Mapping[str, Any]", document["haproxy"]),
cast("Mapping[str, Any]", nginx_section),
nginx_resolver, errors), "resolver", errors) \
or {"available": None, "via": RESOLVE_VIA, "config_directive": None,
"ts": None, "entries": [], "errors": []}
document["errors"] = errors
return document
@@ -2038,13 +2369,15 @@ def parse_nginx_config(main_text: str,
`{servers: [{listen: [{addr, port}], server_names: [str], proxy_passes: [{location, scheme,
target, variable}]}], upstreams: [{name, servers: [{host, port}]}],
maps: [{variable, entries: [{key, target}]}], files: [str], includes: [str], unparsed:
[str], recognised: bool, error: str|null}`.
[str], recognised: bool, error: str|null, resolver: str|null}`.
Three of those keys are the ones `Collector.nginx` does not serve, the way
Four of those keys are the ones `Collector.nginx` does not serve, the way
`parse_haproxy_config`'s `recognised` is not: `error` is why the main text could not be
parsed, `unparsed` names an included document that could not, and `includes` is the raw
parsed, `unparsed` names an included document that could not, `includes` is the raw
argument list (a caller that has no filesystem — every unit test — still gets to see what a
config *would* have read). `files` is the included documents that were merged, in order;
config *would* have read), and `resolver` is the config's own `resolver` directive's
arguments, which the `resolver` section serves instead (`Collector.nginx_document`).
`files` is the included documents that were merged, in order;
the collector puts the main config in front of it to make the section's `files`.
A document that fails to parse is dropped and named in `unparsed`; it never drops the others
@@ -2053,13 +2386,18 @@ def parse_nginx_config(main_text: str,
`error` set, not an exception on a snapshot path.
"""
out: dict[str, Any] = {"servers": [], "upstreams": [], "maps": [], "files": [],
"includes": [], "unparsed": [], "recognised": False, "error": None}
"includes": [], "unparsed": [], "recognised": False, "error": None,
"resolver": None}
try:
main = _nginx_statements(main_text)
except ValueError as exc:
out["error"] = str(exc)
return out
out["recognised"] = any(statement[1] in NGINX_CONTEXTS for statement in main)
# The config's own `resolver` line, for the `resolver` section's `config_directive`. It is
# found here rather than served here on purpose: the `nginx` section stays byte-stable, so
# this phase's additivity control is a strict top-level key comparison with no exception list.
out["resolver"] = _find_resolver_statement(main)
_collect_nginx(main, out, root=True)
documents = included or {}
merged: list[str] = []
@@ -2075,6 +2413,8 @@ def parse_nginx_config(main_text: str,
out["unparsed"].append(path)
continue
merged.append(path)
if out["resolver"] is None:
out["resolver"] = _find_resolver_statement(statements)
# root=False: an include line inside an included file is a second level, and this
# pass is one level deep by design (see NGINX_INCLUDE_MAX_FILES).
_collect_nginx(statements, out, root=False)
@@ -2197,6 +2537,165 @@ def socket_truncation_line(paths: list[str]) -> str:
"read takes — the sockets past the bound are not in this section")
# --------------------------------------------------------------------------- #
# name resolution (the additive `resolver` section)
# --------------------------------------------------------------------------- #
def address_family(value: str) -> str | None:
"""`v4`, `v6`, or None — is this text already an address?
`socket.inet_pton` and nothing else: it is the same test the `sockets` section's decoder
uses, it needs no second module (the agent's import footprint is pinned by
`tests/test_agent.py`), and it is the rule that lets an IP literal in a config be reported
as what it is instead of dialled. An interface scope (`fe80::1%eth0`) is not a pure address,
so it comes back None and travels through the resolver like any other name — libc decides
what a scoped literal means, and it is not this file's job to invent an answer.
"""
for family, label in ((socket.AF_INET, "v4"), (socket.AF_INET6, "v6")):
try:
socket.inet_pton(family, value)
except (OSError, ValueError):
continue
return label
return None
def resolver_key(name: str, port: int | None) -> str:
"""One declared target, as this module's cache key.
The port belongs in the key because a name on two ports is two targets: this fleet routes
`minecraft.reeselink.com` to 25565 and to 25566, two services, and one entry for both would
be one answer to a question nobody asked.
"""
return f"{name}:{port}"
def _resolver_sort_key(value: str) -> tuple[int, bytes]:
"""v4 ascending then v6 ascending, with anything undecodable last and stable.
The order matters for the payload rather than for the lookup: `getaddrinfo` returns whatever
order this host's resolver and `/etc/gai.conf` feel like today, so an unsorted set makes two
polls of a healthy gateway differ for no reason the operator could act on.
"""
for family, rank in ((socket.AF_INET, 0), (socket.AF_INET6, 1)):
try:
return (rank, socket.inet_pton(family, value))
except (OSError, ValueError):
continue
return (2, value.encode("utf-8", errors="replace"))
def sort_resolver_addresses(values: Iterable[str]) -> list[str]:
"""Unique addresses in the payload's order: family first, then the address itself."""
return sorted(set(values), key=_resolver_sort_key)
def target_names(haproxy: Mapping[str, Any] | None,
nginx: Mapping[str, Any] | None) -> list[tuple[str, int | None]]:
"""Every `(name, port)` the two **parsed** proxy configs write as a target, no duplicates.
This is the whole allow-list for the resolver (decision 3 of the phase): haproxy's
`backend`/`server` targets first (config order), then nginx's `upstream` members, its
server-block `proxy_pass` targets, and its `map` entry targets, deduplicated keeping
first-seen order. Nothing else is ever in here — not a name from `/proc/net/tcp`, not a
client address, not anything a request supplied — because resolving a name nobody configured
is a round trip that produces a fact about the internet instead of a fact about this
gateway, and `tests/test_agent.py` pins that the lookups are exactly this list.
A `proxy_pass $var` contributes nothing (the map decides, and the map's own entries are
here); an entry whose target is not an address at all is `target: null` upstream and is
skipped here for the same reason the `nginx` section refused to name it.
"""
targets: list[tuple[str, int | None]] = []
seen: set[tuple[str, int | None]] = set()
def add(host: Any, port: Any) -> None:
if not isinstance(host, str) or not host:
return
pair = (host, port if isinstance(port, int) and not isinstance(port, bool) else None)
if pair in seen:
return
seen.add(pair)
targets.append(pair)
for block in _mapping_list(haproxy, "backends"):
for server in _mapping_list(block, "servers"):
add(server.get("target"), server.get("port"))
for upstream in _mapping_list(nginx, "upstreams"):
for member in _mapping_list(upstream, "servers"):
add(member.get("host"), member.get("port"))
for server in _mapping_list(nginx, "servers"):
for hop in _mapping_list(server, "proxy_passes"):
target = hop.get("target")
if isinstance(target, Mapping):
add(target.get("host"), target.get("port"))
for block in _mapping_list(nginx, "maps"):
for entry in _mapping_list(block, "entries"):
target = entry.get("target")
if isinstance(target, Mapping):
add(target.get("host"), target.get("port"))
return targets
def _mapping_list(section: Any, key: str) -> list[Mapping[str, Any]]:
"""One list of mappings out of a section that may be absent, null, or the wrong shape.
The section is built by this file, so anything else means a caller handed this function
something it did not build — and the honest answer to that is an empty list, not a
`TypeError` on a snapshot path.
"""
if not isinstance(section, Mapping):
return []
rows = section.get(key)
if not isinstance(rows, list):
return []
return [row for row in rows if isinstance(row, Mapping)]
def _section_answered(section: Any) -> bool:
"""Did this proxy section report that it read its config? `available` is the section's own answer."""
return isinstance(section, Mapping) and section.get("available") is True
def resolver_not_found_line(haproxy: Any, nginx: Any) -> str:
"""The one `resolver:` line for the arm where there was nothing to resolve.
It says *not found* when a config is simply absent — the pinned cross-module wording the
dashboard reads to answer "is this proxy installed here?" (the `haproxy:`, `nginx:` and
`smartctl:` lines share it) — and *unreadable* when the file is there and this process could
not read it, because those are two different operator actions. One line either way: the two
sections have already reported their own reasons, and a third restatement per snapshot is the
ink the empty-state rules exist to avoid.
"""
reasons = []
for name, section in (("haproxy", haproxy), ("nginx", nginx)):
available = section.get("available") if isinstance(section, Mapping) else None
reasons.append(f"{name} not found" if available is False else f"{name} unreadable")
return (f"resolver: nothing to resolve — {' and '.join(reasons)}; --no-resolve opts out")
def _find_resolver_statement(statements: _NginxStatements) -> str | None:
"""Depth-first search for one `resolver` directive, in document order.
A `map` body is not searched: inside one, a bare word is a **key**, so a map with a
`resolver` key (a header name, a hostname someone routed on) would otherwise be reported as
this box's DNS server.
"""
for statement in statements:
kind, name, args = statement[0], statement[1], statement[2]
if kind == "d":
if name == "resolver" and args:
return " ".join(str(arg) for arg in args)
continue
if name == "map":
continue
found = _find_resolver_statement(cast("_NginxStatements", statement[3]))
if found is not None:
return found
return None
# --------------------------------------------------------------------------- #
# TLS bundle: what the dashboard signs, what the installer places
# --------------------------------------------------------------------------- #
@@ -2663,8 +3162,12 @@ def main(argv: list[str] | None = None) -> int:
help="skip the smartctl pass entirely (disks report smart: null, "
"no error entries)")
parser.add_argument("--no-proxy", action="store_true",
help="skip the proxy add-on entirely (no `haproxy`, `nginx` or "
"`sockets` key, no error entries)")
help="skip the proxy add-on entirely (no `haproxy`, `nginx`, `sockets` "
"or `resolver` key, no error entries)")
parser.add_argument("--no-resolve", action="store_true",
help="skip the resolver pass (no `resolver` key, no error entries): the "
"two config sections and the socket table are still read, only the "
"name lookups are not")
parser.add_argument("--haproxy-config", default=HAPROXY_CONFIG_PATH,
help="haproxy config to read; a missing file is the documented "
"degradation (available=false, one error entry), not a failure")
@@ -2679,7 +3182,8 @@ def main(argv: list[str] | None = None) -> int:
collector = Collector(args.root, top_processes=args.top, net_all=args.net_all,
smartctl=args.smartctl, smart=not args.no_smart,
proxy=not args.no_proxy, haproxy_config=args.haproxy_config,
proxy=not args.no_proxy, resolve=not args.no_resolve,
haproxy_config=args.haproxy_config,
nginx_config=args.nginx_config)
if args.once:
print(json.dumps({"ok": True, **collector.snapshot()}, indent=2, default=str))