services/dashboards
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
51772ffef2361d8f2fe2a87d07ae58475d45538d
dashboards/agent/host_agent.py
T
ducoterra 51772ffef2 phase: 32_gateway_resolved_target_liveness 
Fixture and live stream.conf are byte-identical (modulo header comment). Let me test the parser directly:
2026-10-26 14:03:42 -04:00

3212 lines
172 KiB
Python
Raw Blame History

#!/usr/bin/env python3
"""host_agent.py — read-only host metrics agent for the homelab dashboards.
**Stdlib only, one file, Python 3.10+.** Copy it to a host, run it, and the dashboard can
read that host. It never writes anything anywhere: every number comes from reading
/proc, /sys and statvfs, and the HTTP server only answers GET.
GET /health -> {"ok": true, "hostname": ..., "version": ...}
GET /info -> slow-changing identity (os, kernel, cpu model, cores, gpu list)
GET /snapshot -> everything, one JSON document (see SCHEMA below)
GET /schema -> this docstring's SCHEMA section, for self-documenting deploys
Design rules
------------
* **Counters stay counters.** The agent reports cumulative values (jiffies, bytes, ops)
and the *dashboard* computes rates with a labelled window. A stateless agent cannot
lie about an interval, survive a restart silently, or be polled twice by two dashboards
and halve its own rate.
* **Absent is null, never 0.** A host without a GPU, or a sysfs file only root can read,
yields `null`. The UI renders `n/a`. Turning "unknown" into 0 is the one mistake a
monitoring agent is not allowed to make.
* **Every read is guarded.** A vanished /proc entry (process exited mid-scan) skips that
item; the snapshot still succeeds. The poll is degraded, never failed.
* **SMART is a best-effort add-on.** The per-disk `smart` block runs `smartctl -i -A -H`
in a subprocess: bounded 5 s per device and a pass budget of 20 s scaled to the disk
count, cached 240 s so a fast dashboard poll cannot re-issue ATA commands every few
seconds. Refreshing an expired
cache happens on a background thread while the request is served the previous result —
20 s of wedged disk must never cost a poll more than its own timeout. No binary, a denied
device or a wedged disk degrades *that disk's* `smart` to `null` plus one `errors`
entry (a missing binary produces exactly one entry for the whole pass) and never
fails the snapshot or `/health`. Absence is a documented state, not a red panel:
install `smartmontools` and give the agent user read access to the block devices
(typically the `disk` group), or pass `--no-smart` to opt out silently.
* **The proxy configs are a best-effort add-on too, with the same three-way answer.** The
`haproxy` section reads the host's haproxy config (`--haproxy-config`, default
/etc/haproxy/haproxy.cfg) and reports the `frontend` blocks (name, each `bind`
address:port, `default_backend`/`use_backend` references) and the `backend` blocks (name,
each `server` line's name and target host:port). The `nginx` section reads the main nginx
config (`--nginx-config`, default /etc/nginx/nginx.conf) **plus the files its own `include`
lines name, one level deep** (`include /etc/nginx/conf.d/*.conf`), and reports the `server`
blocks (each `listen` address:port, `server_name`, every `proxy_pass`), the `upstream`
blocks and the `map` blocks — a `proxy_pass $var` keeps the variable's name and a
`target: null`, because nginx picks that address per connection from the map and a merged
answer would be a number this wire never published. `available` answers three questions, the
way SMART's `smartctl_available` does: `true` the file was read and at least one block
parsed, `false` there is no such file (this host does not run haproxy or nginx — one `errors`
line for the whole pass, saying the file is missing), `null` a file is there but could not be
read or understood (also one line). Whichever arm fired, the snapshot and `/health` are
unaffected and the other sections are intact; `--no-proxy` removes the sections with no
errors line at all, exactly as `--no-smart` leaves every disk's `smart` null in silence.
A host with no proxy on it is unconfigured, not broken.
* **The live socket table** is the `sockets` section: `/proc/net/tcp` and `/proc/net/tcp6`,
read at source and reported as the two states a gateway page can draw pipes from — `0A`
LISTEN (the frontends that are open) and `01` ESTABLISHED (the pairs flowing right now).
Every other state is a socket in transit and is skipped in silence, because an ocean of
`TIME_WAIT` rows would bury the pairs the animation shows. Each table is read through
**at most** `PROC_NET_MAX_LINES` lines (4096, `/proc`'s own header among them, so 4095
sockets per table): a busy proxy writes tens of thousands of rows and a snapshot that
copied them every poll would spend the request on a list no page can draw. A longer table
reports the rows it read and says it was truncated; `available` stays `true`. Addresses are
unrolled from `/proc`'s per-32-bit-word little-endian spelling into the address itself
(`0.0.0.0` and `::` keep their literal wildcard form, because the wildcard *is* the fact the
dashboard's inbound join keys on), and there is **no join with the config here**: a socket
does not know its domain or its backend, so phase 29's page joins these ports to the
`haproxy`/`nginx` sections instead. It is best-effort the way the configs are: neither table
answering is `available: null` with one `sockets:` line and an otherwise intact snapshot,
and `--no-proxy` removes it with the rest of the family, because a box with no proxy has
nothing to watch from either side.
* **Names become addresses where they are actually dialled.** The `resolver` section takes every
`(name, port)` the two parsed configs write as a target — haproxy `server` lines, nginx
`upstream` members, `proxy_pass` targets and `map` entry targets, and **nothing else, ever**
(not a name from `/proc`, not a client address) — and asks libc what each one is:
`getaddrinfo` over `SOCK_STREAM`, which is the resolution haproxy's own libc path performs *on
this box*. That is the point — the dashboard resolving the same name through its own
`resolv.conf` would be a different vantage answering a different question — so `via: "libc"`
and the nginx config's own `resolver` line (`config_directive`) are both published and the
dashboard resolves nothing. An address literal in a config is `status: "literal"` with no
lookup (it is already an address, and dialling it to be told so is a round trip); a name that
does not answer is `nxdomain`/`error` with `addresses: []`, never a stale guess and never a 0.
The pass is SMART's shape for SMART's reason: at most `RESOLVE_MAX_NAMES` lookups inside
`RESOLVE_PASS_BUDGET_SECONDS` of wall clock, cached `RESOLVE_CACHE_SECONDS`, the **first** pass
inline and every later expiry refreshed on one single-flight background thread while the
request is served the previous result — a black-holing resolver must never blow the dashboard's
`HOST_TIMEOUT_SECONDS` and blink the *system* page over an add-on it does not draw. Each entry
carries the time **it** was answered, so a cache-served entry cannot pretend to be current.
`available` is the family's three-way answer (`true` a pass ran, `false` nothing to resolve
because no proxy config was read — one `resolver:` line, saying **not found** when the files
are simply absent — `null` a pass that could not run), and `--no-resolve` removes the key in
silence the way `--no-smart` does; `--no-proxy` removes it too, because a box that collects no
proxy has no declared names to look up.
* **The state** is a small pid->cpu-time cache so `processes.top_cpu` can answer a
question /proc cannot (CPU needs two samples), bounded and cleared when stale, plus
the short-TTL SMART result cache above.
SCHEMA (GET /snapshot)
----------------------
schema int -> 1
ts float unix seconds (agent clock)
duration_ms float collection cost
agent_version str
tls true only when serving with --cert-bundle
bundle_rev int the `# BUNDLE-REV` of the bundle it loaded
host {hostname, os_pretty, kernel, arch, python, boot_ts, uptime_seconds,
cpu_model, cpu_cores_logical, cpu_cores_physical, container}
cpu {modes: {user,nice,system,idle,iowait,irq,softirq,steal,guest,
guest_nice,total} [clock ticks, cumulative],
cores: [{core, modes: {...}}],
load1, load5, load15, procs_running, procs_blocked,
ctxt_total, processes_total,
freq_mhz, freq_max_mhz}
memory {total_bytes, available_bytes, used_bytes, free_bytes, buffers_bytes,
cached_bytes, dirty_bytes, swap_total_bytes, swap_used_bytes,
committed_bytes}
filesystems [{device, mountpoint, fstype, total_bytes, used_bytes, free_bytes,
used_pct, read_only}]
disk_io {devices: [{name, model, rotational, size_bytes, reads_total,
writes_total, read_bytes_total, write_bytes_total,
in_flight, io_time_ms_total, weighted_io_ms_total,
discard_bytes_total,
smart: {model, serial, overall_health, temp_c,
power_on_hours, reallocated_sector_ct,
current_pending_sector_ct, offline_uncorrectable,
percentage_used} | null}],
device_names: [str]} # whole disks from /sys/block, no partitions
# `smart` is `smartctl -i -A -H` parsed; null with an errors[] entry
# when the binary is missing or that device cannot be read. Fields
# the drive does not publish stay null (e.g. NVMe has no
# reallocated_sector_ct), and NVMe `offline_uncorrectable` carries
# the health log's "Media and Data Integrity Errors" counter.
network {interfaces: [{name, rx_bytes_total, tx_bytes_total, rx_packets_total,
tx_packets_total, rx_errors_total, tx_errors_total,
rx_dropped_total, tx_dropped_total, speed_mbps}]}
gpus [{index, card, vendor, model, pci_slot, pci_address, driver, busy_pct,
vram_total_bytes, vram_used_bytes, gtt_total_bytes, gtt_used_bytes,
power_watts, temps: {label: celsius}, clocks: {label: mhz}}]
temperatures [{kind: hwmon|thermal, chip, label, celsius}]
processes {total, top_rss: [{pid, name, rss_bytes}],
top_cpu: [{pid, name, cpu_pct}]}
haproxy {available: true|false|null, config: str, frontends: [{name, binds:
[{addr, port}], default_backend: str|null, backends: [str]}],
backends: [{name, servers: [{name, target, port}]}]}
# parsed from --haproxy-config; additive (phase 28) and absent entirely
# with --no-proxy. `addr` is the bind's address as written (`:::22` ->
# `::`), `port` null when a target names none (never 0), and the
# frontend->backend routing is the names themselves: the dashboard
# joins them, the agent reports only what the config says.
nginx {available: true|false|null, config: str, files: [str],
servers: [{listen: [{addr, port}], server_names: [str], proxy_passes:
[{location: str|null, scheme: str|null,
target: {host, port}|null, variable: str|null}]}],
upstreams: [{name, servers: [{host, port}]}],
maps: [{variable, entries: [{key, target: {host, port}|null}]}]}
# parsed from --nginx-config plus the files its own `include` lines name
# (one level, at most 8 files / 1 MB of included text; `files` is the
# main config then every include actually merged); additive (phase 28)
# and absent entirely with --no-proxy. `listen` keeps the address as
# written (`[::]:8443` -> `"::"`, a bare `80` -> a null addr, never `*`),
# `server_names` keeps the catch-all `_` verbatim, `location` is null for
# a proxy_pass written in the server block itself (the stream form), and
# a `proxy_pass $var` is served with `variable` (the `$` dropped) and
# `target: null`: the map whose variable matches decides per connection,
# so both facts travel unmerged and the dashboard joins them.
sockets {available: true|null, listens: [{family, addr, port}],
established: [{family, local: {addr, port}, remote: {addr, port}}]}
# /proc/net/tcp and /proc/net/tcp6 read at source: kernel state 0A LISTEN
# into `listens`, 01 ESTABLISHED into `established`, every other state
# skipped (a socket in transit is not a pipe); additive (phase 28) and
# absent entirely with --no-proxy. Each table is read through at most
# PROC_NET_MAX_LINES (4096) lines including its own header, so a longer
# one is reported as read-with-truncation in one errors line with
# `available` still true. `family` is "v4"/"v6" and `addr` is that
# family's address with /proc's per-word little-endian spelling unrolled
# (`0.0.0.0` and `::` keep the literal wildcard form the inbound join
# keys on); `addr` is null for a word that is not an address and `port`
# null for a word that is not a number, never 0. `available` is true when
# either table answered and null when neither did, with one `sockets:`
# line saying why: there is no `false` arm, because a table that did not
# answer is "no answer" and not "this host has no sockets", and
# --no-proxy already reports "did not look" by removing the key. No
# config join: a socket does not know its domain, so the gateway page
# joins these ports to `haproxy` and `nginx` itself.
resolver {available: true|false|null, via: "libc", config_directive: str|null,
ts: float|null, entries: [{name, port, status: ok|nxdomain|error|literal,
addresses: [str], ts: float}], errors: [str]}
# one libc lookup per (name, port) the two sections above write as a target,
# performed here because this is the box the proxy dials from; additive
# (phase 32) and absent entirely with --no-resolve or --no-proxy. `via` is
# how the addresses were obtained (getaddrinfo — haproxy's libc path) and
# `config_directive` is the nginx config's own `resolver` line kept verbatim
# (`"10.1.0.1"` here), so the two vantages read side by side and a
# divergence is the operator's finding rather than a second resolver here.
# `ts` is the wall clock of the pass that produced the entries while each
# entry carries the time *it* was answered, because an expired cache is
# served as it was while a background thread refreshes it: never two passes
# per snapshot, never a request that waits on DNS past the first one.
# `addresses` is the unique answer sorted v4-ascending then v6-ascending so
# two polls of a healthy gateway are byte-identical; `status: literal` means
# the config already wrote an address and no lookup was made, and
# `nxdomain`/`error` mean "no address right now" with `addresses: []` — the
# page draws those routes with no count, never with a 0. `available` is
# false only when there was nothing to resolve because no proxy config was
# read (one `resolver:` line, saying **not found** when the files are simply
# absent), and null for a pass that could not run; configs that were read
# and name no target are true with `entries: []`. The lookups are bounded
# (RESOLVE_MAX_NAMES, RESOLVE_PASS_BUDGET_SECONDS) and the excess says so in
# one line, so no resolver however broken holds /snapshot past its timeout.
errors [str] reads that failed; non-empty means the snapshot is partial
Deployment: see deploy/install-agent.sh (systemd unit, binds 127.0.0.1 unless you pass
--host, requires a bearer token via --token or AGENT_TOKEN when bound off-loopback). The
other protection is `--cert-bundle`/AGENT_CERT_BUNDLE: one installed PEM (server cert, server
key, the dashboard's root CA, `# BUNDLE-REV <n>` and `# REVOKED <serial>` lines) turns the
server into HTTPS that answers every path but `/health` only to a client certificate signed
by that root, and refuses one on the denylist on every path — the certificate is the auth, so
no token is read on that path.
"""
from __future__ import annotations
import argparse
import contextlib
import fnmatch
import glob
import json
import os
import re
import shutil
import socket
import ssl
import subprocess
import tempfile
import threading
import time
from collections.abc import Callable, Iterable, Mapping, Sequence
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from typing import Any, cast
AGENT_VERSION = "1.3.0"
SCHEMA_VERSION = 1
PAGE_SIZE = 0 # resolved from sysconf at runtime
# Filesystems that are not storage. Mirrors the spirit of node_exporter's default
# --collector.filesystem.fs-types-exclude, trimmed to what actually appears here.
PSEUDO_FSTYPES = {
"autofs", "binfmt_misc", "bpf", "cgroup", "cgroup2", "configfs", "debugfs", "devpts",
"devtmpfs", "efivarfs", "fuse.gvfsd-fuse", "fusectl", "hugetlbfs", "iso9660", "mqueue",
"nsfs", "overlay", "proc", "procfs", "pstore", "ramfs", "rpc_pipefs", "securityfs",
"selinuxfs", "squashfs", "sysfs", "tmpfs", "tracefs", "fuse.lxcfs", "binfmt",
}
PSEUDO_MOUNT_PREFIXES = ("/proc", "/sys", "/dev", "/run", "/var/lib/containers/storage",
"/var/lib/docker", "/snap", "/boot/grub2", "/mnt/iso")
# Interface name prefixes that are containers/bridges talking to themselves.
VIRTUAL_NET_PREFIXES = ("lo", "veth", "docker", "br-", "virbr", "br", "tap", "vnet",
"cni", "kube", "flannel", "dummy", "ifb", "sit", "teql")
# /sys/block entries that are not physical media. dm-* is deliberately here too: an LVM
# or LUKS volume is carved out of a disk we already count, so charting both double-counts.
VIRTUAL_DISK_PREFIXES = ("loop", "ram", "zram", "sr", "dm-", "md127")
# SMART collection bounds. 5 s per device is generous for a healthy drive and caps how
# long one wedged device can hold a snapshot hostage; the pass budget is a 20 s floor
# scaled to the per-device bound (5 s x disk count), so a 16-disk shelf gets the 80 s a
# real read takes while a single-disk host keeps 20 s and a shelf of bad disks cannot
# stall the HTTP handler; 240 s is below the dashboard's 5-minute disks cadence, so the
# slow disks job gets fresh data while the fast system poll never re-runs smartctl
# every 10 s.
SMARTCTL_TIMEOUT_SECONDS = 5.0
SMARTCTL_BUDGET_SECONDS = 20.0
SMARTCTL_CACHE_SECONDS = 240.0
# The smart block's field set — the shape the `/disks` page (phase 09) consumes. A field
# the drive does not publish stays None; the page renders `n/a`, never 0.
SMART_FIELDS = ("model", "serial", "overall_health", "temp_c", "power_on_hours",
"reallocated_sector_ct", "current_pending_sector_ct",
"offline_uncorrectable", "percentage_used")
_SMART_HEALTH_RE = re.compile(r"^SMART overall-health self-assessment test result:\s*(\S+)")
# The number a SMART attribute RAW_VALUE column leads with. Captured from the deployed
# smartmontools 7.5 on this fleet: a Seagate IronWolf writes `35 (Min/Max 34/44)`, a
# Western Digital writes its packed hours as `43894h+43m+32.519s`, and most drives write a
# bare `30423`. In every shape the drive's own figure is the leading decimal group.
_SMART_RAW_NUMBER_RE = re.compile(r"^\d[\d,]*(?:\.\d+)?")
# ATA SMART attribute table rows, keyed by the exact attribute names the deployed
# smartmontools 7.5 prints (verified against its own drive database, not from memory).
_SMART_ATTR_FIELDS = {
"Temperature_Celsius": "temp_c",
"Airflow_Temperature_Cel": "temp_c",
"Power_On_Hours": "power_on_hours",
"Reallocated_Sector_Ct": "reallocated_sector_ct",
"Current_Pending_Sector": "current_pending_sector_ct",
"Offline_Uncorrectable": "offline_uncorrectable",
"Percent_Lifetime_Used": "percentage_used",
"Percent_Life_Used": "percentage_used",
"Percentage_Lifetime_Use": "percentage_used",
}
# --------------------------------------------------------------------------- #
# haproxy config (the additive `haproxy` section)
# --------------------------------------------------------------------------- #
#: Where haproxy keeps its config on this fleet's gateway host, and the default for
#: `--haproxy-config`. A host that does not run haproxy has no file at this path, which is
#: the `available: false` arm rather than a fault worth more than one `errors` line.
HAPROXY_CONFIG_PATH = "/etc/haproxy/haproxy.cfg"
#: A haproxy config is a few kilobytes of hand-written text, and this file is read on every
#: snapshot. The bound is what keeps a `--haproxy-config` pointed at a device node or a log
#: from costing the request a read that never ends — the lesson the SMART bounds exist for.
HAPROXY_CONFIG_MAX_BYTES = 1_000_000
#: The words that open a haproxy *section*. Used only to tell "a config with no proxies"
#: apart from "text that is not a haproxy config": a file whose top level names none of
#: these is the second fact, and the section says so instead of serving an empty `true`.
HAPROXY_KEYWORDS = frozenset({
"global", "defaults", "frontend", "backend", "listen", "resolvers", "peers",
"mailers", "userlist", "cache", "ring", "programmatic", "extend",
})
# --------------------------------------------------------------------------- #
# nginx config (the additive `nginx` section)
# --------------------------------------------------------------------------- #
#: Where nginx keeps its main config on this fleet's gateway host, and the default for
#: `--nginx-config`. A host that does not run nginx has no file at this path, which is the
#: `available: false` arm rather than a fault worth more than one `errors` line.
NGINX_CONFIG_PATH = "/etc/nginx/nginx.conf"
#: The main config is read on every snapshot, so it gets the same size bound the haproxy one
#: has: a `--nginx-config` aimed at a device node must not cost the request a long read.
NGINX_CONFIG_MAX_BYTES = 1_000_000
#: Include-following is **one level and bounded**, which is what makes the section cheap
#: enough to read on every snapshot. The captured gateway tree is one level deep
#: (`nginx.conf` -> `conf.d/*.conf` + `stream.d/*.conf`), so 8 files covers a host that
#: splits one vhost per file with room to spare, and 1 MB of included text is two orders of
#: magnitude above the 10 KB this fleet's whole ingress occupies. Exceeding a bound drops the
#: excess with one `errors` line naming how much was left out — the SMART pass-budget shape:
#: the pass reports what it did not cover instead of quietly serving a shorter list.
NGINX_INCLUDE_MAX_FILES = 8
NGINX_INCLUDE_MAX_BYTES = 1_000_000
#: The words that open an nginx *context* or name an unmistakably main-level directive. Used
#: only to tell "a config with no proxies" apart from "text that is not an nginx config": a
#: file whose top level names none of these is the second fact, and the section says so
#: instead of serving an empty `true`.
NGINX_CONTEXTS = frozenset({
"http", "stream", "mail", "events", "server", "upstream", "map", "geo", "types",
"match", "split_clients", "limit_req_zone", "limit_conn_zone", "perl", "js_import",
"user", "worker_processes", "worker_cpu_affinity", "worker_rlimit_nofile", "env",
"include", "load_module", "pid", "error_log", "thread_pool", "pcre_jit",
})
#: The characters that make an `include` argument a glob rather than a path. nginx uses the
#: C library's `glob()`; `glob.glob` is the stdlib's, and this is the same test it applies.
_GLOB_SPECIALS = "*?["
# --------------------------------------------------------------------------- #
# live sockets (the additive `sockets` section)
# --------------------------------------------------------------------------- #
#: The two kernel tables the section reads, each with the family label its rows carry. They
#: are read independently because a kernel built without IPv6 has no `/proc/net/tcp6` at all:
#: one table answering is the host telling us what it has, not a failure.
PROC_NET_TABLES: tuple[tuple[str, str], ...] = (("v4", "proc/net/tcp"),
("v6", "proc/net/tcp6"))
#: The read bound, in lines **per table**, `/proc`'s own header among them — so a truncated
#: pass reports at most `PROC_NET_MAX_LINES - 1` sockets. A gateway forwards every connection
#: its owner makes, and `/proc/net/tcp` is one line per socket including the `TIME_WAIT` ones
#: nobody can see: the bound is the cost of the section, not a display choice. The excess is
#: reported as what it is in one `errors` line (the SMART pass-budget and nginx include-bound
#: shape) instead of quietly serving a shorter list.
PROC_NET_MAX_LINES = 4096
#: The two TCP states the gateway page draws pipes from, as `/proc` spells them: `0A` LISTEN
#: (an open frontend) and `01` ESTABLISHED (a pair flowing right now). Every other state is a
#: socket in transit — `TIME_WAIT`, `CLOSE_WAIT`, the SYN queue — and an ocean of those would
#: bury the pairs the animation shows, so they are skipped without a word.
PROC_NET_STATE_LISTEN = "0A"
PROC_NET_STATE_ESTABLISHED = "01"
# --------------------------------------------------------------------------- #
# name resolution (the additive `resolver` section)
# --------------------------------------------------------------------------- #
#: How many declared targets one pass looks up. A gateway's config-declared targets are a few
#: dozen with room to grow, and the bound is what stops a config that grew into the hundreds (or
#: a map written by someone who liked wildcards) from turning a snapshot into a DNS storm. The
#: excess is reported in one `errors` line naming the count — the nginx include-bound and SMART
#: pass-budget shape: say what was not covered instead of serving a shorter list silently.
RESOLVE_MAX_NAMES = 64
#: How long a resolved address set stays fresh. A LAN target moves when the operator moves it,
#: not every poll, and this section exists to be *cheap*: the dashboard polls the agent every
#: 10 s, and a lookup per name per poll would make the agent the noisiest DNS client on the
#: network. Anything older than this is refreshed on a background thread, never on the request.
RESOLVE_CACHE_SECONDS = 300.0
#: The wall-clock ceiling on one inline pass. `socket.getaddrinfo` takes no timeout argument, so
#: the only protection against a black-holing resolver is stopping between lookups — and the
#: reason that matters is the same one SMART learned: blowing `HOST_TIMEOUT_SECONDS` here turns
#: the **system** page red over an add-on it never draws (AGENTS.md invariant 3/4). The first
#: pass runs inline (a fresh agent must answer with real addresses), every later one runs on a
#: background thread, so this bound is the worst cost of a single request.
RESOLVE_PASS_BUDGET_SECONDS = 3.0
#: How the addresses were obtained, published rather than implied: `libc` means
#: `getaddrinfo()`, which is the resolution haproxy's own `init-addr last,libc,none` performs on
#: this box. It is not the `resolver` directive's answer (nginx queries that server directly),
#: and the payload says both so an operator can see a divergence — this module does not hand-roll
#: UDP DNS queries to name it a second time.
RESOLVE_VIA = "libc"
#: The `getaddrinfo` return codes that mean "this name has no address right now" as opposed to
#: "the lookup could not be performed". `EAI_NODATA` is what glibc calls a no-data answer and
#: CPython does not export it (it is deprecated upstream), so the value is spelled out beside a
#: `getattr` rather than guessed at: a temporary DNS failure (`EAI_AGAIN`) is deliberately **not**
#: in this set, because "no answer" and "no server" are different operator problems.
_EAI_NODATA = getattr(socket, "EAI_NODATA", -5) # CPython does not export it
RESOLVE_NXDOMAIN_CODES = frozenset({socket.EAI_NONAME, _EAI_NODATA})
# pci slot -> human model, filled on first look (lspci output cannot change at runtime).
_GPU_MODEL_CACHE: dict[str, str | None] = {}
VENDOR_NAMES = {"0x1002": "AMD", "0x10de": "NVIDIA", "0x8086": "Intel", "0x15ad": "VMware",
"0x1af4": "Virtio", "0x1a03": "ASPEED", "0x10ec": "Realtek"}
# sysfs names worth showing as "the CPU temperature", in preference order.
CPU_TEMP_CHIPS = ("k10temp", "zenpower", "coretemp", "cpu_thermal", "soc_thermal", "acpitz")
# Bracket groups in lspci output that name the vendor rather than the product.
_GPU_VENDOR_TAG = re.compile(r"(?:amd|ati|nvidia|intel|apple|matrox|s3|vmware|red hat|xinxiang"
r"|cirrus logic)(?:\s*/\s*(?:amd|ati|nvidia|intel))?", re.I)
# "Advanced Micro Devices, Inc.", "Intel Corporation", "NVIDIA Corporation" …
# The trailing `\b` matters: without it, "VGA compatible controller" splits on the "co" of
# "compatible" and the legend reads "mpatible controller: …".
_GPU_CORP_SUFFIX = re.compile(r"\b(?:Incorporated|Incorporation|Corporation|Company|Limited"
r"|Inc|Corp|Ltd|Co)\b\.?,?\s*", re.I)
# "00:02.0 VGA compatible controller: Intel …" — slot and class are not part of the name.
_GPU_PCI_PREFIX = re.compile(r"^[0-9a-fA-F]{2,4}:[0-9a-fA-F]{2}\.[0-9a-fA-F]\s*[^:]{0,60}:\s*")
def short_gpu_model(name: str | None) -> str | None:
"""Turn an lspci description into something a chart legend can hold.
lspci says `Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700]`;
the useful product name is the last bracket group, and AMD lists APU aliases separated
by slashes where the last one is the strongest part. The full string is still reported
as `model_full` so nothing is lost — this is display, not identity.
"""
if not name:
return None
groups = re.findall(r"\[([^\]]+)\]", name)
# [AMD/ATI] and [NVIDIA] are vendor tags, not products — keeping them turned an iGPU
# named Phoenix1 into a legend that said "ATI".
products = [g for g in groups if not _GPU_VENDOR_TAG.fullmatch(g.strip())]
if products:
candidate = products[-1].split("/")[-1].strip()
else:
rest = _GPU_PCI_PREFIX.sub("", name)
rest = re.sub(r"\[[^\]]+\]", " ", rest) # drop vendor tags
candidate = _GPU_CORP_SUFFIX.split(rest, maxsplit=1)[-1].strip(" ,;-")
candidate = candidate.strip() or name.strip()
return candidate[:40] + "\u2026" if len(candidate) > 40 else candidate
def _to_float(text: str) -> float | None:
try:
value = float(text)
except (TypeError, ValueError):
return None
return value if value == value and abs(value) != float("inf") else None
def _at(values: list[float | None], index: int) -> float | None:
"""Element `index`, or None. /proc and /sys files are truncated more often than you'd think.
This was a closure over the enclosing loop variable, which is the bug pattern B023 warns
about: escape the loop and it reads the *last* iteration's counters. It takes its data now.
"""
return values[index] if index < len(values) else None
class Collector:
"""Reads one host's /proc and /sys. `root` is injectable so tests can hand it a tree
of fixture files instead of a live machine."""
def __init__(self, root: str | os.PathLike[str] = "/", *, top_processes: int = 5,
net_all: bool = False, smartctl: str = "smartctl",
smart: bool = True, proxy: bool = True, resolve: bool = True,
haproxy_config: str = HAPROXY_CONFIG_PATH,
nginx_config: str = NGINX_CONFIG_PATH) -> None:
self.root = Path(root)
self.top_processes = top_processes
self.net_all = net_all
self.smartctl = smartctl
self.smart_enabled = smart
# The proxy add-on: the two configs and the live socket table. One flag for the family,
# because `--no-proxy` is one operator decision about a host that serves nothing — and a
# box with no proxy has no pipes to watch from either side. Both config paths are
# absolute on purpose: they are resolved under `root`, so the fake-tree tests read
# `root/etc/haproxy/haproxy.cfg`.
self.proxy_enabled = proxy
self.haproxy_config = haproxy_config
self.nginx_config = nginx_config
# Name resolution: its own flag, because it is a different kind of work from reading a
# file — it leaves the host. It is meaningless without the proxy sections (they are the
# only source of names, decision 3), so `--no-proxy` removes it as well as `--no-resolve`.
self.resolve_enabled = resolve
self.resolve_max_names = RESOLVE_MAX_NAMES
self.resolve_cache_seconds = RESOLVE_CACHE_SECONDS
self.resolve_pass_budget_seconds = RESOLVE_PASS_BUDGET_SECONDS
self._resolver_lock = threading.Lock() # a pass may run while a handler serves
# Entries keyed by `resolver_key(name, port)`; the timestamp is the monotonic age of the
# pass that wrote them, `_resolver_pass_ts` the wall clock the section publishes (an entry
# keeps the time *it* was answered, so a served-from-cache entry cannot look current).
self._resolver_cache: dict[str, dict[str, Any]] = {}
self._resolver_cache_ts: float | None = None
self._resolver_pass_ts: float | None = None
self._resolver_errors: list[str] = []
self._resolver_refreshing = False # at most one background refresh at a time
self.smart_timeout_seconds = SMARTCTL_TIMEOUT_SECONDS
self.smart_budget_seconds = SMARTCTL_BUDGET_SECONDS
self.smart_cache_seconds = SMARTCTL_CACHE_SECONDS
self._smart_lock = threading.Lock() # ThreadingHTTPServer may poll while a pass runs
self._smart_cache: tuple[float, dict[str, dict[str, Any] | None],
list[str]] | None = None
self._smart_refreshing = False # at most one background refresh at a time
self._lock = threading.Lock() # one HTTP handler at a time may touch the pid cache
self._pid_cpu: dict[int, tuple[float, int, str]] = {} # pid -> (ts, jiffies, name)
self._boot_ts: float | None = None
self._page_size = os.sysconf("SC_PAGE_SIZE") or 4096
self._ticks = os.sysconf("SC_CLK_TCK") or 100
self._cpu_model_cache: str | None = None
# -- tiny fs helpers ----------------------------------------------------- #
def _path(self, *parts: str) -> Path:
return self.root.joinpath(*parts)
def read_text(self, *parts: str) -> str | None:
try:
return self._path(*parts).read_text(errors="replace")
except (OSError, ValueError):
return None
def read_lines(self, *parts: str) -> list[str]:
text = self.read_text(*parts)
return text.splitlines() if text else []
# -- config-file reads (the proxy-config add-on) ------------------------- #
def _config_path(self, config: str) -> Path:
"""Resolve a config path the way every other path here is resolved.
An absolute path — the default of both `--haproxy-config` and `--nginx-config`, and what
a deployment writes — is read **under `--root`**, so the fake-tree tests read the fake
tree like every other collector here. A relative one is taken as written: haproxy's own
`-f` and nginx's `-c` both resolve it against the working directory, and a path an
operator handed this agent means what it means on that host, not under a chroot nobody
asked for.
"""
return Path(config) if not config.startswith("/") else self._path(config.lstrip("/"))
def read_config(self, config: str, max_bytes: int) -> str:
"""One size-bounded read of a config file, for the proxy add-on's best-effort sections.
Raises rather than deciding what to report, because the two failures are two different
operator actions: `FileNotFoundError` is "nothing installed here" (the section's
`available: false` arm) and any other `OSError`/`ValueError` is "there and unreadable"
(the `null` arm, including a file over `max_bytes` — the bound that keeps a path aimed
at a log or a device node from costing the snapshot an unbounded read).
"""
path = self._config_path(config)
if path.stat().st_size > max_bytes:
raise ValueError(f"larger than {max_bytes} bytes")
return path.read_text(encoding="utf-8", errors="replace")
def _config_label(self, arg: str, resolved: str) -> str:
"""The path a config would write for a file this agent read under `--root`.
`files` and the include join speak the config's paths (`/etc/nginx/conf.d/http.conf`),
because that is the sentence an operator can check against `nginx -t`. Only an absolute
argument is rooted, so only it comes back un-rooted; a relative one was taken as written
and is labelled as written.
"""
if not arg.startswith("/"):
return resolved
try:
return f"/{Path(resolved).relative_to(self.root)}"
except ValueError:
return resolved
def read_number(self, *parts: str) -> float | None:
"""First number in the file.
`/proc/uptime` is `210036.18 9613750.86`, while `/sys/.../scaling_cur_freq` is bare.
Parsing the whole file as a float returns None for exactly the files that matter,
which reads as "this kernel has no uptime" rather than "you forgot to split".
"""
text = self.read_text(*parts)
if not text:
return None
fields = text.split()
return _to_float(fields[0]) if fields else None
# -- cpu ----------------------------------------------------------------- #
def cpu(self) -> dict[str, Any]:
"""Jiffies per mode, cumulatively since boot. Divide by SC_CLK_TCK for seconds."""
modes: dict[str, float] = {}
cores: list[dict[str, Any]] = []
for line in self.read_lines("proc/stat"):
if not line.startswith("cpu"):
break
fields = line.split()
name = fields[0]
values = [float(v) for v in fields[1:11]] if len(fields) > 1 else []
if not values:
continue
while len(values) < 10:
values.append(0.0)
sample = dict(zip(MODE_NAMES, values[:10], strict=True))
# guest and guest_nice are already counted inside user/nice on Linux
sample["total"] = sum(values[:8])
if name == "cpu":
modes = sample
elif name[3:].isdigit():
cores.append({"core": int(name[3:]), "modes": sample})
load = self.read_text("proc/loadavg")
load_parts = load.split() if load else []
procs_running = procs_blocked = ctxt_total = processes_total = None
for line in self.read_lines("proc/stat"):
if line.startswith("procs_running"):
procs_running = _to_float(line.split()[-1])
elif line.startswith("procs_blocked"):
procs_blocked = _to_float(line.split()[-1])
elif line.startswith("ctxt"):
ctxt_total = _to_float(line.split()[-1])
elif line.startswith("processes"):
processes_total = _to_float(line.split()[-1])
freq = self.cpu_frequency()
return {
"modes": modes or None,
"cores": cores,
"clock_ticks_per_second": self._ticks,
"load1": _to_float(load_parts[0]) if len(load_parts) > 0 else None,
"load5": _to_float(load_parts[1]) if len(load_parts) > 1 else None,
"load15": _to_float(load_parts[2]) if len(load_parts) > 2 else None,
"procs_running": procs_running,
"procs_blocked": procs_blocked,
"ctxt_total": ctxt_total,
"processes_total": processes_total,
"freq_mhz": freq["current"],
"freq_max_mhz": freq["max"],
}
def cpu_frequency(self) -> dict[str, float | None]:
"""Average current / best-case maximum, in MHz."""
current: list[float] = []
maximum: list[float] = []
base = self._path("sys/devices/system/cpu")
if base.is_dir():
for cpu in sorted(base.glob("cpu[0-9]*")):
cur = self.read_number(str(cpu.relative_to(self.root)), "cpufreq/scaling_cur_freq")
if cur is None:
cur = self.read_number(str(cpu.relative_to(self.root)), "cpufreq/cpuinfo_cur_freq")
if cur is not None:
current.append(cur / 1000.0)
mx = self.read_number(str(cpu.relative_to(self.root)), "cpufreq/cpuinfo_max_freq")
if mx is not None:
maximum.append(mx / 1000.0)
return {
"current": (sum(current) / len(current)) if current else None,
"max": max(maximum) if maximum else None,
}
def cpu_model(self) -> str | None:
if self._cpu_model_cache:
return self._cpu_model_cache
for line in self.read_lines("proc/cpuinfo"):
if line.lower().startswith(("model name", "cpu model", "hardware")):
_, _, value = line.partition(":")
if value.strip():
self._cpu_model_cache = value.strip()
return self._cpu_model_cache
# arm64 fallback: compatible = "arm,cortex-a76"
text = self.read_text("proc/device-tree/compatible")
if text:
self._cpu_model_cache = text.strip("\x00").split(",")[-1].strip()
return self._cpu_model_cache
return None
def core_counts(self) -> tuple[int | None, int | None]:
text = self.read_text("proc/cpuinfo") or ""
logical = len(re.findall(r"^processor\s*:", text, re.M)) or (os.cpu_count() or None)
# one block per *thread*, so dedupe (socket, core) — counting how many "core id:"
# lines exist yields 1 for the set of colons, which is how a 24-core TR reports
# "1 physical core" and makes a fleet table quietly meaningless.
seen: set[tuple[str, str]] = set()
for block in text.split("\n\n"):
socket = re.search(r"^physical id\s*:\s*(\d+)", block, re.M)
core = re.search(r"^core id\s*:\s*(\d+)", block, re.M)
if socket and core:
seen.add((socket.group(1), core.group(1)))
physical: int | None = len(seen) or None
if physical is None and logical:
threads = self._threads_per_core()
if threads:
physical = max(1, logical // threads)
return (logical or None), physical
def _threads_per_core(self) -> int | None:
"""SMT width from cpu0's sibling list: `0,24` or `0-1` → 2. VMs often omit it."""
listing = self.read_text("sys/devices/system/cpu/cpu0/topology/thread_siblings_list")
if not listing:
return None
count = 0
for part in listing.strip().split(","):
start, dash, end = part.partition("-")
first = _to_float(start)
if first is None:
continue
last = _to_float(end) if dash else None
count += int((last if last is not None else first) - first) + 1
return count or None
# -- memory -------------------------------------------------------------- #
def memory(self) -> dict[str, float | None]:
raw: dict[str, float] = {}
for line in self.read_lines("proc/meminfo"):
name, _, rest = line.partition(":")
value = _to_float(rest.split()[0]) if rest.split() else None
if value is not None:
raw[name.strip()] = value * 1024.0 # meminfo is in kB
if not raw:
return dict.fromkeys(MEMORY_KEYS)
total = raw.get("MemTotal")
available = raw.get("MemAvailable", raw.get("MemFree"))
swap_total = raw.get("SwapTotal", 0.0)
swap_free = raw.get("SwapFree", 0.0)
out: dict[str, float | None] = {
"total_bytes": total,
"free_bytes": raw.get("MemFree"),
"available_bytes": available,
"used_bytes": (total - available) if (total is not None and available is not None) else None,
"buffers_bytes": raw.get("Buffers"),
"cached_bytes": (raw.get("Cached", 0.0) + raw.get("SReclaimable", 0.0)) or None,
"dirty_bytes": raw.get("Dirty"),
"swap_total_bytes": swap_total,
"swap_used_bytes": swap_total - swap_free,
"committed_bytes": raw.get("Committed_AS"),
}
for key in MEMORY_KEYS:
out.setdefault(key, None)
return out
# -- filesystems --------------------------------------------------------- #
def filesystems(self) -> list[dict[str, Any]]:
out: list[dict[str, Any]] = []
seen: set[tuple[str, str]] = set()
for line in self.read_lines("proc/mounts"):
parts = line.split()
if len(parts) < 3:
continue
device, mountpoint, fstype = parts[0], parts[1], parts[2]
mountpoint = mountpoint.replace("\\040", " ").replace("\\011", "\t")
if fstype in PSEUDO_FSTYPES:
continue
if any(mountpoint == p or mountpoint.startswith(p + "/") for p in PSEUDO_MOUNT_PREFIXES):
continue
key = (device, mountpoint)
if key in seen:
continue # bind mounts / btrfs subvolume aliases
seen.add(key)
try:
st = os.statvfs(mountpoint)
except OSError:
continue
block = st.f_frsize or st.f_bsize
total = st.f_blocks * block
free = st.f_bavail * block # what an unprivileged writer can use
used = (st.f_blocks - st.f_bfree) * block
denominator = used + free # df's "Use%" basis, not raw total
out.append({
"device": device,
"mountpoint": mountpoint,
"fstype": fstype,
"total_bytes": total or None,
"used_bytes": used,
"free_bytes": free,
"used_pct": (used / denominator * 100.0) if denominator > 0 else None,
"read_only": bool(st.f_flag & os.ST_RDONLY),
})
out.sort(key=lambda item: item["mountpoint"])
return out
# -- disk io ------------------------------------------------------------- #
def disk_io(self) -> dict[str, Any]:
"""Whole disks via /sys/block/<dev>/stat, which is documented ABI-stable.
/proc/diskstats is NOT: the discard/flush counters were inserted at different
offsets in different kernels, so "which index is in-flight" depends on the build.
Measured on driveripper (kernel 7.2): its diskstats line and its /sys/block/stat
line are the same 17 numbers — classic 11 first (in-flight at index 8, matching
/sys/block/<dev>/inflight), then the discard/flush tail. So the classic 11 are
read from fixed indexes and the tail only when the 17-number shape is present.
/sys/block lists whole disks only (nvme0n1, not nvme0n1p1), which is exactly the
set we want: partitions and dm-* layers would double-count the same bytes.
"""
devices: list[dict[str, Any]] = []
names: list[str] = []
base = self._path("sys/block")
if not base.is_dir():
return {"devices": [], "device_names": []}
for entry in sorted(base.iterdir(), key=lambda p: p.name):
name = entry.name
if name.startswith(VIRTUAL_DISK_PREFIXES):
continue
numbers = (self.read_text("sys/block", name, "stat") or "").split()
if len(numbers) < 11:
continue
values = [_to_float(v) for v in numbers]
rotational = self.read_number("sys/block", name, "queue/rotational")
model = (self.read_text("sys/block", name, "device/model") or "").strip() or None
names.append(name)
devices.append({
"name": name,
"model": model,
"rotational": None if rotational is None else rotational == 1.0,
"size_bytes": (self.read_number("sys/block", name, "size") or 0) * 512.0 or None,
"reads_total": _at(values, 0),
"writes_total": _at(values, 4),
"read_bytes_total": (_at(values, 2) or 0.0) * 512.0,
"write_bytes_total": (_at(values, 6) or 0.0) * 512.0,
"in_flight": _at(values, 8),
"io_time_ms_total": _at(values, 9),
"weighted_io_ms_total": _at(values, 10),
# discard/flush tail: only this shape is verified (see docstring)
"discard_bytes_total": ((_at(values, 13) or 0.0) * 512.0
if len(values) >= 17 else None),
})
return {"devices": devices, "device_names": names}
# -- smart (smartctl) ------------------------------------------------------ #
def smart(self, device_names: list[str]) -> tuple[dict[str, dict[str, Any] | None],
list[str]]:
"""Best-effort `smartctl -i -A -H <device>` per whole disk; never raises.
Returns (per-device parsed dict or None, error lines). Failure semantics (phase
09 task 01): the binary missing produces **one** error entry for the whole pass
and every disk `None`; one denied/wedged device produces its own error entry and
nulls only that disk. Results are cached for `smart_cache_seconds` so the fast
system poll cannot re-issue ATA commands every few seconds; the cache replays the
same error lines, so a snapshot always states why `smart` is null.
An **expired** cache is refreshed on a background thread and the request is served
the previous result, because the pass budget (20 s floor, 5 s per disk) is five
times the dashboard's
`HOST_TIMEOUT_SECONDS` (4 s): run inline on the request path, one wedged disk would
turn the *system* source red every `smart_cache_seconds` for a fault in an add-on
(AGENTS.md invariant 4 — SMART must not blink the page that does not use it). The
very first pass still runs inline, so a freshly started agent answers `/snapshot`
with real SMART fields instead of a page of `n/a` until someone polls it twice.
"""
now = time.monotonic()
with self._smart_lock:
cached = self._smart_cache
if cached is not None:
if now - cached[0] < self.smart_cache_seconds or self._smart_refreshing:
return dict(cached[1]), list(cached[2])
self._smart_refreshing = True
threading.Thread(target=self._refresh_smart, args=(list(device_names),),
name="smart-refresh", daemon=True).start()
return dict(cached[1]), list(cached[2])
return self._smart_pass(device_names)
def _refresh_smart(self, device_names: list[str]) -> None:
"""The expired-cache pass, off the request path (see `smart()`).
A failure here keeps the previous cache (and its age) rather than clearing it: the
snapshot keeps answering with the last truth it had, and the retry comes on the next
expiry instead of hammering a disk that just failed.
"""
try:
self._smart_pass(device_names)
except Exception as exc: # noqa: BLE001 - the thread must always clear its own flag
with self._smart_lock:
previous = self._smart_cache
results = dict(previous[1]) if previous else dict.fromkeys(device_names)
errors = list(previous[2]) if previous else []
self._smart_cache = (time.monotonic(), results,
[*errors,
f"smart: background refresh failed: "
f"{type(exc).__name__}: {exc}"])
finally:
with self._smart_lock:
self._smart_refreshing = False
def _smart_pass(self, device_names: list[str]) -> tuple[dict[str, dict[str, Any] | None],
list[str]]:
"""One `smartctl` pass over the whole disks, and the cache write that goes with it."""
now = time.monotonic()
results: dict[str, dict[str, Any] | None] = {}
errors: list[str] = []
# 20 s floor scaled to the per-device bound: a 16-disk shelf takes more than 20 s
# when the drives answer, and truncating the pass would null the tail drives.
# An explicit override (the budget test sets this to 0) wins over the scale.
budget = (max(SMARTCTL_BUDGET_SECONDS, self.smart_timeout_seconds * len(device_names))
if self.smart_budget_seconds == SMARTCTL_BUDGET_SECONDS
else self.smart_budget_seconds)
deadline = now + budget
for name in device_names:
left = deadline - time.monotonic()
if left <= 0:
remaining = [n for n in device_names if n not in results]
for skipped_name in remaining:
results[skipped_name] = None
errors.append(f"smart: pass budget of {budget:.0f}s "
f"exhausted, {len(remaining)} disk(s) skipped")
break
try:
proc = subprocess.run([self.smartctl, "-i", "-A", "-H", f"/dev/{name}"],
capture_output=True, text=True,
timeout=min(left, self.smart_timeout_seconds),
check=False)
except FileNotFoundError:
# one entry for the pass, not one per disk: a missing binary is one fact
errors.append(f"smart: {self.smartctl} not found — install smartmontools "
"and give this user device read access (disk group) for "
"SMART data")
for missing_name in device_names:
results.setdefault(missing_name, None)
break
except subprocess.TimeoutExpired:
results[name] = None
errors.append(f"smart: {name}: smartctl timed out after "
f"{self.smart_timeout_seconds:.0f}s (device wedged?)")
continue
except OSError as exc:
results[name] = None
errors.append(f"smart: {name}: smartctl could not run: {exc}")
continue
parsed = parse_smart_output(proc.stdout)
if any(value is not None for value in parsed.values()):
results[name] = parsed # partial output is still what the drive published
else:
results[name] = None
errors.append(f"smart: {name}: "
f"{smart_failure_line(proc.returncode, proc.stdout, proc.stderr)}")
with self._smart_lock:
self._smart_cache = (time.monotonic(), results, errors)
return dict(results), list(errors)
def attach_smart(self, disk_io: dict[str, Any], errors: list[str]) -> None:
"""Put `smart` on every disk row; `--no-smart` leaves them all null silently."""
devices = disk_io.get("devices") or []
for device in devices:
device["smart"] = None
if not devices or not self.smart_enabled:
return
results, smart_errors = self.smart([str(device.get("name")) for device in devices])
errors.extend(smart_errors)
for device in devices:
parsed = results.get(str(device.get("name")))
if not parsed:
continue
smart = dict(parsed)
if not smart.get("model"):
# smartctl's identity block can be absent (some bridges refuse -i while
# answering the health log); the agent's sysfs model is the fallback.
smart["model"] = device.get("model")
device["smart"] = smart
# -- proxy configs (haproxy) ---------------------------------------------- #
def haproxy(self, errors: list[str]) -> dict[str, Any]:
"""Read and parse the host's haproxy config — never raises, never fails the snapshot.
`available` is the three-way answer `attach_smart`'s `smartctl_available` established,
because the three states say three different things and only one of them is a fault:
`true` the file was read and parsed, `false` there is no config at this path (this host
runs no haproxy — unconfigured, not broken), `null` a file is there but could not be
read or understood. Each non-`true` arm writes exactly **one** `haproxy: ` line for the
whole pass; the missing one says the file is **not found** and names the path, which is
what lets a dashboard tell "no haproxy here" from "haproxy config unreadable" without a
second probe, and what `tests/test_agent.py` pins.
Everything else in the section is the config's own text: a bind keeps the address it was
written with (`:::22` splits into `"::"` and `22`), a `server` that names no port keeps
`port: null` rather than a made-up `0`, and the frontend to backend routing stays the
**names**, so the dashboard joins them and this file never interprets a rule.
"""
config = self.haproxy_config
section: dict[str, Any] = {"available": None, "config": config,
"frontends": [], "backends": []}
# Path resolution and the size bound live in `read_config`, the one place both
# proxy-config sections decide how a config file is read.
try:
text = self.read_config(config, HAPROXY_CONFIG_MAX_BYTES)
except FileNotFoundError:
section["available"] = False
errors.append(f"haproxy: {config} not found — haproxy is not installed here, or "
"its config lives elsewhere (--haproxy-config); --no-proxy opts out")
return section
except (OSError, ValueError) as exc:
errors.append(f"haproxy: {config} is there but could not be read: {exc}")
return section
parsed = parse_haproxy_config(text)
if not parsed["recognised"]:
errors.append(f"haproxy: {config} holds no haproxy section (global, defaults, "
"frontend, backend, listen) — not a haproxy config")
return section
section["available"] = True
section["frontends"] = parsed["frontends"]
section["backends"] = parsed["backends"]
return section
# -- proxy configs (nginx) ------------------------------------------------ #
def nginx(self, errors: list[str]) -> dict[str, Any]:
"""The `nginx` section on its own — `nginx_document()`'s first half."""
return self.nginx_document(errors)[0]
def nginx_document(self, errors: list[str]) -> tuple[dict[str, Any], str | None]:
"""Read and parse the host's nginx configs — never raises, never fails the snapshot.
Returns `(section, resolver directive)`. The second half is the only reason this method
is not the section itself: the `resolver` section prints nginx's own `resolver` line next
to `via: "libc"` so an operator can see the two vantages disagree, and taking it from the
text this pass already read is what keeps that free (no second file read per snapshot).
It deliberately does **not** go in the section: the `nginx` document stays byte-stable,
which is what makes this phase's additivity control a strict top-level key comparison.
The same three-way answer as `haproxy()`, for the other half of the gateway's ingress:
`true` the main config was read and understood, `false` there is no config at this path
(this host runs no nginx — one `nginx: ` line saying the file is **not found**), `null`
a file is there but could not be read, could not be parsed, or holds no nginx context
(one line either way). The
wording is the cross-module contract `collectors.ADDON_ERROR_PREFIXES` exists for: the
dashboard reads "not found" to answer "is nginx installed here?", and the line never
makes the *system* page amber for a box that never had nginx.
The main config is not the whole picture — nginx keeps its vhosts in the files its
`include` lines name — so those are followed **one level** and bounded
(`NGINX_INCLUDE_MAX_FILES` / `NGINX_INCLUDE_MAX_BYTES`, in `nginx_include_files`), and
`files` names the main config plus every included document that was actually merged. What
is left out is said out loud: each reason the pass could not take an include it found
costs one more line, never one per file.
"""
config = self.nginx_config
section: dict[str, Any] = {"available": None, "config": config, "files": [],
"servers": [], "upstreams": [], "maps": []}
try:
text = self.read_config(config, NGINX_CONFIG_MAX_BYTES)
except FileNotFoundError:
section["available"] = False
errors.append(f"nginx: {config} not found — nginx is not installed here, or its "
"config lives elsewhere (--nginx-config); --no-proxy opts out")
return section, None
except (OSError, ValueError) as exc:
errors.append(f"nginx: {config} is there but could not be read: {exc}")
return section, None
# The two extra return values are already spoken about in `errors`; the section itself
# carries no counters, because "which files were read" is what `files` is for.
included, _dropped, _unreadable = self.nginx_include_files(nginx_include_paths(text),
errors)
parsed = parse_nginx_config(text, included)
if parsed["unparsed"]:
errors.append(f"nginx: {len(parsed['unparsed'])} included file(s) could not be "
f"parsed: {parsed['unparsed'][0]}")
if parsed["error"] is not None:
errors.append(f"nginx: {config} is there but could not be parsed: {parsed['error']}")
return section, None
if not parsed["recognised"]:
errors.append(f"nginx: {config} holds no nginx context (http, stream, server, "
"upstream, map) — not an nginx config")
return section, None
section["available"] = True
# The main config first: `files` is the answer to "where do these servers come from",
# and an operator comparing it with `nginx -t` should see the file nginx was started on.
section["files"] = [config, *parsed["files"]]
section["servers"] = parsed["servers"]
section["upstreams"] = parsed["upstreams"]
section["maps"] = parsed["maps"]
return section, cast("str | None", parsed["resolver"])
def nginx_include_files(self, include_args: list[str],
errors: list[str]) -> tuple[dict[str, str], int, list[str]]:
"""Read the files the main config's `include` lines name. Returns (documents, dropped, unreadable).
One level, bounded by `NGINX_INCLUDE_MAX_FILES` / `NGINX_INCLUDE_MAX_BYTES`: this runs on
every snapshot, and an `include /etc/nginx/**` written by someone who liked globs must
not turn a 2 KB read into a directory walk. A glob is expanded under `--root` like every
other path here and sorted, so the same tree always produces the same order; a literal
path that does not exist contributes nothing and **no line** (an `include` of a file this
host does not have is absence, the `available: false` arm one level down), while the two
facts that *are* worth a sentence get exactly one line each — the bound that dropped
`dropped` files, and the `unreadable` count with its first path.
Each document is keyed by the path the **config** writes (`_config_label`), not by the
path under `--root` this host happens to have: that is what lets `parse_nginx_config`
match its own `include` lines against the documents, and what makes `files` readable to
an operator holding `nginx -t` output instead of a temp directory.
"""
documents: dict[str, str] = {}
unreadable: list[str] = []
seen: set[str] = set()
used_bytes = 0
dropped = 0
for arg in include_args:
resolved = str(self._config_path(arg))
candidates = sorted(glob.glob(resolved)) if any(c in arg for c in _GLOB_SPECIALS) \
else [resolved]
for real in candidates:
label = self._config_label(arg, real)
if label in seen:
continue # two include lines naming one file read it once
seen.add(label)
if len(documents) >= NGINX_INCLUDE_MAX_FILES:
dropped += 1
continue
path = Path(real)
try:
size = path.stat().st_size
if used_bytes + size > NGINX_INCLUDE_MAX_BYTES:
dropped += 1
continue
documents[label] = path.read_text(encoding="utf-8", errors="replace")
except FileNotFoundError:
continue
except (OSError, ValueError):
unreadable.append(label)
continue
used_bytes += size
if dropped:
errors.append(f"nginx: {dropped} included file(s) beyond the "
f"{NGINX_INCLUDE_MAX_FILES}-file / {NGINX_INCLUDE_MAX_BYTES}-byte "
"include bound were not read — the section covers what it could read")
if unreadable:
errors.append(f"nginx: {len(unreadable)} included file(s) matched an include line but "
f"could not be read: {unreadable[0]}")
return documents, dropped, unreadable
# -- live sockets -------------------------------------------------------- #
def read_proc_table(self, relative: str,
max_lines: int = PROC_NET_MAX_LINES) -> tuple[list[str], bool]:
"""Read at most `max_lines` lines of a kernel table. Returns (lines, truncated).
Raises `FileNotFoundError` for "this kernel publishes no such table" and any other
`OSError` for "it is there and I could not read it", because the section has to tell
those two apart and `read_text`/`read_lines` both swallow the difference into a value —
`read_lines` answering `[]` would make an unreadable table indistinguishable from a
host with no sockets, which is rule 2 with a helper's name on it.
Stopping at the bound is the point of the method: the read never walks a table longer
than the section reports, so the cost of `sockets` belongs to this constant and not to
whatever connection count the box happens to be carrying.
"""
lines: list[str] = []
with self._path(relative).open(encoding="utf-8", errors="replace") as handle:
for line in handle:
if len(lines) >= max_lines:
return lines, True
lines.append(line)
return lines, False
def sockets(self, errors: list[str]) -> dict[str, Any]:
"""Read the kernel's two TCP tables. Never raises, never fails the snapshot.
`{available, listens, established}`: every LISTEN socket and every ESTABLISHED pair in
`/proc/net/tcp` and `/proc/net/tcp6`, each row carrying the family it came from, the
addresses unrolled from `/proc`'s per-word byte order, and **no join with the config**
— a socket does not know its domain or its backend, so phase 29's page joins these
ports to the `haproxy`/`nginx` sections and an unclaimed socket stays visible as one.
The tables are read independently and either one answering makes `available: true`,
because a kernel built without IPv6 has no `tcp6` file at all: the host told us what it
has. An absent table therefore costs **no line** when the other answered (the nginx
include rule — absence is not a fault), while a present-but-denied one costs exactly
one, because that is the operator's fix. Neither answering is `available: null` with one
line naming both paths; nothing here can zero a snapshot.
There is deliberately no `available: false` arm: `/proc/net/tcp` is not something a host
can leave uninstalled, so the arm the two config sections use for "nothing installed
here" has nothing to name, and `false` beside an empty list would read as "this host has
no listeners" — a fact this wire never published.
"""
section: dict[str, Any] = {"available": None, "listens": [], "established": []}
listens: list[dict[str, Any]] = []
established: list[dict[str, Any]] = []
missing: list[str] = []
unreadable: list[tuple[str, str]] = []
truncated: list[str] = []
for family, relative in PROC_NET_TABLES:
label = f"/{relative}"
try:
lines, cut = self.read_proc_table(relative)
except FileNotFoundError:
missing.append(label)
continue
except (OSError, ValueError) as exc:
unreadable.append((label, str(exc)))
continue
if cut:
truncated.append(label)
table_listens, table_established = parse_proc_net_tcp("\n".join(lines), family)
listens.extend(table_listens)
established.extend(table_established)
answered = len(missing) + len(unreadable) < len(PROC_NET_TABLES)
# an absent table when the other answered is the kernel not having IPv6, not a fault, so
# it costs no line; being denied one does, and nothing answering always does.
if unreadable or not answered:
errors.append(socket_gap_line(missing, unreadable, not answered))
if truncated:
errors.append(socket_truncation_line(truncated))
section["listens"] = listens
section["established"] = established
section["available"] = True if answered else None
return section
# -- name resolution ------------------------------------------------------ #
def resolver(self, haproxy: Mapping[str, Any], nginx: Mapping[str, Any],
config_directive: str | None, errors: list[str]) -> dict[str, Any]:
"""Resolve the names the two proxy configs declare. Never raises, never fails a snapshot.
The point is that a route written as `server mc1 minecraft.reeselink.com:25565` ends at an
address, and the kernel's socket table speaks addresses: without this section the
dashboard can only count the connections whose target the config wrote *as an address*,
which on this ingress is 2 routes out of 33. The lookup happens here, on the box that
opens the connections, because that is the vantage the proxy itself resolves from; the
dashboard resolving the name through its own `resolv.conf` would answer a different
question (and `sources/gateway.py` is spied to fail if it ever tries).
`available` is the family's three-way answer, with **one** `resolver:` line per non-`true`
arm: `true` a pass ran, `false` there was nothing to resolve because neither proxy section
was read (the line says *not found* when the files simply are not here, which is the
wording the dashboard reads), `null` a pass that could not run. Configs that were read and
name no target are `true` with `entries: []` — that is absence, not a fault.
The cache is SMART's idiom, and for the SMART reason: the first pass runs inline so a
freshly started agent answers with real addresses, and every later expiry is refreshed on
a single-flight background thread while the request is served the previous result. A
resolver that has started black-holing must not cost the *system* poll its timeout over an
add-on that page never draws (AGENTS.md invariant 3/4).
"""
targets = target_names(haproxy, nginx)
if not targets and not _section_answered(haproxy) and not _section_answered(nginx):
# Nothing was read, so there is nothing to resolve. The two config sections have
# already said why in their own lines; this one says the resolver's own state, once.
line = resolver_not_found_line(haproxy, nginx)
errors.append(line)
return {"available": False, "via": RESOLVE_VIA, "config_directive": config_directive,
"ts": None, "entries": [], "errors": [line]}
now = time.monotonic()
with self._resolver_lock:
stamp = self._resolver_cache_ts
if stamp is not None:
if now - stamp >= self.resolve_cache_seconds and not self._resolver_refreshing:
self._resolver_refreshing = True
threading.Thread(target=self._refresh_resolver, args=(list(targets),),
name="resolver-refresh", daemon=True).start()
# Served from cache either way — the fresh-cache case and the "a refresh is
# already running" case — because waiting here is what this cache exists to avoid.
return self._resolver_section(self._resolver_cache, self._resolver_pass_ts,
self._resolver_errors, config_directive, errors)
entries, pass_errors = self.resolver_pass(targets)
return self._resolver_section(entries, self._resolver_pass_ts, pass_errors,
config_directive, errors)
def _resolver_section(self, entries: Mapping[str, dict[str, Any]], pass_ts: float | None,
pass_errors: Sequence[str], config_directive: str | None,
errors: list[str]) -> dict[str, Any]:
"""The section out of one pass's (or one cache's) result — the shape, in one place.
The lines go to **both** readers, because there are two: the snapshot's shared `errors`
list, which is what `collectors.ADDON_ERROR_PREFIXES` filters to decide the *system* page
is not degraded, and the section's own `errors`, which is the gateway page's sentence
beside the entries they belong to. Same strings, one append each.
"""
section: dict[str, Any] = {"available": True, "via": RESOLVE_VIA,
"config_directive": config_directive, "ts": pass_ts,
"entries": [dict(entry) for entry in entries.values()],
"errors": []}
for line in pass_errors:
errors.append(line)
section["errors"].append(line)
return section
def _refresh_resolver(self, targets: list[tuple[str, int | None]]) -> None:
"""The expired-cache pass, off the request path (see `resolver()`).
A failure keeps the previous entries **and** re-stamps the cache: the snapshot keeps
answering with the last truth it had, and the retry comes on the next expiry instead of
every poll against a resolver that just failed. The line is replaced rather than stacked,
because a refresh that fails every `resolve_cache_seconds` for a week is one fact, not
2000 — the repeated-failure rule the poller applies to its own events.
"""
try:
self.resolver_pass(targets)
except Exception as exc: # noqa: BLE001 - the thread must always clear its own flag
with self._resolver_lock:
line = (f"resolver: background refresh failed: {type(exc).__name__}: {exc}")
self._resolver_errors = [x for x in self._resolver_errors
if not x.startswith("resolver: background refresh failed")]
self._resolver_errors = [*self._resolver_errors, line]
self._resolver_cache_ts = time.monotonic()
finally:
with self._resolver_lock:
self._resolver_refreshing = False
def resolver_pass(self, targets: Sequence[tuple[str, int | None]]
) -> tuple[dict[str, dict[str, Any]], list[str]]:
"""One bounded libc lookup per declared target. Never raises out of this function.
Entries come back keyed by `resolver_key`, in target order, merged over the previous
cache so a pass that stopped at the budget keeps the entries it did not reach (each with
**its own** `ts`, so a stale address is visibly stale rather than freshly wrong). A name
that never got looked up at all and has no previous answer simply has no entry: the line
says how many were reached, and inventing an `error` status for a lookup that was never
attempted would be rule 2 with a status field on it.
"""
errors: list[str] = []
fresh: dict[str, dict[str, Any]] = {}
deadline = time.monotonic() + self.resolve_pass_budget_seconds
need_lookup = [pair for pair in targets if address_family(pair[0]) is None]
over_bound = len(need_lookup) - self.resolve_max_names
looked_up = 0
stopped = False
for name, port in targets:
# Literals first, and never counted against either bound: reporting one is a fact and
# dialling it would be a round trip to be told what the config already wrote, so a
# truncated pass must not be the reason an address the config spells is missing.
if address_family(name) is None:
continue
fresh[resolver_key(name, port)] = {"name": name, "port": port, "status": "literal",
"addresses": [name], "ts": time.time()}
for name, port in need_lookup:
if looked_up >= self.resolve_max_names:
continue # one line for the whole excess, after the loop
fresh[resolver_key(name, port)] = self._resolve_target(name, port)
looked_up += 1
if time.monotonic() >= deadline:
stopped = True
break
if stopped:
errors.append(f"resolver: pass budget {self.resolve_pass_budget_seconds:.1f}s "
f"exhausted, {looked_up} of {len(need_lookup)} targets looked up")
if over_bound > 0:
errors.append(f"resolver: {over_bound} of {len(need_lookup)} targets are beyond the "
f"{self.resolve_max_names}-name bound and were not looked up — the "
"section covers what it could resolve")
entries = self._store_resolver_cache(targets, fresh, errors)
return entries, errors
def _store_resolver_cache(self, targets: Sequence[tuple[str, int | None]],
fresh: Mapping[str, dict[str, Any]], errors: Sequence[str]
) -> dict[str, dict[str, Any]]:
"""Merge this pass into the cache in target order, and stamp it. Returns the entries.
Order is not cosmetics: `entries` is a list a page renders, so the same config must draw
in the same order on every poll. Names the config dropped leave the cache with it — they
are no longer declared targets, not targets that stopped resolving.
"""
merged: dict[str, dict[str, Any]] = {}
with self._resolver_lock:
previous = self._resolver_cache
for name, port in targets:
key = resolver_key(name, port)
entry = fresh.get(key) or previous.get(key)
if entry is not None:
merged[key] = entry
self._resolver_cache = merged
self._resolver_cache_ts = time.monotonic()
self._resolver_pass_ts = time.time()
# The pass's own lines replace the previous ones: a cache replay has to say the same
# thing the pass that produced it said (SMART's cache does the same), and a line from
# an older pass — a budget hit that has since stopped happening — would be a stale
# complaint about a reading that is now complete.
self._resolver_errors = list(errors)
return dict(merged)
def _resolve_target(self, name: str, port: int | None) -> dict[str, Any]:
"""One `getaddrinfo`, and the entry that says what it answered.
Both families are kept (`getaddrinfo` answers v4 and v6 when the host has both) because
the gateway connects over whichever one routing picks and the socket table speaks either.
A `gaierror` naming no-name/no-data is `nxdomain` — the name is answered and has no
address — and anything else is `error`, because "the server did not answer" and "the name
has no record" are different things to fix. Both carry `addresses: []`, never a stale
guess and never a `0`. The `ts` is taken here rather than passed in, so an entry says when
**it** was answered even in a pass that then stopped at the budget.
The keyword is `type=`, and that is load-bearing in a way only a real libc can teach:
`proto` is the **protocol number** (`IPPROTO_TCP` = 6) while `SOCK_STREAM` = 1 is a socket
*type*, so `proto=SOCK_STREAM` asks libc for a stream socket over ICMP and it answers
`EAI_SERVICE` for **every** name — a resolver that reports `error` on a healthy gateway,
with every route on the page uncounted, in a way no stubbed test can see because a stub
takes any keyword. What we want is libc's answer to "where do I connect a TCP socket",
which is exactly haproxy's `init-addr last,libc,none` question. Pinned by
`tests/test_agent.py` with no stub in sight, against an address libc parses by itself.
"""
try:
infos = socket.getaddrinfo(name, port, type=socket.SOCK_STREAM)
except socket.gaierror as exc:
code = exc.errno if isinstance(exc.errno, int) else 0
return {"name": name, "port": port,
"status": "nxdomain" if code in RESOLVE_NXDOMAIN_CODES else "error",
"addresses": [], "ts": time.time()}
except OSError as exc: # a resolver that fails as an I/O error, not as a DNS one
return {"name": name, "port": port, "status": "error", "addresses": [],
"detail": str(exc), "ts": time.time()}
except Exception as exc: # noqa: BLE001 - the promise is that this function never raises
# `getaddrinfo` is the one call in this agent that leaves the host, and whatever it
# does — a platform-specific `ValueError` for a name the C library cannot take, a
# stub in a test — the answer is one entry that says "error" beside the targets that
# answered, never a half-snapshot and never the *system* page going red over an
# add-on it does not draw (AGENTS.md invariant 4).
return {"name": name, "port": port, "status": "error", "addresses": [],
"detail": f"{type(exc).__name__}: {exc}", "ts": time.time()}
addresses = sort_resolver_addresses(str(info[4][0]) for info in infos)
# An empty answer list is libc saying the name resolved to nothing: no-data, not a fault.
return {"name": name, "port": port, "status": "ok" if addresses else "nxdomain",
"addresses": addresses, "ts": time.time()}
# -- network ------------------------------------------------------------- #
def network(self) -> dict[str, Any]:
interfaces: list[dict[str, Any]] = []
for line in self.read_lines("proc/net/dev"):
name, sep, rest = line.partition(":")
if not sep:
continue
name = name.strip()
if not self.net_all and name.startswith(VIRTUAL_NET_PREFIXES):
continue
fields = [_to_float(v) for v in rest.split()]
if len(fields) < 9:
continue
interfaces.append({
"name": name,
"rx_bytes_total": _at(fields, 0),
"rx_packets_total": _at(fields, 1),
"rx_errors_total": _at(fields, 2),
"rx_dropped_total": _at(fields, 3),
"tx_bytes_total": _at(fields, 8),
"tx_packets_total": _at(fields, 9),
"tx_errors_total": _at(fields, 10),
"tx_dropped_total": _at(fields, 11),
"speed_mbps": self._net_speed(name),
})
interfaces.sort(key=lambda item: item["name"])
return {"interfaces": interfaces}
def _net_speed(self, name: str) -> float | None:
value = self.read_number("sys/class/net", name, "speed")
return value if value and value > 0 else None
# -- gpus ---------------------------------------------------------------- #
def gpus(self) -> list[dict[str, Any]]:
"""AMD GPUs via DRM sysfs (gpu_busy_percent, mem_info_vram_*, power, hwmon temps).
That set exists on amdgpu since kernel ~4.x and is world-readable, which is why
this agent needs no root. NVIDIA/Intel are not wired up: the fleet has no such
card, and guessing at sysfs paths we cannot test would be worse than `null`.
"""
out: list[dict[str, Any]] = []
base = self._path("sys/class/drm")
if not base.is_dir():
return out
for card in sorted(base.glob("card[0-9]*")):
name = card.name
if "-" in name or name == "card": # only cardN, not cardN-DP-1 connectors
continue
device = card / "device"
if not device.is_dir():
continue
busy = self.read_number(str(device.relative_to(self.root)), "gpu_busy_percent")
vram_total = self.read_number(str(device.relative_to(self.root)), "mem_info_vram_total")
vram_used = self.read_number(str(device.relative_to(self.root)), "mem_info_vram_used")
if busy is None and vram_total is None:
continue # not a compute-capable GPU (connector-only / virtual)
pci_id = self._read_uevent(device, "PCI_ID") # "1002:7551"
slot = self._read_uevent(device, "PCI_SLOT_NAME") # "0000:4d:00.0"
vendor_id, _, device_id = (pci_id or "").partition(":")
driver = self._driver_name(device)
full_model = self._gpu_model(slot, vendor_id, device_id)
out.append({
"index": len(out),
"card": name,
"vendor": VENDOR_NAMES.get(f"0x{vendor_id}" if vendor_id else "", vendor_id or None),
"model": short_gpu_model(full_model),
"model_full": full_model,
"pci_slot": slot,
"pci_address": pci_id or None,
"driver": driver,
"busy_pct": busy,
"vram_total_bytes": vram_total,
"vram_used_bytes": vram_used,
# On an APU (Strix Halo) the dedicated carve-out is tiny and the model
# actually lives in GTT (shared system RAM): report both, never silently
# add them into one number that hides which pool was used.
"gtt_total_bytes": self.read_number(str(device.relative_to(self.root)),
"mem_info_gtt_total"),
"gtt_used_bytes": self.read_number(str(device.relative_to(self.root)),
"mem_info_gtt_used"),
"power_watts": self._gpu_power(device),
"temps": self._gpu_temps(device),
"clocks": self._gpu_clocks(device),
})
return out
def _gpu_power(self, device: Path) -> float | None:
"""Board power in watts. hwmon `power*_input`/`*_average`, microWatts, PPT preferred.
Not always present: the amdgpu build on driveripper ships power1_label=PPT with no
matching input file, so this honestly returns None instead of inventing a number.
"""
hwmon_dir = device / "hwmon"
if not hwmon_dir.is_dir():
return None
best: float | None = None
for hwmon in hwmon_dir.glob("hwmon*"):
for suffix in ("input", "average", "max"):
for candidate in sorted(hwmon.glob(f"power*_{suffix}")):
key = candidate.stem.split("_")[0]
label_file = hwmon / f"{key}_label"
label = (self.read_text(str(label_file.relative_to(self.root))) or "").strip().lower()
value = self.read_number(str(candidate.relative_to(self.root)))
if value is None:
continue
watts = value / 1_000_000.0
if best is None:
best = watts
if "ppt" in label or "package" in label:
return watts
return best
def _gpu_clocks(self, device: Path) -> dict[str, float]:
"""Clocks in MHz, keyed by their sysfs label (sclk / mclk)."""
clocks: dict[str, float] = {}
hwmon_dir = device / "hwmon"
if not hwmon_dir.is_dir():
return clocks
for hwmon in hwmon_dir.glob("hwmon*"):
for input_file in sorted(hwmon.glob("freq*_input")):
key = input_file.stem.replace("_input", "")
label_file = hwmon / f"{key}_label"
label = (self.read_text(str(label_file.relative_to(self.root))) or key).strip() \
if label_file.exists() else key
value = self.read_number(str(input_file.relative_to(self.root)))
if value is not None:
clocks[label] = value / 1_000_000.0
return clocks
def _read_uevent(self, device: Path, key: str) -> str | None:
for line in self.read_lines(str(device.relative_to(self.root)), "uevent"):
name, _, value = line.partition("=")
if name == key:
return value.strip().lower()
return None
def _driver_name(self, device: Path) -> str | None:
driver = device / "driver"
try:
return os.path.basename(os.readlink(driver))
except OSError:
return None
def _gpu_model(self, slot: str | None, vendor_id: str | None, device_id: str | None) -> str | None:
"""Human GPU name. lspci when it exists (cached: the answer never changes at
runtime), otherwise vendor + device id — never a bare "gpu"."""
cache_key = slot or f"{vendor_id}:{device_id}"
if cache_key in _GPU_MODEL_CACHE:
return _GPU_MODEL_CACHE[cache_key]
model: str | None = None
if slot:
found = self._lspci_name(slot.split(":", 1)[-1] if slot.count(":") == 2 else slot)
if found:
model = found
if model is None:
vendor = VENDOR_NAMES.get(f"0x{vendor_id}" if vendor_id else "")
if vendor and device_id:
model = f"{vendor} GPU ({vendor_id}:{device_id})"
elif vendor:
model = vendor
_GPU_MODEL_CACHE[cache_key] = model
return model
def _lspci_name(self, slot: str) -> str | None:
"""`lspci -nns <slot>` -> the text after the class, e.g. 'Navi 48 [Radeon ...]'.
Optional: lspci is not guaranteed to be installed, so the caller must cope with
None. The subprocess is bounded and never raises into the snapshot.
"""
try:
import subprocess # noqa: PLC0415 - only needed for one lspci call
raw = subprocess.run(["lspci", "-nns", slot], capture_output=True, text=True,
timeout=3, check=False)
except Exception: # noqa: BLE001 - missing lspci / no permissions is normal
return None
if raw.returncode != 0 or not raw.stdout.strip():
return None
line = raw.stdout.strip().splitlines()[0]
_, _, rest = line.partition(":")
_, _, rest = rest.partition(":")
rest = re.sub(r"\s*\[[0-9a-fA-F]{4}:[0-9a-fA-F]{4}\]", "", rest)
rest = re.sub(r"\s*\(rev [0-9a-fA-F]+\)\s*$", "", rest)
return rest.strip() or None
def _gpu_temps(self, device: Path) -> dict[str, float]:
temps: dict[str, float] = {}
hwmon_dir = device / "hwmon"
if not hwmon_dir.is_dir():
return temps
for hwmon in hwmon_dir.glob("hwmon*"):
for input_file in sorted(hwmon.glob("temp*_input")):
label = input_file.stem.replace("_input", "")
named = hwmon / f"{label}_label"
if named.exists():
label = (self.read_text(str(named.relative_to(self.root))) or label).strip()
value = self.read_number(str(input_file.relative_to(self.root)))
if value is not None:
temps[label] = value / 1000.0
return temps
# -- temperatures -------------------------------------------------------- #
def temperatures(self) -> list[dict[str, Any]]:
out: list[dict[str, Any]] = []
hwmon = self._path("sys/class/hwmon")
if hwmon.is_dir():
for unit in sorted(hwmon.glob("hwmon*")):
chip = (self.read_text(str(unit.relative_to(self.root)), "name") or unit.name).strip()
for input_file in sorted(unit.glob("temp*_input")):
key = input_file.stem.replace("_input", "")
label_file = unit / f"{key}_label"
label = chip
if label_file.exists():
label = (self.read_text(str(label_file.relative_to(self.root))) or chip).strip()
value = self.read_number(str(input_file.relative_to(self.root)))
if value is not None:
out.append({"kind": "hwmon", "chip": chip, "label": label,
"celsius": value / 1000.0})
thermal = self._path("sys/class/thermal")
if thermal.is_dir():
for zone in sorted(thermal.glob("thermal_zone*")):
ztype = (self.read_text(str(zone.relative_to(self.root)), "type") or zone.name).strip()
value = self.read_number(str(zone.relative_to(self.root)), "temp")
if value is not None:
out.append({"kind": "thermal", "chip": ztype, "label": ztype,
"celsius": value / 1000.0})
return out
def cpu_temperature(self, temps: list[dict[str, Any]]) -> float | None:
for chip in CPU_TEMP_CHIPS:
picks = [t["celsius"] for t in temps if t["chip"] == chip and t["celsius"] is not None]
if picks:
return max(picks) # Tctl/Tdie style: the hottest sensor on that chip
return None
# -- processes ----------------------------------------------------------- #
def processes(self) -> dict[str, Any]:
"""Top-N by RSS (instant) and by CPU (needs the pid cache — see module docstring)."""
now = time.time()
seen: set[int] = set()
rss_rows: list[dict[str, Any]] = []
cpu_rows: list[dict[str, Any]] = []
total = 0
proc = self._path("proc")
try:
entries = list(proc.iterdir())
except OSError:
return {"total": None, "top_rss": [], "top_cpu": []}
for entry in entries:
if not entry.name.isdigit():
continue
pid = int(entry.name)
stat = self.read_text("proc", entry.name, "stat")
if not stat:
continue
fields = _split_stat(stat)
if not fields:
continue
total += 1
seen.add(pid)
name = fields[0]
utime = _to_float(fields[11]) if len(fields) > 11 else None
stime = _to_float(fields[12]) if len(fields) > 12 else None
jiffies = (utime + stime) if (utime is not None and stime is not None) else None
rss_pages = self._rss_pages(entry.name)
if rss_pages is not None:
rss_rows.append({"pid": pid, "name": name, "rss_bytes": rss_pages * self._page_size})
previous = self._pid_cpu.get(pid)
if jiffies is not None:
if previous is not None and previous[0] < now:
elapsed = now - previous[0]
used = (jiffies - previous[1]) / self._ticks
if elapsed > 0 and used >= 0:
cpu_rows.append({"pid": pid, "name": name,
"cpu_pct": used / elapsed * 100.0})
self._pid_cpu[pid] = (now, jiffies, name)
with self._lock:
for pid in list(self._pid_cpu):
if pid not in seen:
del self._pid_cpu[pid]
rss_rows.sort(key=lambda row: row["rss_bytes"], reverse=True)
cpu_rows.sort(key=lambda row: row["cpu_pct"], reverse=True)
return {"total": total, "top_rss": rss_rows[: self.top_processes],
"top_cpu": cpu_rows[: self.top_processes]}
def _rss_pages(self, pid_dir: str) -> float | None:
"""statm field 2 is resident set size in pages."""
parts = (self.read_text("proc", pid_dir, "statm") or "").split()
return _to_float(parts[1]) if len(parts) > 1 else None
# -- host identity ------------------------------------------------------- #
def host_info(self) -> dict[str, Any]:
uptime = self.read_number("proc/uptime")
if uptime is not None and self._boot_ts is None:
self._boot_ts = time.time() - uptime
release: dict[str, str] = {}
for line in self.read_lines("etc/os-release"):
key, sep, value = line.partition("=")
if sep:
release[key] = value.strip('"')
logical, physical = self.core_counts()
meminfo = self.memory()
try:
uname = os.uname()
except AttributeError: # pragma: no cover - non-POSIX
uname = None
return {
"hostname": socket.gethostname(),
"os_pretty": release.get("PRETTY_NAME") or " ".join(
v for v in (release.get("NAME"), release.get("VERSION")) if v) or None,
"os_id": release.get("ID"),
"kernel": uname.release if uname else None,
"arch": uname.machine if uname else None,
"python": f"{os.sys.version_info.major}.{os.sys.version_info.minor}.{os.sys.version_info.micro}",
"cpu_model": self.cpu_model(),
"cpu_cores_logical": logical,
"cpu_cores_physical": physical,
"memory_total_bytes": meminfo.get("total_bytes"),
"boot_ts": self._boot_ts,
"uptime_seconds": uptime,
"container": self._in_container(),
"agent_version": AGENT_VERSION,
}
def _in_container(self) -> bool:
for marker in ("run/.containerenv", ".dockerenv"):
if self._path(marker).exists():
return True
cgroup = self.read_text("proc/1/cgroup") or ""
return bool(re.search(r"(docker|containerd|kubepods|libpod)", cgroup))
# -- everything ---------------------------------------------------------- #
def snapshot(self, *, top: int | None = None) -> dict[str, Any]:
started = time.perf_counter()
errors: list[str] = []
info = self._safe(self.host_info, "host", errors)
temps = self._safe(self.temperatures, "temperatures", errors) or []
if top is not None:
self.top_processes = top
# no lock here: processes() takes it itself, and threading.Lock is not reentrant
processes = self._safe(self.processes, "processes", errors) or {"total": None,
"top_rss": [],
"top_cpu": []}
disk_io = self._safe(self.disk_io, "disk_io", errors) or {"devices": [], "device_names": []}
# best-effort, after the cheap reads: a wedged disk can cost the pass budget, not
# the snapshot (see the module docstring and smart() for the bounds)
self._safe(lambda: self.attach_smart(disk_io, errors), "smart", errors)
document: dict[str, Any] = {
"schema": SCHEMA_VERSION,
"agent_version": AGENT_VERSION,
"ts": time.time(),
"duration_ms": round((time.perf_counter() - started) * 1000.0, 2),
"host": info or {},
"cpu": self._safe(self.cpu, "cpu", errors) or {},
"memory": self._safe(self.memory, "memory", errors) or {},
"filesystems": self._safe(self.filesystems, "filesystems", errors) or [],
"disk_io": disk_io,
"network": self._safe(self.network, "network", errors) or {"interfaces": []},
"gpus": self._safe(self.gpus, "gpus", errors) or [],
"temperatures": temps,
"cpu_temperature_celsius": self.cpu_temperature(temps),
"processes": processes,
}
# The proxy add-on (phase 28) — the two configs and the live socket table — gated on
# `--no-proxy`: disabled means the keys are **absent** ("not collected"), not empty, and
# a failure inside one is an `available: null` section plus its own one line — never a
# failed snapshot. Inserted before `errors` because the sections write into that list.
if self.proxy_enabled:
document["haproxy"] = self._safe(lambda: self.haproxy(errors), "haproxy", errors) \
or {"available": None, "config": self.haproxy_config,
"frontends": [], "backends": []}
nginx_pair = self._safe(lambda: self.nginx_document(errors), "nginx", errors)
nginx_section, nginx_resolver = nginx_pair or (
{"available": None, "config": self.nginx_config, "files": [],
"servers": [], "upstreams": [], "maps": []}, None)
document["nginx"] = nginx_section
document["sockets"] = self._safe(lambda: self.sockets(errors), "sockets", errors) \
or {"available": None, "listens": [], "established": []}
# The resolver rides with the family and is removed by both flags: `--no-resolve` is
# the operator turning off the one add-on that leaves the host, and `--no-proxy` means
# there are no declared names to resolve in the first place (decision 3). Absent, not
# empty — "not collected" stays a different fact from "collected and empty".
if self.resolve_enabled:
document["resolver"] = self._safe(
lambda: self.resolver(cast("Mapping[str, Any]", document["haproxy"]),
cast("Mapping[str, Any]", nginx_section),
nginx_resolver, errors), "resolver", errors) \
or {"available": None, "via": RESOLVE_VIA, "config_directive": None,
"ts": None, "entries": [], "errors": []}
document["errors"] = errors
return document
def _safe(self, fn: Callable[[], Any], name: str, errors: list[str]) -> Any:
"""One failing collector degrades its own section to empty; never the snapshot."""
try:
return fn()
except Exception as exc: # noqa: BLE001 - a bad /proc read must not fail the poll
errors.append(f"{name}: {type(exc).__name__}: {exc}")
return None
MODE_NAMES = ("user", "nice", "system", "idle", "iowait", "irq", "softirq", "steal",
"guest", "guest_nice")
MEMORY_KEYS = ("total_bytes", "free_bytes", "available_bytes", "used_bytes", "buffers_bytes",
"cached_bytes", "dirty_bytes", "swap_total_bytes", "swap_used_bytes",
"committed_bytes")
_SMART_FAILURE_HINT_RE = re.compile(r"failed|denied|error|unable|unsupported|not supported", re.I)
def smart_failure_line(returncode: int, stdout: str | None, stderr: str | None) -> str:
"""The sentence that says *why* smartctl answered nothing.
Never its first line: smartctl opens every run with its version banner, and errors
land on stdout, on stderr, or both. Take the first line that reads like a failure,
else the last non-empty line, else the exit status — an error entry that quotes the
copyright line is worse than no error entry.
"""
blobs = [stderr or "", stdout or ""]
for blob in blobs:
for line in blob.splitlines():
if line.strip() and _SMART_FAILURE_HINT_RE.search(line):
return line.strip()
for blob in blobs: # nothing that reads like a failure: last real line of stderr first
lines = [line.strip() for line in blob.splitlines() if line.strip()]
if lines:
return lines[-1]
return f"exit status {returncode}"
def _smart_number(text: str) -> float | None:
"""`8,760` and `38` are numbers; a hex-packed vendor raw is not one we can decode
without a drive-specific table, so it stays None — an invented sector count is the
exact failure mode this agent exists to avoid."""
return _to_float(text.strip().replace(",", ""))
def _smart_raw_number(raw: str) -> float | None:
"""The value of a SMART attribute row's RAW_VALUE column (column 9, `raw`).
smartctl prints one bare number for most attributes and the shapes above for the rest,
all of which are read here; `0x...` is a packed raw, and a packed raw read as its
leading `0` would report a healthy-looking zero for a value we never decoded, so that
one answers None."""
token = raw.strip().split(" ")[0]
if not token or token.lower().startswith("0x"):
return None
match = _SMART_RAW_NUMBER_RE.match(token)
return _smart_number(match.group(0)) if match else None
def parse_smart_output(text: str | None) -> dict[str, Any]:
"""Parse `smartctl -i -A -H` stdout (ATA, NVMe and SCSI share this entry point).
Only labels the deployed smartmontools 7.5 actually prints are matched (they were
verified against that binary's own format strings and drive database, and are pinned
by tests/fixtures/smartctl_*.txt). Values the drive or the parser cannot supply stay
None. `overall_health` keeps smartctl's own verdict word (`PASSED`, `FAILED`,
`UNKNOWN`, trailing `!` stripped); NVMe `offline_uncorrectable` carries the health
log's `Media and Data Integrity Errors` counter — same meaning, different name, and
the ATA `Offline_Uncorrectable` attribute when both exist wins because they are the
drive's own words for it. ATA rows are read from the RAW_VALUE column rather than the
end of the line, because this binary annotates that column — `35 (Min/Max 34/44)` on a
Seagate, `43894h+43m+32.519s` on a Western Digital — and the last token of such a row
is not the value.
"""
out: dict[str, Any] = dict.fromkeys(SMART_FIELDS)
if not text:
return out
for line in text.splitlines():
stripped = line.strip()
if not stripped:
continue
label, sep, value = stripped.partition(":")
if sep:
label, value = label.strip(), value.strip()
if label in ("Device Model", "Model Number"):
if value:
out["model"] = value
elif label == "Serial Number":
if value:
out["serial"] = value
elif label == "Temperature": # NVMe health log: `38 Celsius`
number = _smart_number(value.split()[0]) if value.split() else None
if number is not None:
out["temp_c"] = number
elif label == "Power On Hours":
out["power_on_hours"] = _smart_number(value)
elif label == "Percentage Used":
out["percentage_used"] = _smart_number(value.rstrip("%"))
elif label == "Media and Data Integrity Errors" and out["offline_uncorrectable"] is None:
out["offline_uncorrectable"] = _smart_number(value)
health = _SMART_HEALTH_RE.match(stripped)
if health:
out["overall_health"] = health.group(1).rstrip("!")
continue
fields = stripped.split()
if fields and fields[0].isdigit() and len(fields) >= 6:
attribute = _SMART_ATTR_FIELDS.get(fields[1])
if attribute is not None:
# ID NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE —
# the raw is column 9, and reading the *last* token instead loses every
# drive that annotates it: `35 (Min/Max 34/44)` ends in `44)`.
number = _smart_raw_number(fields[9] if len(fields) > 9 else fields[-1])
if number is not None:
out[attribute] = number
return out
# /proc/<pid>/stat comm is in parens and may itself contain spaces and parens.
_STAT_COMM_RE = re.compile(r"^\d+ \((?P<comm>.*)\) (?P<rest>.*)$", re.S)
def _split_stat(stat: str) -> list[str] | None:
match = _STAT_COMM_RE.match(stat.strip())
if not match:
return None
return [match.group("comm"), *match.group("rest").split()]
_HAPROXY_ADDR_PORT = re.compile(r"^(?P<addr>.*):(?P<port>[0-9]+)$")
def _haproxy_addr_port(token: str) -> tuple[str | None, int] | None:
"""`:::22` → `("::", 22)`, `[::1]:8080` → `("[::1]", 8080)`, `:80` → `(None, 80)`.
Split at the **last** colon, because that is where haproxy's own address grammar puts the
port: an IPv6 bind survives it (`:::22` is the address `::` on port `22`) and a bracketed one
survives it, while splitting at the first colon would call the address of `[::1]:8080` just
`[`. A token whose tail is not digits has no port for this parser — haproxy allows bind forms
with none (`unix@/run/sock.sock`) and port ranges (`:8000-8100`) — and the caller skips those
rather than invent one. An address the config left empty (`bind :8080`) is `None`: "every
address" is what haproxy means there, and `*` is a spelling it did not use.
"""
match = _HAPROXY_ADDR_PORT.match(token)
if match is None:
return None
return (match.group("addr") or None), int(match.group("port"))
def _haproxy_backend_name(words: list[str]) -> str | None:
"""The backend a `default_backend`/`use_backend` line names, or None when it names none.
haproxy also lets the target be computed per request (`use_backend %[req.hdr(host),
map_dom(...)]`, `use_backend { bk1 } if ...`). That is not a name, and reporting the
expression as one would put a fetch expression where the dashboard expects a backend, so it
contributes nothing: the config does not say which backend serves that request.
"""
if len(words) < 2:
return None
name = words[1]
return None if name.startswith(("%", "{", '"')) else name
def _scan_haproxy_config(text: str) -> tuple[list[dict[str, Any]], list[dict[str, Any]], bool]:
"""One pass over a haproxy config: (frontends, backends, whether it is one at all).
A haproxy config is a list of sections whose bodies are **indented**, so the indent is the
block structure and the first word of a top-level line is the section's kind. That is the
whole parser: a top-level line switches the current section (`frontend`, `backend`,
`listen`) or ends it (anything else, which is how `global`, `defaults` and every section this
page does not model are skipped), and an indented line is a directive of the current section —
`bind`, `default_backend`, `use_backend` on a frontend, `server` on a backend. Everything else
is ignored with no error: haproxy has hundreds of directives and this section is about what
listens and where it sends, not about what `timeout client` says.
"""
frontends: list[dict[str, Any]] = []
backends: list[dict[str, Any]] = []
recognised = False
current_fe: dict[str, Any] | None = None
current_be: dict[str, Any] | None = None
for line in text.splitlines():
stripped = line.strip()
if not stripped or stripped.startswith("#"):
continue # a blank line keeps its section open; a comment holds no directive
words = stripped.split()
directive = words[0]
if not line[:1].isspace():
current_fe = current_be = None # a top-level line ends the previous body
if directive in HAPROXY_KEYWORDS:
recognised = True
if len(words) < 2:
continue
name = words[1]
if directive in ("frontend", "listen"):
current_fe = {"name": name, "binds": [], "default_backend": None, "backends": []}
frontends.append(current_fe)
if directive in ("backend", "listen"):
current_be = {"name": name, "servers": []}
backends.append(current_be)
if directive == "listen":
# haproxy's own shorthand: a `listen` section *is* a frontend and a backend
# sharing one name, so the pair is reported as such and the routing name is
# the section's — the config's structure, not an inference about a request.
current_fe["default_backend"] = name
current_fe["backends"] = [name]
continue
if current_fe is not None:
if directive == "bind":
for token in words[1:]:
found = _haproxy_addr_port(token)
if found is not None:
current_fe["binds"].append({"addr": found[0], "port": found[1]})
break # haproxy writes the address first; whatever follows is options
elif directive == "default_backend":
current_fe["default_backend"] = _haproxy_backend_name(words)
elif directive == "use_backend":
backend = _haproxy_backend_name(words)
if backend is not None and backend not in current_fe["backends"]:
current_fe["backends"].append(backend)
if current_be is not None and directive == "server" and len(words) > 2:
target = _haproxy_addr_port(words[2])
current_be["servers"].append({
"name": words[1],
# a target with no parsable port keeps the text the config wrote and a null
# port — "absent is never 0", on the one field haproxy makes optional
"target": words[2] if target is None else target[0],
"port": None if target is None else target[1],
})
return frontends, backends, recognised
def parse_haproxy_config(text: str) -> dict[str, Any]:
"""Parse a haproxy config's **text** into the `haproxy` section's data. Pure, never raises.
`{frontends: [{name, binds: [{addr, port}], default_backend, backends}], backends:
[{name, servers: [{name, target, port}]}], recognised: bool}` — the frontend→backend routing
is the names themselves (`default_backend` plus every `use_backend`), so the dashboard joins
them and this file interprets no rule.
`recognised` answers the separate question "is this text a haproxy config at all" (does its
top level name a section keyword?) and is deliberately **not** part of the served section:
`Collector.haproxy` turns it into `available`, the only verdict taken here.
"""
frontends, backends, recognised = _scan_haproxy_config(text)
return {"frontends": frontends, "backends": backends, "recognised": recognised}
# --------------------------------------------------------------------------- #
# nginx config: the tokenizer, the token parsers, the block walk
# --------------------------------------------------------------------------- #
def _nginx_tokens(text: str) -> list[tuple[str, str]]:
"""Split nginx config text into `("w", word)`, `("{", "")`, `("}", "")`, `(";", "")`.
nginx's grammar is words closed by `;` or opened by `{`, with `#` comments and quoted
strings that may contain any of those three — the captured gateway config writes
`add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;`, whose
value holds a semicolon and a space. So the three cases are handled before any split:
cutting on `;` alone would halve every `add_header`, and cutting on whitespace would turn
one quoted value into three words. Raises `ValueError` only for an unterminated quote;
every other shape is somebody's config and comes back as tokens.
A `#` opens a comment only where a word would have started, which is where nginx's own
lexer reads one: `proxy_pass http://host/#/app` keeps its fragment, and the
`# include /etc/nginx/old.conf;` line an operator commented out contributes nothing.
"""
tokens: list[tuple[str, str]] = []
word: list[str] = []
started = False # an empty quoted string is still a word: the `''` key of a map
i, n = 0, len(text)
while i < n:
char = text[i]
if char == "#" and not started and (i == 0 or text[i - 1] in " \t\r\n"):
while i < n and text[i] != "\n":
i += 1
continue
if char in "\"'":
quote = char
i += 1
while i < n:
if text[i] == "\\" and i + 1 < n:
word.append(text[i + 1])
i += 2
continue
if text[i] == quote:
break
word.append(text[i])
i += 1
else:
raise ValueError("a quoted string is never closed")
started = True
i += 1 # past the closing quote
continue
if char in "{};":
if started:
tokens.append(("w", "".join(word)))
word, started = [], False
tokens.append((char, ""))
i += 1
continue
if char.isspace():
if started:
tokens.append(("w", "".join(word)))
word, started = [], False
i += 1
continue
word.append(char)
started = True
i += 1
if started:
tokens.append(("w", "".join(word)))
return tokens
def _nginx_statements(text: str) -> list[tuple[Any, ...]]:
"""The brace-block scanner: tokens in, nested statements out. Raises on an unbalanced file.
A statement is `("d", name, args)` for a directive or `("b", name, args, children)` for a
block. That is the whole difference between the two config languages this agent reads:
haproxy's structure is its indentation, nginx's is its braces, so nginx gets a scanner
instead of a line loop — and with it the two shapes that make the language what it is, a
`server { … location { … } … }` nested as deep as it goes and an `upstream` body whose
`server` lines are *targets* while an `http` body's `server {` is a *block*.
`ValueError` here means "nginx itself would refuse to start on this file": a stray brace or
a directive that runs into the end of the file without its `;`. The caller turns that into
the section's `null` arm — a config that cannot parse is a fact about the config, and the
snapshot never learns the difference.
"""
root: list[tuple[Any, ...]] = []
containers: list[list[tuple[Any, ...]]] = [root]
words: list[str] = []
for kind, value in _nginx_tokens(text):
if kind == "w":
words.append(value)
continue
if kind == ";":
if words: # a bare `;` is whitespace to nginx
containers[-1].append(("d", words[0], words[1:]))
words = []
continue
if kind == "{":
if not words:
raise ValueError("a '{' with no block name before it")
block: tuple[Any, ...] = ("b", words[0], words[1:], [])
containers[-1].append(block)
containers.append(cast("list[tuple[Any, ...]]", block[3]))
words = []
continue
if words:
raise ValueError(f"a closing brace ends the block while {words[0]!r} is still open")
if len(containers) == 1:
raise ValueError("a closing brace with no opening one before it")
containers.pop()
if words:
raise ValueError(f"the file ends inside the directive {words[0]!r}, expecting ';' "
"or a closing brace")
if len(containers) != 1:
raise ValueError("an opening brace is never closed")
return root
#: The shape `_nginx_statements` returns, spelled once so the walkers below say what they take.
_NginxStatements = list[tuple[Any, ...]]
def _nginx_addr_port(token: str) -> tuple[str | None, int] | None:
"""`80` → `(None, 80)`, `[::]:8443` → `("::", 8443)`, `10.1.0.9:8080` → itself.
The three forms `listen` writes, in the spelling the config used: a bare port leaves the
address `None` (nginx's own "every address", and `*` is a spelling it did not use), and a
bracketed IPv6 address loses its brackets because the brackets are nginx's disambiguation
of the colon, not part of the address. A token with no digits at the end has no port here —
`unix:/run/nginx.sock`, a port range, a `default_server` written where an address belongs —
and the caller skips it rather than inventing one.
"""
if token.startswith("["):
host, closed, tail = token[1:].partition("]")
if not closed or not tail.startswith(":"):
return None
addr, port_text = host, tail[1:]
elif ":" in token:
addr, _, port_text = token.rpartition(":")
else:
addr, port_text = "", token
if not port_text.isdigit():
return None
return (addr or None), int(port_text)
def _nginx_variable(token: str) -> str:
"""The name of a nginx variable, without its `$` — the one spelling both halves agree on.
The config writes `$targetBackend` in the `map` line and again in `proxy_pass`, and the
dashboard joins the two. Serving one with the sigil and one without would make that join a
string fix-up in the page, so the `$` — nginx's syntax for "this is a variable", not part of
its name — is dropped here, once.
"""
return token[1:] if token.startswith("$") else token
def _nginx_proxy_pass(token: str, location: str | None) -> dict[str, Any]:
"""One `proxy_pass` line: the address it names, the variable it names, or neither.
`proxy_pass http://nextcloud.example.com:11000;` is an address, and `proxy_pass
$targetBackend;` is not one — nginx resolves the variable per connection from the `map`
block that defines it — so the second is served with `variable` set and `target: null`. The
agent reports the two facts the config states and never merges them: a merged target would
be a number this wire never published (and the map's `default` is not "the" backend, it is
what a name nobody mapped gets). A trailing path (`/api/`) is dropped with the same reason
as haproxy's `check inter 5s`: it changes what is requested, not where the pipe lands.
`location` is the enclosing `location` path, or `None` for the stream form, where the
directive is written in the `server` block itself and there is no path to name.
"""
entry: dict[str, Any] = {"location": location, "scheme": None, "target": None,
"variable": None}
if token.startswith("$"):
entry["variable"] = _nginx_variable(token)
return entry
scheme, sep, rest = token.partition("://")
if not sep:
# Not `scheme://…`: a unix socket (`unix:/run/x.sock`) is not a host and a bare word is
# not an address, so the line is reported as a proxy_pass with nothing resolved instead
# of guessing which of the two it is.
return entry
entry["scheme"] = scheme
authority = rest.split("/", 1)[0].split("?", 1)[0].split("#", 1)[0]
found = _nginx_addr_port(authority)
if found is not None:
entry["target"] = {"host": found[0], "port": found[1]}
elif ":" not in authority and authority:
# `proxy_pass http://backend_pool` names an `upstream` block, which is a host with no
# port: the name is the fact, and the port it does not state stays null.
entry["target"] = {"host": authority, "port": None}
return entry
def _collect_nginx(statements: _NginxStatements, out: dict[str, Any], *, root: bool,
server: dict[str, Any] | None = None, location: str | None = None,
upstream: dict[str, Any] | None = None,
map_ctx: dict[str, Any] | None = None) -> None:
"""Walk one statement list, filling `out`. Pure over the parsed tree; interprets nothing.
What is *not* here matters as much as what is: no `include` is followed below the level the
caller passed (that is the collector's bounded job, and `root=False` for an included file is
what makes one level structural rather than a promise), no `listen` flag is read (`ssl`,
`proxy_protocol`, `http2` change how the bytes travel, not what the page's pipes show), and
a directive outside a `server` block is ignored instead of being promoted into a synthetic
one — there is no such thing as a proxy_pass with no server.
"""
for statement in statements:
kind, name, args = statement[0], statement[1], statement[2]
if kind == "d":
if name == "include" and args:
if root:
out["includes"].append(args[0])
continue
if upstream is not None:
if name == "server" and args:
found = _nginx_addr_port(args[0])
upstream["servers"].append({
# an upstream body's `server` line *is* a host, so a target with no port
# keeps its name and a null port rather than losing the line entirely
"host": args[0] if found is None else found[0],
"port": None if found is None else found[1],
})
continue
if map_ctx is not None:
# Inside a `map` body the *directive name* is the key and its first argument is
# the value — `map $ssl_preread_server_name $targetBackend { a.example.com
# backend.internal:443; }` is one entry, not a directive called `a.example.com`.
found = _nginx_addr_port(args[0]) if args else None
map_ctx["entries"].append({
"key": name,
# only an address becomes a target: the `$connection_upgrade` map's
# `upgrade` / `close` values are header words, and calling one a host
# would be this section inventing a fact to fill a field
"target": None if found is None else {"host": found[0], "port": found[1]},
})
continue
if server is not None:
if name == "listen" and args:
found = _nginx_addr_port(args[0])
if found is not None:
server["listen"].append({"addr": found[0], "port": found[1]})
elif name == "server_name":
# every name on the line, and the catch-all `_` kept exactly as written —
# "any domain" is what the page says about it, not what the agent decides
server["server_names"].extend(args)
elif name == "proxy_pass" and args:
server["proxy_passes"].append(_nginx_proxy_pass(args[0], location))
continue
children = cast("_NginxStatements", statement[3])
if name == "upstream" and args:
opened: dict[str, Any] = {"name": args[0], "servers": []}
out["upstreams"].append(opened)
_collect_nginx(children, out, root=root, upstream=opened)
elif name == "map" and len(args) >= 2:
opened_map: dict[str, Any] = {"variable": _nginx_variable(args[1]), "entries": []}
out["maps"].append(opened_map)
_collect_nginx(children, out, root=root, map_ctx=opened_map)
elif name == "server":
opened_server: dict[str, Any] = {"listen": [], "server_names": [], "proxy_passes": []}
out["servers"].append(opened_server)
_collect_nginx(children, out, root=root, server=opened_server)
elif name == "location":
_collect_nginx(children, out, root=root, server=server,
location=(" ".join(args) or None))
else:
_collect_nginx(children, out, root=root, server=server, location=location)
def nginx_include_paths(text: str) -> list[str]:
"""Every `include` argument the main config names, in document order. Never raises.
The collector needs the list *before* it can read anything, and a config that will not
tokenize has no include lines to ask about — it comes back empty, and the parse failure is
reported once, by `parse_nginx_config`, rather than twice by two functions that disagree.
Includes nested inside `http { }` / `stream { }` count: that is exactly where the captured
gateway config keeps the two lines that hold its whole ingress map.
"""
try:
statements = _nginx_statements(text)
except ValueError:
return []
out: dict[str, Any] = {"servers": [], "upstreams": [], "maps": [], "includes": []}
_collect_nginx(statements, out, root=True)
return cast("list[str]", out["includes"])
def _nginx_include_matches(arg: str, included: dict[str, str]) -> list[str]:
"""Which of the supplied documents this `include` argument names.
Exact for a literal path, `fnmatch` for a glob, in the order the documents were handed over
(the collector reads them sorted, so that order is stable). `fnmatch`'s `*` crosses `/`
where the C library's `glob()` does not, which costs nothing here: the collector did the
real globbing and this only decides which side of the join a document belongs to.
"""
if any(c in arg for c in _GLOB_SPECIALS):
return [path for path in included if fnmatch.fnmatchcase(path, arg)]
return [path for path in included if path == arg]
def parse_nginx_config(main_text: str,
included: dict[str, str] | None = None) -> dict[str, Any]:
"""Parse an nginx main config plus the documents its includes name. Pure, never raises.
`{servers: [{listen: [{addr, port}], server_names: [str], proxy_passes: [{location, scheme,
target, variable}]}], upstreams: [{name, servers: [{host, port}]}],
maps: [{variable, entries: [{key, target}]}], files: [str], includes: [str], unparsed:
[str], recognised: bool, error: str|null, resolver: str|null}`.
Four of those keys are the ones `Collector.nginx` does not serve, the way
`parse_haproxy_config`'s `recognised` is not: `error` is why the main text could not be
parsed, `unparsed` names an included document that could not, `includes` is the raw
argument list (a caller that has no filesystem — every unit test — still gets to see what a
config *would* have read), and `resolver` is the config's own `resolver` directive's
arguments, which the `resolver` section serves instead (`Collector.nginx_document`).
`files` is the included documents that were merged, in order;
the collector puts the main config in front of it to make the section's `files`.
A document that fails to parse is dropped and named in `unparsed`; it never drops the others
and never raises. The main text failing to parse is the one case that yields an empty result,
because there is nothing to merge it with — and even then the answer is this dict with
`error` set, not an exception on a snapshot path.
"""
out: dict[str, Any] = {"servers": [], "upstreams": [], "maps": [], "files": [],
"includes": [], "unparsed": [], "recognised": False, "error": None,
"resolver": None}
try:
main = _nginx_statements(main_text)
except ValueError as exc:
out["error"] = str(exc)
return out
out["recognised"] = any(statement[1] in NGINX_CONTEXTS for statement in main)
# The config's own `resolver` line, for the `resolver` section's `config_directive`. It is
# found here rather than served here on purpose: the `nginx` section stays byte-stable, so
# this phase's additivity control is a strict top-level key comparison with no exception list.
out["resolver"] = _find_resolver_statement(main)
_collect_nginx(main, out, root=True)
documents = included or {}
merged: list[str] = []
seen: set[str] = set()
for arg in cast("list[str]", out["includes"]):
for path in _nginx_include_matches(arg, documents):
if path in seen:
continue
seen.add(path)
try:
statements = _nginx_statements(documents[path])
except ValueError:
out["unparsed"].append(path)
continue
merged.append(path)
if out["resolver"] is None:
out["resolver"] = _find_resolver_statement(statements)
# root=False: an include line inside an included file is a second level, and this
# pass is one level deep by design (see NGINX_INCLUDE_MAX_FILES).
_collect_nginx(statements, out, root=False)
out["files"] = merged
return out
# --------------------------------------------------------------------------- #
# live sockets (the additive `sockets` section)
# --------------------------------------------------------------------------- #
def _proc_net_addr(raw: str, family: str) -> str | None:
"""One `/proc/net/tcp` address word as the address it actually is.
`/proc` writes the address as one 32-bit word per group, each group little-endian, in
hex: `0100007F` is 127.0.0.1 and v6 gets four of those groups. Unrolling therefore means
reversing the bytes **inside every group of four**, which is what `socket.inet_ntop` then
renders in the family's own spelling — so a wildcard stays the literal `0.0.0.0` / `::` the
dashboard's inbound join keys on and a v6 address arrives in the form the nginx config
writes it in (`[::]:8443`), not a hex blob the page would have to understand twice. A word
of the wrong length or with non-hex in it is `None`: an address nobody published is worse
than a gap beside a real socket.
"""
if len(raw) != (32 if family == "v6" else 8):
return None
try:
packed = bytes.fromhex(raw)
except ValueError:
return None
unrolled = b"".join(packed[i:i + 4][::-1] for i in range(0, len(packed), 4))
try:
return socket.inet_ntop(socket.AF_INET6 if family == "v6" else socket.AF_INET,
unrolled)
except (OSError, ValueError):
return None
def _proc_net_endpoint(token: str, family: str) -> tuple[str | None, int | None]:
"""One `ADDRESS:PORT` column into `(address, port)`; either half may be `None`.
The port is hex (`221B` is 8731) and travels after the last colon, because a v6 address
has colons of its own in the canonical spelling `/proc` would have used. A token with no
colon is not an endpoint, and a port that is not hex is `None` rather than 0 — the haproxy
`server` line with no port keeps a null port for the same reason.
"""
addr_hex, sep, port_hex = token.rpartition(":")
if not sep:
return None, None
port: int | None
try:
port = int(port_hex, 16)
except ValueError:
port = None
return _proc_net_addr(addr_hex, family), port
def parse_proc_net_tcp(text: str, family: str) -> tuple[list[dict[str, Any]],
list[dict[str, Any]]]:
"""Parse one `/proc/net/tcp{,6}` table into `(listens, established)`. Pure, never raises.
`family` is the label the rows carry ("v4"/"v6") and decides how the address word is
unrolled. Only the two states a gateway page can use are returned: `0A` LISTEN becomes
`{family, addr, port}` and `01` ESTABLISHED becomes
`{family, local: {addr, port}, remote: {addr, port}}`. Every other state — `06` TIME_WAIT,
`08` CLOSE_WAIT, the SYN queue — is a socket in transit, and reporting it would bury the
live pairs in rows that stopped being true before the page drew them; skipping one costs no
`errors` line because the kernel is not broken, it is busy.
`/proc`'s own header line costs nothing special: its state column reads `st`, which is not
a state, so the filter above drops it. A row whose endpoint will not decode is still
reported, with `addr`/`port` null — the socket exists, and the gap is the honest fact about
the word, not about the connection.
"""
listens: list[dict[str, Any]] = []
established: list[dict[str, Any]] = []
for line in text.splitlines():
fields = line.split()
if len(fields) < 4:
continue
state = fields[3].upper()
if state == PROC_NET_STATE_LISTEN:
addr, port = _proc_net_endpoint(fields[1], family)
listens.append({"family": family, "addr": addr, "port": port})
elif state == PROC_NET_STATE_ESTABLISHED:
local_addr, local_port = _proc_net_endpoint(fields[1], family)
remote_addr, remote_port = _proc_net_endpoint(fields[2], family)
established.append({
"family": family,
"local": {"addr": local_addr, "port": local_port},
"remote": {"addr": remote_addr, "port": remote_port},
})
return listens, established
def socket_gap_line(missing: list[str], unreadable: list[tuple[str, str]],
nothing_answered: bool) -> str:
"""The one `sockets:` line for the tables that did not answer, whichever arms they are.
A missing table and a denied one are two sentences — the first is `/proc` as shipped (a
kernel without IPv6 has no `tcp6`), the second is a read this process was not allowed — and
the line names each path with its own reason. When nothing answered the line says the whole
section is empty and names `--no-proxy`, the way the two config sections do, so the operator
reading one line knows whether to fix a mount or turn the section off.
"""
parts = [f"{path} not found" for path in missing]
parts.extend(f"{path}: {reason}" for path, reason in unreadable)
tail = (" — this host publishes no kernel socket table; --no-proxy opts out"
if nothing_answered else " — the tables that did answer are reported in full")
return f"sockets: {', '.join(parts)}{tail}"
def socket_truncation_line(paths: list[str]) -> str:
"""The one `sockets:` line for a table longer than the read bound.
It names the bound, because that is the number an operator would want to raise (in the
source, since there is no flag for it) and the difference between "the gateway has 4096
sockets" and "the gateway has more sockets than this read takes".
"""
return (f"sockets: {', '.join(paths)} hold more than the {PROC_NET_MAX_LINES} lines this "
"read takes — the sockets past the bound are not in this section")
# --------------------------------------------------------------------------- #
# name resolution (the additive `resolver` section)
# --------------------------------------------------------------------------- #
def address_family(value: str) -> str | None:
"""`v4`, `v6`, or None — is this text already an address?
`socket.inet_pton` and nothing else: it is the same test the `sockets` section's decoder
uses, it needs no second module (the agent's import footprint is pinned by
`tests/test_agent.py`), and it is the rule that lets an IP literal in a config be reported
as what it is instead of dialled. An interface scope (`fe80::1%eth0`) is not a pure address,
so it comes back None and travels through the resolver like any other name — libc decides
what a scoped literal means, and it is not this file's job to invent an answer.
"""
for family, label in ((socket.AF_INET, "v4"), (socket.AF_INET6, "v6")):
try:
socket.inet_pton(family, value)
except (OSError, ValueError):
continue
return label
return None
def resolver_key(name: str, port: int | None) -> str:
"""One declared target, as this module's cache key.
The port belongs in the key because a name on two ports is two targets: this fleet routes
`minecraft.reeselink.com` to 25565 and to 25566, two services, and one entry for both would
be one answer to a question nobody asked.
"""
return f"{name}:{port}"
def _resolver_sort_key(value: str) -> tuple[int, bytes]:
"""v4 ascending then v6 ascending, with anything undecodable last and stable.
The order matters for the payload rather than for the lookup: `getaddrinfo` returns whatever
order this host's resolver and `/etc/gai.conf` feel like today, so an unsorted set makes two
polls of a healthy gateway differ for no reason the operator could act on.
"""
for family, rank in ((socket.AF_INET, 0), (socket.AF_INET6, 1)):
try:
return (rank, socket.inet_pton(family, value))
except (OSError, ValueError):
continue
return (2, value.encode("utf-8", errors="replace"))
def sort_resolver_addresses(values: Iterable[str]) -> list[str]:
"""Unique addresses in the payload's order: family first, then the address itself."""
return sorted(set(values), key=_resolver_sort_key)
def target_names(haproxy: Mapping[str, Any] | None,
nginx: Mapping[str, Any] | None) -> list[tuple[str, int | None]]:
"""Every `(name, port)` the two **parsed** proxy configs write as a target, no duplicates.
This is the whole allow-list for the resolver (decision 3 of the phase): haproxy's
`backend`/`server` targets first (config order), then nginx's `upstream` members, its
server-block `proxy_pass` targets, and its `map` entry targets, deduplicated keeping
first-seen order. Nothing else is ever in here — not a name from `/proc/net/tcp`, not a
client address, not anything a request supplied — because resolving a name nobody configured
is a round trip that produces a fact about the internet instead of a fact about this
gateway, and `tests/test_agent.py` pins that the lookups are exactly this list.
A `proxy_pass $var` contributes nothing (the map decides, and the map's own entries are
here); an entry whose target is not an address at all is `target: null` upstream and is
skipped here for the same reason the `nginx` section refused to name it.
"""
targets: list[tuple[str, int | None]] = []
seen: set[tuple[str, int | None]] = set()
def add(host: Any, port: Any) -> None:
if not isinstance(host, str) or not host:
return
pair = (host, port if isinstance(port, int) and not isinstance(port, bool) else None)
if pair in seen:
return
seen.add(pair)
targets.append(pair)
for block in _mapping_list(haproxy, "backends"):
for server in _mapping_list(block, "servers"):
add(server.get("target"), server.get("port"))
for upstream in _mapping_list(nginx, "upstreams"):
for member in _mapping_list(upstream, "servers"):
add(member.get("host"), member.get("port"))
for server in _mapping_list(nginx, "servers"):
for hop in _mapping_list(server, "proxy_passes"):
target = hop.get("target")
if isinstance(target, Mapping):
add(target.get("host"), target.get("port"))
for block in _mapping_list(nginx, "maps"):
for entry in _mapping_list(block, "entries"):
target = entry.get("target")
if isinstance(target, Mapping):
add(target.get("host"), target.get("port"))
return targets
def _mapping_list(section: Any, key: str) -> list[Mapping[str, Any]]:
"""One list of mappings out of a section that may be absent, null, or the wrong shape.
The section is built by this file, so anything else means a caller handed this function
something it did not build — and the honest answer to that is an empty list, not a
`TypeError` on a snapshot path.
"""
if not isinstance(section, Mapping):
return []
rows = section.get(key)
if not isinstance(rows, list):
return []
return [row for row in rows if isinstance(row, Mapping)]
def _section_answered(section: Any) -> bool:
"""Did this proxy section report that it read its config? `available` is the section's own answer."""
return isinstance(section, Mapping) and section.get("available") is True
def resolver_not_found_line(haproxy: Any, nginx: Any) -> str:
"""The one `resolver:` line for the arm where there was nothing to resolve.
It says *not found* when a config is simply absent — the pinned cross-module wording the
dashboard reads to answer "is this proxy installed here?" (the `haproxy:`, `nginx:` and
`smartctl:` lines share it) — and *unreadable* when the file is there and this process could
not read it, because those are two different operator actions. One line either way: the two
sections have already reported their own reasons, and a third restatement per snapshot is the
ink the empty-state rules exist to avoid.
"""
reasons = []
for name, section in (("haproxy", haproxy), ("nginx", nginx)):
available = section.get("available") if isinstance(section, Mapping) else None
reasons.append(f"{name} not found" if available is False else f"{name} unreadable")
return (f"resolver: nothing to resolve — {' and '.join(reasons)}; --no-resolve opts out")
def _find_resolver_statement(statements: _NginxStatements) -> str | None:
"""Depth-first search for one `resolver` directive, in document order.
A `map` body is not searched: inside one, a bare word is a **key**, so a map with a
`resolver` key (a header name, a hostname someone routed on) would otherwise be reported as
this box's DNS server.
"""
for statement in statements:
kind, name, args = statement[0], statement[1], statement[2]
if kind == "d":
if name == "resolver" and args:
return " ".join(str(arg) for arg in args)
continue
if name == "map":
continue
found = _find_resolver_statement(cast("_NginxStatements", statement[3]))
if found is not None:
return found
return None
# --------------------------------------------------------------------------- #
# TLS bundle: what the dashboard signs, what the installer places
# --------------------------------------------------------------------------- #
#: The two metadata lines inside a bundle, spelled exactly as `src/dashboards/certs.py` writes
#: them. The agent stays stdlib (no `cryptography` on a monitored host), so the format is agreed
#: here by text rather than imported, and `tests/test_agent.py` parses a bundle that `certs.py`
#: actually produced — the only way the two halves can stay honest.
BUNDLE_REV_LINE = "# BUNDLE-REV "
REVOKED_LINE = "# REVOKED "
#: Where the bundle is unpacked when systemd gave the unit a `RuntimeDirectory` (task 06): tmpfs,
#: private to the service user, gone when the unit stops. Otherwise a `mkdtemp` directory that is
#: removed as soon as the SSL context holds the material.
BUNDLE_RUNTIME_DIR = Path("/run/dashboards-agent")
_PEM_BLOCK_RE = re.compile(r"-----BEGIN [A-Z0-9 ]+-----.*?-----END [A-Z0-9 ]+-----\n", re.DOTALL)
_CERT_HEADER = "-----BEGIN CERTIFICATE-----"
#: The unpacked file names, in the order `_bundle_blocks` hands the blocks back.
_BUNDLE_FILES = ("agent-cert.pem", "agent-key.pem", "root-ca.pem")
def _line_value(line: str, prefix: str) -> str:
"""The first token after a bundle line's prefix, or "" when the line carries nothing."""
rest = line[len(prefix):].strip()
fields = rest.split()
return fields[0] if fields else ""
def bundle_revision(text: str) -> int | None:
"""The `# BUNDLE-REV <n>` number, or None when the line is absent or is not a number.
Absent is an error rather than a default of 0: the revision is how the dashboard tells a stale
bundle from the one it just issued, so an agent serving "rev 0" would be reporting a number it
never had.
"""
for line in text.splitlines():
stripped = line.strip()
if not stripped.startswith(BUNDLE_REV_LINE):
continue
token = _line_value(stripped, BUNDLE_REV_LINE)
try:
return int(token)
except ValueError:
return None
return None
def bundle_revoked(text: str) -> frozenset[str]:
"""Every `# REVOKED <serial>` value in a bundle, kept exactly as written."""
serials: set[str] = set()
for line in text.splitlines():
stripped = line.strip()
if stripped.startswith(REVOKED_LINE):
serial = _line_value(stripped, REVOKED_LINE)
if serial:
serials.add(serial)
return frozenset(serials)
def _bundle_blocks(text: str) -> tuple[str, str, str]:
"""(agent certificate, agent key, root certificate) — positionally, because that is the format.
Two certificates and exactly one private key is the shape of a bundle. A bundle with the root
key in it is the failure this catches: an exporter needs to *verify* its clients, which is a
public-key operation, so a stolen agent install must never be a stolen CA.
"""
blocks = _PEM_BLOCK_RE.findall(text)
certs = [block for block in blocks if block.startswith(_CERT_HEADER)]
keys = [block for block in blocks if not block.startswith(_CERT_HEADER)]
if len(certs) != 2 or len(keys) != 1:
raise ValueError("expected the agent certificate, the agent key and the root certificate")
return certs[0], keys[0], certs[1]
def _file_stamp(path: str) -> tuple[int, int] | None:
"""(mtime_ns, size) for the denylist re-read, or None when the file cannot be stated.
Nanoseconds rather than the float `st_mtime`: two writes in the same millisecond are a real
redeploy, and a change this agent misses is a revoked certificate that keeps working.
"""
try:
info = os.stat(path)
except OSError:
return None
return (info.st_mtime_ns, info.st_size)
def _material_dir() -> Path:
"""The private directory to unpack into: the unit's RuntimeDirectory when usable, else tmp."""
if BUNDLE_RUNTIME_DIR.is_dir() and os.access(BUNDLE_RUNTIME_DIR, os.W_OK | os.X_OK):
return BUNDLE_RUNTIME_DIR
directory = Path(tempfile.mkdtemp(prefix="dashboards-agent-"))
os.chmod(directory, 0o700) # mkdtemp is 0700 already; the claim is this agent's, not the umask's
return directory
def _unlink(path: Path) -> None:
with contextlib.suppress(FileNotFoundError):
path.unlink()
def _write_private(path: Path, text: str) -> None:
"""Create `path` at 0600 in one `open()`, so a private key never exists at the umask default."""
_unlink(path) # O_EXCL would refuse a leftover from a killed run, and a symlink would be worse
handle = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(handle, "w", encoding="ascii") as stream:
stream.write(text)
def resolve_bundle(path: str) -> tuple[ssl.SSLContext | None, int, frozenset[str], str | None]:
"""(context, bundle_rev, revoked serials, error) for one installed bundle.
The `resolve_token` idiom: no globals, no server state, every failure an error string the
caller prints and exits on. A bundle this agent cannot load must stop the startup rather than
answer plain HTTP behind a token nobody configured — "degraded to no TLS" is the one answer
that is worse than a crash.
The three PEM blocks are grepped out by text and written into a private directory, because
`ssl` loads a chain from paths; they are unlinked as soon as the context has them, so the key
then lives in this process's memory and nowhere else (and there is no shutdown path that could
forget to remove it). The root CA goes in as the *verify* list, which is the whole of what an
exporter needs: it authenticates its clients, it is never authenticated by them.
"""
try:
text = Path(path).read_text(encoding="utf-8", errors="replace")
except OSError as exc:
return None, 0, frozenset(), f"cannot read --cert-bundle {path}: {exc}"
if not text.strip():
return None, 0, frozenset(), f"--cert-bundle {path} is empty"
revision = bundle_revision(text)
if revision is None:
return None, 0, frozenset(), (
f"--cert-bundle {path} has no readable `# BUNDLE-REV <n>` line — the revision is how "
"the dashboard tells this bundle from a stale one, so a bundle without it is not one "
"this app signed")
try:
server_cert, server_key, root_cert = _bundle_blocks(text)
except ValueError as exc:
return None, 0, frozenset(), f"--cert-bundle {path} is not a bundle: {exc}"
directory: Path | None = None
ephemeral = False
try:
directory = _material_dir()
ephemeral = directory != BUNDLE_RUNTIME_DIR
written = [directory / name for name in _BUNDLE_FILES]
for target, body in zip(written, (server_cert, server_key, root_cert), strict=True):
_write_private(target, body)
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.minimum_version = ssl.TLSVersion.TLSv1_2
context.load_cert_chain(str(written[0]), str(written[1]))
context.load_verify_locations(str(written[2]))
# Optional at the handshake, enforced by the handler: `CERT_REQUIRED` would make the
# installer's `--status` and systemd's probe fail, because neither holds a client
# certificate the installer was never given, and a service that cannot health-check is a
# dead install. See `Handler._certificate_allows` for who may read what.
context.verify_mode = ssl.CERT_OPTIONAL
except (OSError, ValueError, ssl.SSLError) as exc:
return None, 0, frozenset(), f"cannot load --cert-bundle {path}: {exc}"
finally:
if directory is not None:
for name in _BUNDLE_FILES:
_unlink(directory / name)
if ephemeral:
shutil.rmtree(directory, ignore_errors=True)
return context, revision, bundle_revoked(text), None
class TlsBundle:
"""One loaded bundle: the SSL context, the revision it carries, and the live denylist.
The context and the revision are fixed for the process's life — `ssl` cannot swap a chain
under a live socket, so a new certificate needs a restart (and says so on `/certs`). The
`# REVOKED` list is not fixed: re-reading one small text file when its stamp changes costs
nothing per poll and is the difference between "revoked" and "revoked on the next deploy",
which is the only reason the lines live in the bundle instead of in the dashboard alone.
"""
def __init__(self, path: str, context: ssl.SSLContext, bundle_rev: int,
revoked: frozenset[str]) -> None:
self.path = path
self.context = context
self.bundle_rev = bundle_rev
self._lock = threading.Lock() # ThreadingHTTPServer may re-read on several requests
self._revoked = revoked
self._stamp = _file_stamp(path)
@classmethod
def load(cls, path: str) -> tuple[TlsBundle | None, str | None]:
"""`resolve_bundle` plus the file stamp the denylist re-read compares against."""
context, revision, revoked, error = resolve_bundle(path)
if error is not None or context is None:
return None, error
return cls(path, context, revision, revoked), None
def revoked_serials(self) -> frozenset[str]:
"""The denylist as the file holds it right now, re-read only when the file changed."""
stamp = _file_stamp(self.path)
with self._lock:
if stamp is None or stamp == self._stamp:
return self._revoked
try:
text = Path(self.path).read_text(encoding="utf-8", errors="replace")
except OSError:
# Unreadable mid-redeploy: keep the list this agent already had. Forgetting a
# revocation because a file vanished for one second is the wrong direction.
return self._revoked
self._revoked = bundle_revoked(text)
self._stamp = stamp
return self._revoked
def _peer_certificate(connection: Any) -> dict[str, Any]:
"""The verified client certificate of this connection, or {} when there is none.
`getpeercert()` is only there on a TLS socket and only answers after a handshake, so a
plain-HTTP connection to a bundle-configured agent (a scanner, a port probe) reads as "no
certificate" and gets the same 403 as a certless TLS client rather than a traceback.
"""
getter = getattr(connection, "getpeercert", None)
if getter is None:
return {}
try:
peer = getter()
except (OSError, ValueError): # ValueError is `ssl` for "the handshake has not run"
return {}
return peer or {}
def _serial_forms(raw: str) -> frozenset[str]:
"""Every spelling one certificate serial can be written in.
`getpeercert()` renders a serial as **hexadecimal** text while `certs.py` writes the **decimal**
number into the bundle's `# REVOKED` line, and an agent that compared one spelling to the other
would accept a revoked client forever — a revocation that silently does nothing is worse than
no revocation list, because the operator believes it happened. Accepting both spellings on both
sides can only ever deny a certificate that is not revoked (two different serials whose hex and
decimal renderings coincide), never allow one that is.
"""
token = (raw or "").strip()
if not token:
return frozenset()
forms = {token}
for base in (10, 16):
try:
value = int(token, base)
except ValueError:
continue
forms.add(str(value))
forms.add(format(value, "X"))
return frozenset(forms)
def serial_is_revoked(cert: dict[str, Any], revoked: frozenset[str]) -> bool:
"""Is this peer certificate on the bundle's denylist?"""
peer = _serial_forms(str(cert.get("serialNumber") or ""))
if not peer:
return False
return any(_serial_forms(serial) & peer for serial in revoked)
# --------------------------------------------------------------------------- #
# HTTP server
# --------------------------------------------------------------------------- #
STATUS_PAGE = """<!doctype html>
<meta charset=utf-8><title>{hostname} — host agent</title>
<style>
body{{font:14px/1.5 ui-monospace,SFMono-Regular,Menlo,monospace;background:#0f1115;color:#e6e8ec;
margin:2rem;max-width:46rem}}
h1{{font-size:1.1rem;letter-spacing:.02em}} a{{color:#f7931e}}
table{{border-collapse:collapse}} td{{padding:.15rem .8rem .15rem 0}}
.k{{color:#8b93a7}} code{{background:#1a1e26;padding:.1rem .35rem;border-radius:4px}}
</style>
<h1>{hostname}</h1>
<table>
<tr><td class=k>agent</td><td>v{version}</td></tr>
<tr><td class=k>os</td><td>{os_pretty}</td></tr>
<tr><td class=k>kernel</td><td>{kernel}</td></tr>
<tr><td class=k>cpu</td><td>{cpu_model}</td></tr>
<tr><td class=k>uptime</td><td>{uptime}</td></tr>
</table>
<p><a href="/snapshot">/snapshot</a> · <a href="/info">/info</a> ·
<a href="/health">/health</a> · <a href="/schema">/schema</a></p>
"""
class Handler(BaseHTTPRequestHandler):
server_version = f"host_agent/{AGENT_VERSION}"
collector: Collector
token: str | None = None
bundle: TlsBundle | None = None
def do_GET(self) -> None: # the BaseHTTPRequestHandler API name — keep it
path = self.path.split("?", 1)[0].rstrip("/") or "/"
bundle = self.bundle
if bundle is not None:
# The handshake is the auth on this path, so the bearer check below never runs: a
# bundle *and* a token would let a stolen token read a host whose certificate the
# operator just revoked, which is the one thing the denylist exists to prevent.
if not self._certificate_allows(bundle, path):
return
elif self.token and path != "/health":
given = self.headers.get("Authorization", "")
if given != f"Bearer {self.token}":
self._send(401, {"ok": False, "detail": "missing or bad bearer token"})
return
if path == "/":
self._status_page()
elif path == "/health":
self._send(200, {"ok": True, "hostname": socket.gethostname(),
"version": AGENT_VERSION, "ts": time.time()})
elif path == "/info":
self._send(200, {"ok": True, "info": self.collector.host_info(),
"gpus": self.collector.gpus()})
elif path == "/schema":
self._send(200, {"ok": True, "schema": SCHEMA_VERSION, "doc": __doc__})
elif path == "/snapshot":
query = self.path.split("?", 1)[1] if "?" in self.path else ""
top = None
for part in query.split("&"):
if part.startswith("top="):
try:
top = max(0, int(part[4:]))
except ValueError:
top = None
body: dict[str, Any] = {"ok": True, **self.collector.snapshot(top=top)}
if bundle is not None:
# Additive, and only in TLS mode: the dashboard (task 03) reads the revision to
# tell a stale installed bundle from the one `/certs` just issued, and an absent
# field means "this agent predates certificates" rather than "revision 0".
body["tls"] = True
body["bundle_rev"] = bundle.bundle_rev
self._send(200, body)
else:
self._send(404, {"ok": False, "detail": f"no such path: {path}"})
def _certificate_allows(self, bundle: TlsBundle, path: str) -> bool:
"""Does this connection's client certificate authorise `path`? Sends the refusal if not.
Two answers, because they are two different operator actions: 403 "required" is a client
that never proved who it is (nothing installed, or the wrong CA), 403 "revoked" is a client
that proved it and was withdrawn.
The order is the design. A revoked identity is refused on **every** path, `/health` first:
the exemption is for the probe that holds the root CA and no client certificate at all, not
for a certificate the operator has just taken away. Then `/health` passes a certless probe,
because `install-agent.sh --status` and systemd's probe were never handed a client
certificate and a service that cannot health-check is a dead install.
"""
peer = _peer_certificate(self.connection)
present = bool(peer and peer.get("subject"))
if present and serial_is_revoked(peer, bundle.revoked_serials()):
self._send(403, {"ok": False, "detail": "client certificate revoked"})
return False
if path == "/health":
return True
if not present:
self._send(403, {"ok": False, "detail": "client certificate required"})
return False
return True
def _status_page(self) -> None:
info = self.collector.host_info()
uptime = info.get("uptime_seconds")
body = STATUS_PAGE.format(
hostname=info.get("hostname") or "unknown",
version=AGENT_VERSION,
os_pretty=info.get("os_pretty") or "unknown",
kernel=info.get("kernel") or "unknown",
cpu_model=info.get("cpu_model") or "unknown",
uptime=f"{uptime / 86400:.1f} days" if uptime else "unknown",
)
data = body.encode()
self.send_response(200)
self.send_header("content-type", "text/html; charset=utf-8")
self.send_header("content-length", str(len(data)))
self.send_header("cache-control", "no-store")
self.end_headers()
self.wfile.write(data)
def _send(self, status: int, body: dict[str, Any]) -> None:
data = json.dumps(body, default=str).encode()
self.send_response(status)
self.send_header("content-type", "application/json")
self.send_header("content-length", str(len(data)))
self.send_header("cache-control", "no-store")
self.end_headers()
self.wfile.write(data)
def log_message(self, fmt: str, *args: Any) -> None:
# One line per poll, journald-friendly, and never with query strings (a token
# pasted by mistake in a URL should not live in the journal).
if "%s" in fmt:
fmt = fmt % args # the log_message idiom from BaseHTTPRequestHandler
print(f"[agent] {self.address_string()} {fmt}", flush=True)
def serve(host: str, port: int, collector: Collector, token: str | None,
bundle: TlsBundle | None = None) -> None:
"""Serve forever on host:port — plain HTTP with an optional token, or HTTPS with a bundle.
All three class attributes are assigned on every call: `Handler` is process-wide state, so a
server started without a bundle must not inherit one from a server that had it (that would
answer a plain-HTTP socket with 403 on every path, or worse, accept a client certificate the
operator revoked).
"""
Handler.collector = collector
Handler.token = token
Handler.bundle = bundle
httpd = ThreadingHTTPServer((host, port), Handler)
if bundle is not None:
httpd.socket = bundle.context.wrap_socket(httpd.socket, server_side=True)
print(f"[agent] listening on https://{host}:{port} with tls "
f"(client cert required, rev {bundle.bundle_rev}); pid {os.getpid()}", flush=True)
else:
print(f"[agent] listening on http://{host}:{port} "
f"({'token required' if token else 'no token'}); pid {os.getpid()}", flush=True)
try:
httpd.serve_forever()
except KeyboardInterrupt:
print("[agent] shutting down", flush=True)
finally:
httpd.server_close()
def resolve_token(token: str, token_file: str) -> tuple[str | None, str | None]:
"""(token, error). A file wins over a flag because `ps` shows argv to every user on the
box, and this token guards hostname, mounts, GPU model and process names."""
if not token_file:
return (token or None), None
try:
value = Path(token_file).read_text(encoding="utf-8").strip()
except OSError as exc:
return None, f"cannot read --token-file {token_file}: {exc}"
if not value:
return None, f"--token-file {token_file} is empty"
return value, None
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0],
formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument("--host", default=os.environ.get("AGENT_HOST", "127.0.0.1"),
help="bind address; 0.0.0.0 exposes host metrics to the network")
parser.add_argument("--port", type=int, default=int(os.environ.get("AGENT_PORT", "9120")))
parser.add_argument("--token", default=os.environ.get("AGENT_TOKEN", ""),
help="bearer token required on every request except /health")
parser.add_argument("--token-file", default=os.environ.get("AGENT_TOKEN_FILE", ""),
help="read the bearer token from a file (keeps it out of ps); wins "
"over --token")
parser.add_argument("--cert-bundle", default=os.environ.get("AGENT_CERT_BUNDLE", ""),
help="one installed PEM from the dashboard: this agent's server "
"certificate and key, the dashboard's root CA, a `# BUNDLE-REV` "
"line and the `# REVOKED` denylist. Serves https and answers "
"every path but /health only to a client certificate signed by "
"that root and not on the list; on that path the token is not read")
parser.add_argument("--root", default="/", help="filesystem root to read (for tests)")
parser.add_argument("--top", type=int, default=5, help="processes per top-N list")
parser.add_argument("--net-all", action="store_true", help="include veth/bridge interfaces")
parser.add_argument("--smartctl", default="smartctl",
help="smartctl binary; missing is a documented degradation "
"(every disk's smart is null, one errors entry)")
parser.add_argument("--no-smart", action="store_true",
help="skip the smartctl pass entirely (disks report smart: null, "
"no error entries)")
parser.add_argument("--no-proxy", action="store_true",
help="skip the proxy add-on entirely (no `haproxy`, `nginx`, `sockets` "
"or `resolver` key, no error entries)")
parser.add_argument("--no-resolve", action="store_true",
help="skip the resolver pass (no `resolver` key, no error entries): the "
"two config sections and the socket table are still read, only the "
"name lookups are not")
parser.add_argument("--haproxy-config", default=HAPROXY_CONFIG_PATH,
help="haproxy config to read; a missing file is the documented "
"degradation (available=false, one error entry), not a failure")
parser.add_argument("--nginx-config", default=NGINX_CONFIG_PATH,
help=f"nginx main config to read, together with the files its own "
f"include lines name (one level, {NGINX_INCLUDE_MAX_FILES} files / "
f"{NGINX_INCLUDE_MAX_BYTES // 1000} KB); a missing file is the "
f"documented degradation (available=false, one error entry), "
f"not a failure")
parser.add_argument("--once", action="store_true", help="print one snapshot as JSON and exit")
args = parser.parse_args(argv)
collector = Collector(args.root, top_processes=args.top, net_all=args.net_all,
smartctl=args.smartctl, smart=not args.no_smart,
proxy=not args.no_proxy, resolve=not args.no_resolve,
haproxy_config=args.haproxy_config,
nginx_config=args.nginx_config)
if args.once:
print(json.dumps({"ok": True, **collector.snapshot()}, indent=2, default=str))
return 0
token, error = resolve_token(args.token, args.token_file)
if error:
print(f"[agent] {error}", flush=True)
return 2
bundle: TlsBundle | None = None
if args.cert_bundle:
bundle, bundle_error = TlsBundle.load(args.cert_bundle)
if bundle_error is not None:
print(f"[agent] {bundle_error}", flush=True)
return 2
if args.host not in ("127.0.0.1", "localhost", "::1") and not token and bundle is None:
print("[agent] refusing to bind off-loopback without --token or --cert-bundle (host "
"metrics are sensitive); pass --token, --token-file, --cert-bundle or set "
"AGENT_TOKEN / AGENT_CERT_BUNDLE", flush=True)
return 2
serve(args.host, args.port, collector, token, bundle)
return 0
if __name__ == "__main__":
raise SystemExit(main())
Reference in New Issue View Git Blame Copy Permalink