phase: 10_dev_direct_dial
**Phase 10 final pass — repo green; live criteria blocked off-repo.**
- Verified: `gateway` runs `dashboards-agent` (active/enabled, `10.1.1.27:9120` → `/health` 200, `/snapshot` 401 no-token / 200 bearer on-host, token matches `/etc/dashboards-agent.token`); `driveripper`/`deskwork` inactive, `sudo -n` → "a password is required", `:9120` empty.
- New measurement (transient runtime rule on gateway `firewall-cmd --add-port=9120/tcp`, **reverted** — runtime+permanent port lists verified back to the original set, no rich rules): laptop still gets no SYN through, while `:22`/`:80` on the same address answer in ms ⇒ the block is the lab gateway `10.2.0.1` port filter, not this app/host firewall/agent. No other live change made (no sudoers, no upsd, no UPS command).
- Tests/lint (exact): `bash .agents/validate.sh` → **EXIT 0** — `uv run pytest --cov --cov-fail-under=94` **793 passed, 11 skipped, coverage 98.33 %**; `ruff check src tests agent` clean; `pyright` 0 errors; `node --check` clean; hygiene clean; `uv pip list --outdated` empty (bumped fastapi 0.143.3→0.143.5, `uv.lock` only).
- `uv run pytest -m live --live` (`.env.dev` sourced): **4 passed, 1 skipped, 6 failed** — all six are `:9120`/`:3051` connect timeouts; zero parse/contract failures.
- Dev server restarted on current code: engines `ok`; 3×`host` + 3×`disks` + `nut` `DOWN` with real reasons (designed degradation, stub rows intact, no credential leakage).
- Completion criteria: screenshots ✓ (`data/logs/phase10-{index,system,disks,llm,ups}-live.png`); tunnel-vocabulary grep ✓ outside `.agents/` history (PLAN.md locked line, phase overviews, past reports; fixtures/`uv.lock` hits are digit substrings — `tests/test_deploy_scripts.py` guards operational files); no page-payload change ✓ (contract tests). **Not met, operator-side:** agents on all three hosts, `LIST UPS` from the laptop, `dev-server.sh status` all-ok, live suite green.
- Fixed in this pass: `sources/nut.py` unreachable-upsd message no longer prints a bare `(TimeoutError: )` (now `(timed out after 4s)` / `(ConnectionRefusedError: …)`), pinned by a new test + extended refused-port test.
- Docs: README verified consistent (every env var in `config.py` documented, routes/flags/`--with-llm`/pager sizes checked) — no changes needed. AGENTS.md fixed 4 inconsistencies: per-host sudo reality (gateway grants `NOPASSWD ALL`, which is why it installed without `agent-sudoers`; driveripper scoped; deskwork unrelated), the now-proven firewall finding, `/system` table typo ("agent per host"), gap-4 DB/WAL sizes + test count. `.env.example` `upsd.users` sample now one `instcmds` per line (it contradicted README and the in-app copy). `.env.dev` (gitignored) comment corrected to match the evidence.
- Flagged, untouched (locked): `.agents/PLAN.md` L92–95 still says upsd is "forwarded to `127.0.0.1:3494` in dev" — false after this phase; evidence + exact operator commands appended to `data/logs/NOTES-verify.md` §11.
- Next pending phase: `11_admin_login`.
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
#!/usr/bin/env bash
|
||||
# Re-record ground-truth captures into tests/fixtures/.
|
||||
#
|
||||
# ./scripts/capture-fixtures.sh # needs the tunnels up: scripts/dev-tunnels.sh start
|
||||
# ./scripts/capture-fixtures.sh # dials HOST_TARGETS from .env.dev, directly
|
||||
# ./scripts/capture-fixtures.sh --with-llm # also scrape the live engines' /metrics
|
||||
# # (gufo /slots is captured without the flag)
|
||||
# SMART_HOSTS="deskwork-ops driveripper-ops" ./scripts/capture-fixtures.sh
|
||||
@@ -19,25 +19,37 @@ ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
FIX="$ROOT/tests/fixtures"
|
||||
mkdir -p "$FIX"
|
||||
|
||||
HOSTS=(driveripper:9121 deskwork:9122 gateway:9123)
|
||||
# Engine URLs come from the same env the app uses, so this script carries no deployment
|
||||
# detail of its own — and never records a tokenised URL into a fixture by accident.
|
||||
# Agents come from the same env the app reads: HOST_TARGETS in .env.dev, `name=url` pairs
|
||||
# (space or comma separated — the shape config.parse_targets accepts), the bearer token in
|
||||
# the URL userinfo used as-is. Nothing is hard-coded here, so a capture can never come from
|
||||
# a different topology than the dashboard polls.
|
||||
if [ -f "$ROOT/.env.dev" ]; then
|
||||
set -a; . "$ROOT/.env.dev"; set +a
|
||||
fi
|
||||
HOSTS=()
|
||||
targets="${HOST_TARGETS:-}"
|
||||
for entry in ${targets//,/ }; do # space or comma separated, like config.parse_targets
|
||||
HOSTS+=("$entry")
|
||||
done
|
||||
# Engine URLs come from the same env too, so this script carries no deployment
|
||||
# detail of its own — and never records a tokenised URL into a fixture by accident.
|
||||
ENGINES=()
|
||||
[ -n "${GUFO_BASE_URL:-}" ] && ENGINES+=("gufo:${GUFO_BASE_URL%/}/metrics")
|
||||
[ -n "${VLLM_BASE_URL:-}" ] && ENGINES+=("vllm:${VLLM_BASE_URL%/}/metrics")
|
||||
|
||||
failed=0
|
||||
|
||||
if [ ${#HOSTS[@]} -eq 0 ]; then
|
||||
echo " (skipped agent captures — HOST_TARGETS unset; see .env.dev.example)"
|
||||
fi
|
||||
for entry in "${HOSTS[@]}"; do
|
||||
name=${entry%%:*}; port=${entry##*:}
|
||||
url="http://127.0.0.1:${port}/snapshot"
|
||||
if curl -fsS --max-time 8 "$url" -o "$FIX/agent_${name}.json"; then
|
||||
name=${entry%%=*}; url=${entry#*=}
|
||||
# The failure line prints the address without its userinfo: a token belongs in the
|
||||
# dial, not in the terminal scrollback or whatever shells this output into.
|
||||
if curl -fsS --max-time 8 "${url%/}/snapshot" -o "$FIX/agent_${name}.json"; then
|
||||
echo " ✓ agent_${name}.json ($(wc -c <"$FIX/agent_${name}.json") bytes)"
|
||||
else
|
||||
echo " ✗ agent_${name}.json — no agent on $url (scripts/dev-tunnels.sh start?)"
|
||||
echo " ✗ agent_${name}.json — no agent answered at ${url#*@} (deploy/install-agent.sh? firewall?)"
|
||||
failed=1
|
||||
fi
|
||||
done
|
||||
@@ -105,7 +117,7 @@ PY
|
||||
then
|
||||
: # per-UPS lines above
|
||||
else
|
||||
echo " ✗ nut_*.vars — upsd at $NUT_HOST:${NUT_PORT:-3493} unreachable (scripts/dev-tunnels.sh start?)"
|
||||
echo " ✗ nut_*.vars — upsd at $NUT_HOST:${NUT_PORT:-3493} unreachable (down? firewall? NUT_PORT is the port upsd really listens on)"
|
||||
failed=1
|
||||
fi
|
||||
else
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# Local development server: start|stop|status|restart|logs|tunnel-check
|
||||
# Local development server: start|stop|status|restart|logs
|
||||
#
|
||||
# ./scripts/dev-server.sh restart # reads .env.dev if present, runs on DASH_PORT
|
||||
#
|
||||
|
||||
@@ -1,174 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Dev-only: reach live services on each host from the laptop without opening a single port.
|
||||
#
|
||||
# ./scripts/dev-tunnels.sh start|stop|status|restart
|
||||
#
|
||||
# DEV_TUNNELS starts agent/host_agent.py remotely and forwards it:
|
||||
# ssh -L <local>:127.0.0.1:<rport> <host> 'python3 host_agent.py --port <rport>'
|
||||
# DEV_FORWARDS forwards a service that is already listening remotely (e.g. upsd on 3493):
|
||||
# nothing is started over there, the port just appears on localhost.
|
||||
#
|
||||
# Both supervise themselves: a dropped link retries every 5s. Production is the opposite —
|
||||
# deploy/install-agent.sh installs a systemd unit that binds the LAN address with a bearer
|
||||
# token, and the dashboard dials it directly.
|
||||
set -uo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
RUN="$ROOT/data/tunnels"
|
||||
LOGS="$ROOT/data/logs"
|
||||
# Where the agent lives on each host. `~` is written into the generated script literally so
|
||||
# the REMOTE shell expands it (a local $HOME would be the wrong path over there).
|
||||
AGENT_REMOTE_DIR="${AGENT_REMOTE_DIR:-~/lab-agent}"
|
||||
AGENT_COMMAND="${AGENT_COMMAND:-python3 host_agent.py}"
|
||||
|
||||
# Targets come from .env.dev (see .env.dev.example) — never hardcoded here, so this file
|
||||
# stays portable and holds no hostnames of ours.
|
||||
[ -f "$ROOT/.env.dev" ] && . "$ROOT/.env.dev"
|
||||
DEV_TUNNELS="${DEV_TUNNELS:-}"
|
||||
DEV_FORWARDS="${DEV_FORWARDS:-}"
|
||||
if [ -z "$DEV_TUNNELS" ] && [ -z "$DEV_FORWARDS" ]; then
|
||||
echo "no DEV_TUNNELS or DEV_FORWARDS set — copy .env.dev.example to .env.dev first"; exit 2
|
||||
fi
|
||||
TUNNELS=()
|
||||
for entry in $DEV_TUNNELS; do
|
||||
# 9121:driveripper-ops:9120 -> "9121 driveripper-ops 9120"
|
||||
TUNNELS+=("$(echo "$entry" | tr ':' ' ')")
|
||||
done
|
||||
FORWARDS=()
|
||||
for entry in $DEV_FORWARDS; do
|
||||
FORWARDS+=("$(echo "$entry" | tr ':' ' ')")
|
||||
done
|
||||
|
||||
mkdir -p "$RUN" "$LOGS"
|
||||
|
||||
start() {
|
||||
for spec in "${TUNNELS[@]}"; do
|
||||
read -r local host remote <<<"$(echo "$spec" | tr -s ' ')"
|
||||
# pidfiles are named by host AND port: one host can carry an agent tunnel plus an upsd
|
||||
# forward. The older host-only name is cleared so a restart does not double-bind.
|
||||
pidfile="$RUN/$host-$local.pid"
|
||||
if [ -f "$RUN/$host.pid" ] && kill -0 "$(cat "$RUN/$host.pid")" 2>/dev/null; then
|
||||
pkill -P "$(cat "$RUN/$host.pid")" 2>/dev/null
|
||||
kill "$(cat "$RUN/$host.pid")" 2>/dev/null
|
||||
fi
|
||||
rm -f "$RUN/$host.pid"
|
||||
if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile")" 2>/dev/null; then
|
||||
echo " $host already tunnelling on :$local (pid $(cat "$pidfile"))"
|
||||
continue
|
||||
fi
|
||||
# A previous agent survives the ssh session that started it, and the next agent then
|
||||
# dies on bind() with "address already in use" — which looks like a broken agent rather
|
||||
# than a leaked process. Clear it in a SEPARATE ssh session: putting pkill in the same
|
||||
# command line as the agent makes pkill match that very command line and shoot itself.
|
||||
cat > "$RUN/$host.sh" <<EOF
|
||||
#!/usr/bin/env bash
|
||||
# generated by scripts/dev-tunnels.sh — do not edit, it is rewritten on every start
|
||||
while true; do
|
||||
ssh -n -o BatchMode=yes "$host" \
|
||||
"pkill -f '[h]ost_agent.py --host 127.0.0.1 --port $remote'" >/dev/null 2>&1
|
||||
sleep 1
|
||||
ssh -nT -o ServerAliveInterval=30 -o ServerAliveCountMax=4 -o ExitOnForwardFailure=yes \
|
||||
-L $local:127.0.0.1:$remote "$host" \
|
||||
"cd $AGENT_REMOTE_DIR && exec $AGENT_COMMAND --host 127.0.0.1 --port $remote"
|
||||
echo "[tunnel] $host link dropped at \$(date -Is) (exit \$?); retrying in 5s"
|
||||
sleep 5
|
||||
done
|
||||
EOF
|
||||
chmod +x "$RUN/$host.sh"
|
||||
nohup "$RUN/$host.sh" >>"$LOGS/tunnel-$host.log" 2>&1 &
|
||||
echo $! >"$pidfile"
|
||||
echo " $host -> http://127.0.0.1:$local (pid $!)"
|
||||
done
|
||||
|
||||
# Plain forwards: `-N` because there is no remote command to run, and nothing to clean up
|
||||
# over there when the link goes.
|
||||
for spec in "${FORWARDS[@]}"; do
|
||||
read -r local host remote <<<"$(echo "$spec" | tr -s ' ')"
|
||||
pidfile="$RUN/$host-$local.pid"
|
||||
if [ -f "$pidfile" ] && kill -0 "$(cat "$pidfile")" 2>/dev/null; then
|
||||
echo " $host:$remote already forwarded on :$local (pid $(cat "$pidfile"))"
|
||||
continue
|
||||
fi
|
||||
cat > "$RUN/$host-$local.sh" <<EOF
|
||||
#!/usr/bin/env bash
|
||||
# generated by scripts/dev-tunnels.sh — do not edit, it is rewritten on every start
|
||||
while true; do
|
||||
ssh -n -N -o ServerAliveInterval=30 -o ServerAliveCountMax=4 -o ExitOnForwardFailure=yes \
|
||||
-L $local:127.0.0.1:$remote "$host"
|
||||
echo "[forward] $host:$remote dropped at \$(date -Is) (exit \$?); retrying in 5s"
|
||||
sleep 5
|
||||
done
|
||||
EOF
|
||||
chmod +x "$RUN/$host-$local.sh"
|
||||
nohup "$RUN/$host-$local.sh" >>"$LOGS/tunnel-$host-$local.log" 2>&1 &
|
||||
echo $! >"$pidfile"
|
||||
echo " $host:$remote -> 127.0.0.1:$local (pid $!)"
|
||||
done
|
||||
}
|
||||
|
||||
stop() {
|
||||
for spec in "${TUNNELS[@]}"; do
|
||||
read -r local host remote <<<"$(echo "$spec" | tr -s ' ')"
|
||||
pidfile="$RUN/$host-$local.pid"
|
||||
if [ -f "$RUN/$host.pid" ]; then
|
||||
pkill -P "$(cat "$RUN/$host.pid")" 2>/dev/null
|
||||
kill "$(cat "$RUN/$host.pid")" 2>/dev/null
|
||||
rm -f "$RUN/$host.pid"
|
||||
fi
|
||||
[ -f "$pidfile" ] || continue
|
||||
pid=$(cat "$pidfile")
|
||||
pkill -P "$pid" 2>/dev/null
|
||||
kill "$pid" 2>/dev/null
|
||||
# -N would mean "no remote command", so the forward is kept by a plain ssh running the
|
||||
# agent; kill any leftover forward for this host by pattern, not just by parent.
|
||||
pkill -f "ssh -nT.*-L $local:127.0.0.1:$remote $host" 2>/dev/null
|
||||
# The remote agent outlives its ssh session, and the next one then dies on bind() with
|
||||
# "address already in use" — which looks like a broken agent, not a leaked process.
|
||||
timeout 10 ssh -n "$host" "pkill -f '[h]ost_agent.py --host 127.0.0.1 --port $remote'" \
|
||||
>/dev/null 2>&1
|
||||
rm -f "$pidfile"
|
||||
echo " stopped $host"
|
||||
done
|
||||
|
||||
for spec in "${FORWARDS[@]}"; do
|
||||
read -r local host remote <<<"$(echo "$spec" | tr -s ' ')"
|
||||
pidfile="$RUN/$host-$local.pid"
|
||||
[ -f "$pidfile" ] || continue
|
||||
pkill -P "$(cat "$pidfile")" 2>/dev/null
|
||||
kill "$(cat "$pidfile")" 2>/dev/null
|
||||
pkill -f "ssh -n -N.*-L $local:127.0.0.1:$remote $host" 2>/dev/null
|
||||
rm -f "$pidfile" "$RUN/$host-$local.sh"
|
||||
echo " stopped forward $host:$remote (:$local)"
|
||||
done
|
||||
}
|
||||
|
||||
status() {
|
||||
for spec in "${TUNNELS[@]}"; do
|
||||
read -r local host remote <<<"$(echo "$spec" | tr -s ' ')"
|
||||
body=$(curl -s --max-time 4 "http://127.0.0.1:$local/health" || true)
|
||||
if [ -n "$body" ]; then
|
||||
echo " OK $host http://127.0.0.1:$local $body"
|
||||
else
|
||||
echo " DOWN $host http://127.0.0.1:$local"
|
||||
fi
|
||||
done
|
||||
|
||||
for spec in "${FORWARDS[@]}"; do
|
||||
read -r local host remote <<<"$(echo "$spec" | tr -s ' ')"
|
||||
# no HTTP behind these (upsd speaks its own line protocol), so the check is a bare TCP
|
||||
# handshake — enough to tell "the tunnel is up" from "nothing listens over there".
|
||||
if timeout 3 bash -c "exec 3<>/dev/tcp/127.0.0.1/$local" 2>/dev/null; then
|
||||
echo " OK $host:$remote 127.0.0.1:$local"
|
||||
else
|
||||
echo " DOWN $host:$remote 127.0.0.1:$local"
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
case "${1:-start}" in
|
||||
start) start ;;
|
||||
stop) stop ;;
|
||||
restart) stop; sleep 1; start ;;
|
||||
status) status ;;
|
||||
*) echo "usage: $0 start|stop|status|restart"; exit 2 ;;
|
||||
esac
|
||||
Reference in New Issue
Block a user