test(agent): controlled fixture KB + one-command fast loop for tool-calling iterations

The phase-72 iteration loop cleared the database, git-cloned the homelab repo, re-imported 38-51 documents and re-embedded per run — many minutes per iteration against a different KB every time (owner directive 2026-09-04: stop importing the homelab repo on every test run). Replace it with: - tests/fixtures/agent_kb/: 8 hand-written markdown docs (sources 'deployments'/'homelab') whose specifics (rack7, 10.77.42.0/24, VLAN 130, rbm-8842, 17 2 * * *, obsidian-bor:2026.7.14, 18765, 18443, ...) no model can guess; read targets carry non-topical filenames so their questions do not lexically seed them (the read must actually happen) - tests/fixtures/test_kb.dump.sql: data-only snapshot (TRUNCATE + INSERTs incl. embeddings, self-contained git_sources rows, static KB overview) — verified by round-trip checksum at build time - scripts/load_test_kb.py: one-off rebuild (real pipeline + embeddings, ~2s) that also prints the per-question retrieval report (all 10 battery questions must be grounded) - scripts/restore_test_kb.py: sub-second one-transaction restore (no git clone, no re-embedding) - scripts/agent_realmodel_check.py: the gate gains --restore / --mode fixture (curated 10-question battery with one unambiguously correct tool behavior per question) / --turns N (12s micro-loop) / --concurrency / per-turn + total wall timing, and a second accuracy metric (contract accuracy: well-formed calls targeting resolvable entities) alongside the phase-72 locked executed ratio — the re-read of a seeded doc is a copy-invariant model behavior (5 variants, 0/15 flipped) that the dedupe refusal counts as a failure - TOOL_CALLING_TESTING.md: the human-readable methodology (fast loop, design rules, metrics, copy levers + tried-and-reverted table, current standing, open design question) Measured: restore 0.03s; micro-loop ~12s; full loop ~43-55s; concurrency 2/3 gives no gain (endpoint serializes).
2026-09-04 13:10:15 -04:00
parent 575d6c88d0
commit 7909bdb8da
13 changed files with 2090 additions and 0 deletions
@@ -0,0 +1,17 @@
+# Lab Ansible Inventory
+
+The lab deployment inventory lives in `op-build:~/lab/inventory/hosts.yml`
+and is managed with ansible-core 2.19.4.
+
+    lab:
+      hosts:
+        rack7-pbx1: { ip: 10.77.42.11, role: hypervisor }
+        rack7-pbx2: { ip: 10.77.42.12, role: aio }
+        minio01:    { ip: 10.77.42.31, role: storage }
+      vars:
+        ansible_user: reese
+        ansible_become: true
+
+- Playbooks run from `op-build` only; the shelf has no internet route.
+- The `lab-upgrade` play requires the `pve-upgrade` tag on the target.
+- These files are the single source of truth for shelf addresses.
@@ -0,0 +1,12 @@
+# GitLab Runner (lab-ci)
+
+CI for the shelf runs on a single GitLab Runner registered against
+`git.reeseapps.com`.
+
+- Executor: docker; the runner daemon lives on `rack7-pbx2`.
+- Registration token format: `glrt-` plus 20 hex chars (rotated 2026-06).
+- Tags: `lab-ci` (default) and `pve-upgrade` (the gated upgrade job).
+- Job images: `ghcr.io/reese/lab-ci:2026.7` for ansible plays,
+  `debian:13-slim` for lint.
+- Concurrency is capped at 2; the `pve-upgrade` job never runs in
+  parallel with itself.
@@ -0,0 +1,15 @@
+# BOR Vault Sync Unit
+
+The Brain-of-Reese vault syncs from `obsidian.container` on `rack7-pbx2`.
+
+    [Container]
+    Image=ghcr.io/reese/obsidian-bor:2026.7.14
+    PublishPort=127.0.0.1:18765:8080
+    Environment=VAULT_DIR=/vault/rack7
+    Restart=always
+    MemoryMax=2g
+
+- The vault is a bind mount from `/data/vaults/rack7` (Btrfs subvolume).
+- Only `rack7-pbx2` may hold the vault — do not clone it to another node.
+- Image tags are date-stamped; bump by re-pushing
+  `ghcr.io/reese/obsidian-bor` and restarting the unit.
@@ -0,0 +1,11 @@
+# Restic Backups for Rack7
+
+Every config and document on the cluster is backed up nightly with
+restic into the minio repo `minio01:/backups/rack7`.
+
+- Machine ID: **rbm-8842**
+- Password: `op vault view rack7/restic`
+- Schedule: `17 2 * * *` (02:17 nightly)
+- Retention: `--keep-daily 7 --keep-weekly 4 --keep-monthly 12`
+- Prune runs only when the repo grows more than 5 GiB.
+- Restores are tested quarterly; the last dry-run passed on 2026-07-01.
@@ -0,0 +1,15 @@
+# Qwen 3.8 on llama.cpp (pbx-node3)
+
+The GPU box (`pbx-node3`) serves Qwen 3.8 through llama.cpp.
+
+Working launch line (verified 2026-08):
+
+    ./llama-server -m qwen3.8-30b-a3b-instruct.Q8_0.gguf \
+      -c 32768 -ngl 99 --mlock --threads 12 -sm layer \
+      --cache-type-k q8_0 --cache-type-v q8_0 \
+      --jinja --port 18180
+
+- Context 32768, KV cache q8_0/q8_0, prompt cache persisted to
+  `/srv/llamacpp/cache`.
+- First-token target is under 400 ms; sustained throughput about 28 tok/s.
+- Do not add `-ctk f16` — it doubled VRAM usage with no quality gain.
@@ -0,0 +1,11 @@
+# Uptime Kuma (AIO)
+
+Uptime Kuma runs on `rack7-pbx2` as the AIO image, listening on
+**18443** with TLS terminated by `caddy-rack7`.
+
+- Health endpoint: `https://uptime.rack7.local/ping`
+- Every monitor posts to the ntfy topic `reese-uptime-7` on failure.
+- History is kept for 30 days; the database is backed up hourly to
+  `/opt/kuma/backup/`.
+- Lab-iot monitors watch every VLAN 130 device; the shelf watchdog
+  restarts the container after 3 missed checks.
@@ -0,0 +1,10 @@
+# Rack7 DNS and DHCP
+
+`minio01` (10.77.42.31) serves DNS and DHCP for the whole lab shelf.
+
+- dnsmasq answers port 53 with upstream `1.1.1.1` and `9.9.9.9`.
+- VLAN 130 (lab-iot) pool: 10.77.130.50–10.77.130.200, 6-hour leases,
+  netmask 255.255.255.0.
+- Pi-hole runs on 10.77.42.53; the admin UI is on port 18553 behind
+  caddy-rack7 and uses the blocklist `reese-abuse-v3`.
+- VLAN 42 has no DHCP at all — every device there gets a static address.
@@ -0,0 +1,18 @@
+# Rack7 Proxmox Cluster
+
+Three-node Proxmox VE cluster on the rack7 shelf: `pbx-node1`, `pbx-node2`
+and `pbx-node3`, all running PVE 8.3.4 (build `8.3.4-1-lab1`).
+
+## Virtual interfaces
+
+| vmbr  | purpose             | config                                             |
+|-------|---------------------|----------------------------------------------------|
+| vmbr0 | management          | 10.77.42.0/24, gateway 10.77.42.1 on pbx-node1     |
+| vmbr1 | lab (VLAN 42)       | tag 42, no DHCP, NAT-only to the uplink            |
+| vmbr2 | lab-iot (VLAN 130)  | tag 130, DHCP served by minio01                    |
+
+- Cluster firewall is enabled on every node; VLAN 42 traffic is NAT-only.
+- Corosync heartbeat rides vmbr0; keep the management iface out of the
+  lab-iot VLAN.
+- Node roles: pbx-node1 = hypervisor + gateway, pbx-node2 = AIO
+  containers, pbx-node3 = GPU box.