test(agent): controlled fixture KB + one-command fast loop for tool-calling iterations
The phase-72 iteration loop cleared the database, git-cloned the homelab repo, re-imported 38-51 documents and re-embedded per run — many minutes per iteration against a different KB every time (owner directive 2026-09-04: stop importing the homelab repo on every test run). Replace it with: - tests/fixtures/agent_kb/: 8 hand-written markdown docs (sources 'deployments'/'homelab') whose specifics (rack7, 10.77.42.0/24, VLAN 130, rbm-8842, 17 2 * * *, obsidian-bor:2026.7.14, 18765, 18443, ...) no model can guess; read targets carry non-topical filenames so their questions do not lexically seed them (the read must actually happen) - tests/fixtures/test_kb.dump.sql: data-only snapshot (TRUNCATE + INSERTs incl. embeddings, self-contained git_sources rows, static KB overview) — verified by round-trip checksum at build time - scripts/load_test_kb.py: one-off rebuild (real pipeline + embeddings, ~2s) that also prints the per-question retrieval report (all 10 battery questions must be grounded) - scripts/restore_test_kb.py: sub-second one-transaction restore (no git clone, no re-embedding) - scripts/agent_realmodel_check.py: the gate gains --restore / --mode fixture (curated 10-question battery with one unambiguously correct tool behavior per question) / --turns N (12s micro-loop) / --concurrency / per-turn + total wall timing, and a second accuracy metric (contract accuracy: well-formed calls targeting resolvable entities) alongside the phase-72 locked executed ratio — the re-read of a seeded doc is a copy-invariant model behavior (5 variants, 0/15 flipped) that the dedupe refusal counts as a failure - TOOL_CALLING_TESTING.md: the human-readable methodology (fast loop, design rules, metrics, copy levers + tried-and-reverted table, current standing, open design question) Measured: restore 0.03s; micro-loop ~12s; full loop ~43-55s; concurrency 2/3 gives no gain (endpoint serializes).
This commit is contained in:
@@ -0,0 +1,15 @@
|
||||
# Qwen 3.8 on llama.cpp (pbx-node3)
|
||||
|
||||
The GPU box (`pbx-node3`) serves Qwen 3.8 through llama.cpp.
|
||||
|
||||
Working launch line (verified 2026-08):
|
||||
|
||||
./llama-server -m qwen3.8-30b-a3b-instruct.Q8_0.gguf \
|
||||
-c 32768 -ngl 99 --mlock --threads 12 -sm layer \
|
||||
--cache-type-k q8_0 --cache-type-v q8_0 \
|
||||
--jinja --port 18180
|
||||
|
||||
- Context 32768, KV cache q8_0/q8_0, prompt cache persisted to
|
||||
`/srv/llamacpp/cache`.
|
||||
- First-token target is under 400 ms; sustained throughput about 28 tok/s.
|
||||
- Do not add `-ctk f16` — it doubled VRAM usage with no quality gain.
|
||||
@@ -0,0 +1,11 @@
|
||||
# Uptime Kuma (AIO)
|
||||
|
||||
Uptime Kuma runs on `rack7-pbx2` as the AIO image, listening on
|
||||
**18443** with TLS terminated by `caddy-rack7`.
|
||||
|
||||
- Health endpoint: `https://uptime.rack7.local/ping`
|
||||
- Every monitor posts to the ntfy topic `reese-uptime-7` on failure.
|
||||
- History is kept for 30 days; the database is backed up hourly to
|
||||
`/opt/kuma/backup/`.
|
||||
- Lab-iot monitors watch every VLAN 130 device; the shelf watchdog
|
||||
restarts the container after 3 missed checks.
|
||||
Reference in New Issue
Block a user