test(agent): controlled fixture KB + one-command fast loop for tool-calling iterations

The phase-72 iteration loop cleared the database, git-cloned the homelab repo, re-imported 38-51 documents and re-embedded per run — many minutes per iteration against a different KB every time (owner directive 2026-09-04: stop importing the homelab repo on every test run). Replace it with:

- tests/fixtures/agent_kb/: 8 hand-written markdown docs (sources 'deployments'/'homelab') whose specifics (rack7, 10.77.42.0/24, VLAN 130, rbm-8842, 17 2 * * *, obsidian-bor:2026.7.14, 18765, 18443, ...) no model can guess; read targets carry non-topical filenames so their questions do not lexically seed them (the read must actually happen)
- tests/fixtures/test_kb.dump.sql: data-only snapshot (TRUNCATE + INSERTs incl. embeddings, self-contained git_sources rows, static KB overview) — verified by round-trip checksum at build time
- scripts/load_test_kb.py: one-off rebuild (real pipeline + embeddings, ~2s) that also prints the per-question retrieval report (all 10 battery questions must be grounded)
- scripts/restore_test_kb.py: sub-second one-transaction restore (no git clone, no re-embedding)
- scripts/agent_realmodel_check.py: the gate gains --restore / --mode fixture (curated 10-question battery with one unambiguously correct tool behavior per question) / --turns N (12s micro-loop) / --concurrency / per-turn + total wall timing, and a second accuracy metric (contract accuracy: well-formed calls targeting resolvable entities) alongside the phase-72 locked executed ratio — the re-read of a seeded doc is a copy-invariant model behavior (5 variants, 0/15 flipped) that the dedupe refusal counts as a failure
- TOOL_CALLING_TESTING.md: the human-readable methodology (fast loop, design rules, metrics, copy levers + tried-and-reverted table, current standing, open design question)

Measured: restore 0.03s; micro-loop ~12s; full loop ~43-55s; concurrency 2/3 gives no gain (endpoint serializes).

This commit is contained in:

ducoterra

2026-09-04 13:10:15 -04:00

parent 575d6c88d0

commit 7909bdb8da

13 changed files with 2090 additions and 0 deletions

									
										tests/fixtures/agent_kb/homelab/backups/restic-rack7.md
									
Vendored

		+11
		
												View File
												
				@@ -0,0 +1,11 @@

				# Restic Backups for Rack7

				Every config and document on the cluster is backed up nightly with

				restic into the minio repo `minio01:/backups/rack7`.

				- Machine ID: **rbm-8842**

				- Password: `op vault view rack7/restic`

				- Schedule: `17 2 * * *` (02:17 nightly)

				- Retention: `--keep-daily 7 --keep-weekly 4 --keep-monthly 12`

				- Prune runs only when the repo grows more than 5 GiB.

				- Restores are tested quarterly; the last dry-run passed on 2026-07-01.