phase: 94_ls_tree_drilldown
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Build and Push Containers / build-and-push-db (push) Successful in 25s

All green. Verification complete.

**Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)**

- Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal
- Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths
- Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met
- `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched)
- Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed
- Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol)

**Next pending phase:** `95_read_truncation_cap`
This commit is contained in:
2026-09-11 00:59:35 -04:00
parent 9188be259b
commit d4943b4822
61 changed files with 6289 additions and 666 deletions
+51 -11
View File
@@ -15,23 +15,30 @@ guessable by any model. This script:
``BOR_GIT_SOURCES`` env var),
3. imports the fixture documents through the real pipeline
(``import_sources`` — real chunking + real ``embed``-model
embeddings; this is the ONLY step that burns model calls, and only
at build time),
embeddings; this is the only step that burns ``embed`` calls, and
only at build time),
4. stores the static KB overview + the sources-version row,
5. prints a **retrieval report** for every fixture-battery question
5. generates the sync-time **folder summaries** through the real
generator (``app.rag.folder_summaries``) against the live ``lite``
endpoint — the drill-down ``ls`` (phase 94) shows the stored rows,
so the dump must carry them (the build always truncates first, so
the sync paths' table-empty first-run trigger holds; a failed batch
aborts the build — a summary-less dump would be a broken fixture),
6. prints a **retrieval report** for every fixture-battery question
(grounded or deflected, which documents would seed) — the battery
must be all-grounded for the gate to exercise the tools,
6. snapshots the resulting database state into
7. snapshots the resulting database state into
``tests/fixtures/test_kb.dump.sql`` — a data-only SQL script
(TRUNCATE + one multi-row ``INSERT`` per app table, generated
in-process — the same file runs in psql or psycopg, in one
transaction) — and **verifies the snapshot by restoring it and
comparing a per-table checksum**.
Re-run it only when the fixture documents, the chunker, or the
embedding model change — everyday iterations restore the dump in
sub-second time (``scripts/restore_test_kb`` / the gate's
``--restore``), never re-embedding (see ``TOOL_CALLING_TESTING.md``).
Re-run it only when the fixture documents, the chunker, the embedding
model, or the folder-summary prompt (its ``lite`` output is baked into
the dump) change — everyday iterations restore the dump in sub-second
time (``scripts/restore_test_kb`` / the gate's ``--restore``), never
re-embedding (see ``TOOL_CALLING_TESTING.md``).
Exit codes: **0** built + verified, **1** build/verification failure,
**2** precondition failure.
@@ -59,6 +66,7 @@ from app.models import (
Chunk,
DocDraft,
Document,
FolderSummary,
GitSource,
KbOverview,
QueryLog,
@@ -66,6 +74,7 @@ from app.models import (
SourcesMeta,
SteeringNote,
)
from app.rag.folder_summaries import generate_folder_summaries
from app.rag.importer import import_sources
from app.rag.llm import LLMClient
from app.rag.retriever import retrieve
@@ -111,6 +120,12 @@ _TABLES: tuple[tuple[str, type, tuple[str, ...]], ...] = (
("doc_drafts", DocDraft, ("id", "token", "title", "path", "body",
"status", "branch", "commit_sha", "created_at",
"updated_at")),
# Phase 94 (task 05): the sync-time folder summaries the drill-down
# ``ls`` shows — generated against the live ``lite`` endpoint in
# step 5 of :func:`_build` (the build truncates first, so the sync
# paths' table-empty first-run trigger holds).
("folder_summaries", FolderSummary, ("source", "folder_path",
"summary", "updated_at")),
)
@@ -257,7 +272,31 @@ async def _build(kb_dir: Path, dump_path: Path) -> int:
db.add(meta)
db.commit()
# 4. Retrieval report (all battery questions must stay grounded).
# 4. The sync-time folder summaries (phase 94, task 05): the
# drill-down ``ls`` shows the stored rows, so the dump carries
# them. Generated against the live ``lite`` endpoint through the
# real sync-time generator — the build just truncated the table,
# so the sync paths' table-empty first-run trigger holds (no
# change-gate bookkeeping needed on a fresh build). The generator
# only flushes; this build commits (the phase-53 convention the
# sync paths follow). A failed batch aborts: the dump's folder
# lines are part of the controlled fixture.
with SessionLocal() as db:
folder_stats = await generate_folder_summaries(db, llm)
db.commit()
logger.info(
"load_test_kb: folder summaries generated=%d failed=%d pruned=%d",
folder_stats["generated"], folder_stats["failed"], folder_stats["pruned"],
)
if folder_stats["failed"]:
print(
f"load_test_kb: {folder_stats['failed']} folder summary call(s) "
"failed — the fixture dump needs the stored rows; check the LLM "
"endpoint (BOR_LLM_SUMMARY_MODEL) and re-run"
)
return 1
# 5. Retrieval report (all battery questions must stay grounded).
n_deflected = await _retrieval_report(llm, list(FIXTURE_BATTERY))
if n_deflected:
print(
@@ -267,7 +306,7 @@ async def _build(kb_dir: Path, dump_path: Path) -> int:
"question before running the gate."
)
# 5. Snapshot (data-only) + round-trip verification.
# 6. Snapshot (data-only) + round-trip verification.
with SessionLocal() as db:
before = {table: _table_checksum(db, table) for table, _m, _c in _TABLES}
parts = [
@@ -309,7 +348,8 @@ async def _build(kb_dir: Path, dump_path: Path) -> int:
wall = time.monotonic() - started
print(
f"load_test_kb: ok — docs={summary.added} chunks={summary.chunks} "
f"sources={len(FIXTURE_SOURCES)} dump={dump_path} "
f"sources={len(FIXTURE_SOURCES)} folder_summaries={folder_stats['generated']} "
f"dump={dump_path} "
f"({dump_path.stat().st_size // 1024} KB, verified by round-trip) "
f"in {wall:.1f}s"
)