phase: 94_ls_tree_drilldown
All green. Verification complete. **Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)** - Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal - Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths - Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met - `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %) - `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings - E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched) - Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed - Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol) **Next pending phase:** `95_read_truncation_cap`
This commit is contained in:
+51
-11
@@ -15,23 +15,30 @@ guessable by any model. This script:
|
||||
``BOR_GIT_SOURCES`` env var),
|
||||
3. imports the fixture documents through the real pipeline
|
||||
(``import_sources`` — real chunking + real ``embed``-model
|
||||
embeddings; this is the ONLY step that burns model calls, and only
|
||||
at build time),
|
||||
embeddings; this is the only step that burns ``embed`` calls, and
|
||||
only at build time),
|
||||
4. stores the static KB overview + the sources-version row,
|
||||
5. prints a **retrieval report** for every fixture-battery question
|
||||
5. generates the sync-time **folder summaries** through the real
|
||||
generator (``app.rag.folder_summaries``) against the live ``lite``
|
||||
endpoint — the drill-down ``ls`` (phase 94) shows the stored rows,
|
||||
so the dump must carry them (the build always truncates first, so
|
||||
the sync paths' table-empty first-run trigger holds; a failed batch
|
||||
aborts the build — a summary-less dump would be a broken fixture),
|
||||
6. prints a **retrieval report** for every fixture-battery question
|
||||
(grounded or deflected, which documents would seed) — the battery
|
||||
must be all-grounded for the gate to exercise the tools,
|
||||
6. snapshots the resulting database state into
|
||||
7. snapshots the resulting database state into
|
||||
``tests/fixtures/test_kb.dump.sql`` — a data-only SQL script
|
||||
(TRUNCATE + one multi-row ``INSERT`` per app table, generated
|
||||
in-process — the same file runs in psql or psycopg, in one
|
||||
transaction) — and **verifies the snapshot by restoring it and
|
||||
comparing a per-table checksum**.
|
||||
|
||||
Re-run it only when the fixture documents, the chunker, or the
|
||||
embedding model change — everyday iterations restore the dump in
|
||||
sub-second time (``scripts/restore_test_kb`` / the gate's
|
||||
``--restore``), never re-embedding (see ``TOOL_CALLING_TESTING.md``).
|
||||
Re-run it only when the fixture documents, the chunker, the embedding
|
||||
model, or the folder-summary prompt (its ``lite`` output is baked into
|
||||
the dump) change — everyday iterations restore the dump in sub-second
|
||||
time (``scripts/restore_test_kb`` / the gate's ``--restore``), never
|
||||
re-embedding (see ``TOOL_CALLING_TESTING.md``).
|
||||
|
||||
Exit codes: **0** built + verified, **1** build/verification failure,
|
||||
**2** precondition failure.
|
||||
@@ -59,6 +66,7 @@ from app.models import (
|
||||
Chunk,
|
||||
DocDraft,
|
||||
Document,
|
||||
FolderSummary,
|
||||
GitSource,
|
||||
KbOverview,
|
||||
QueryLog,
|
||||
@@ -66,6 +74,7 @@ from app.models import (
|
||||
SourcesMeta,
|
||||
SteeringNote,
|
||||
)
|
||||
from app.rag.folder_summaries import generate_folder_summaries
|
||||
from app.rag.importer import import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from app.rag.retriever import retrieve
|
||||
@@ -111,6 +120,12 @@ _TABLES: tuple[tuple[str, type, tuple[str, ...]], ...] = (
|
||||
("doc_drafts", DocDraft, ("id", "token", "title", "path", "body",
|
||||
"status", "branch", "commit_sha", "created_at",
|
||||
"updated_at")),
|
||||
# Phase 94 (task 05): the sync-time folder summaries the drill-down
|
||||
# ``ls`` shows — generated against the live ``lite`` endpoint in
|
||||
# step 5 of :func:`_build` (the build truncates first, so the sync
|
||||
# paths' table-empty first-run trigger holds).
|
||||
("folder_summaries", FolderSummary, ("source", "folder_path",
|
||||
"summary", "updated_at")),
|
||||
)
|
||||
|
||||
|
||||
@@ -257,7 +272,31 @@ async def _build(kb_dir: Path, dump_path: Path) -> int:
|
||||
db.add(meta)
|
||||
db.commit()
|
||||
|
||||
# 4. Retrieval report (all battery questions must stay grounded).
|
||||
# 4. The sync-time folder summaries (phase 94, task 05): the
|
||||
# drill-down ``ls`` shows the stored rows, so the dump carries
|
||||
# them. Generated against the live ``lite`` endpoint through the
|
||||
# real sync-time generator — the build just truncated the table,
|
||||
# so the sync paths' table-empty first-run trigger holds (no
|
||||
# change-gate bookkeeping needed on a fresh build). The generator
|
||||
# only flushes; this build commits (the phase-53 convention the
|
||||
# sync paths follow). A failed batch aborts: the dump's folder
|
||||
# lines are part of the controlled fixture.
|
||||
with SessionLocal() as db:
|
||||
folder_stats = await generate_folder_summaries(db, llm)
|
||||
db.commit()
|
||||
logger.info(
|
||||
"load_test_kb: folder summaries generated=%d failed=%d pruned=%d",
|
||||
folder_stats["generated"], folder_stats["failed"], folder_stats["pruned"],
|
||||
)
|
||||
if folder_stats["failed"]:
|
||||
print(
|
||||
f"load_test_kb: {folder_stats['failed']} folder summary call(s) "
|
||||
"failed — the fixture dump needs the stored rows; check the LLM "
|
||||
"endpoint (BOR_LLM_SUMMARY_MODEL) and re-run"
|
||||
)
|
||||
return 1
|
||||
|
||||
# 5. Retrieval report (all battery questions must stay grounded).
|
||||
n_deflected = await _retrieval_report(llm, list(FIXTURE_BATTERY))
|
||||
if n_deflected:
|
||||
print(
|
||||
@@ -267,7 +306,7 @@ async def _build(kb_dir: Path, dump_path: Path) -> int:
|
||||
"question before running the gate."
|
||||
)
|
||||
|
||||
# 5. Snapshot (data-only) + round-trip verification.
|
||||
# 6. Snapshot (data-only) + round-trip verification.
|
||||
with SessionLocal() as db:
|
||||
before = {table: _table_checksum(db, table) for table, _m, _c in _TABLES}
|
||||
parts = [
|
||||
@@ -309,7 +348,8 @@ async def _build(kb_dir: Path, dump_path: Path) -> int:
|
||||
wall = time.monotonic() - started
|
||||
print(
|
||||
f"load_test_kb: ok — docs={summary.added} chunks={summary.chunks} "
|
||||
f"sources={len(FIXTURE_SOURCES)} dump={dump_path} "
|
||||
f"sources={len(FIXTURE_SOURCES)} folder_summaries={folder_stats['generated']} "
|
||||
f"dump={dump_path} "
|
||||
f"({dump_path.stat().st_size // 1024} KB, verified by round-trip) "
|
||||
f"in {wall:.1f}s"
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user