brain-of-reese/.agents/phases/complete/94_ls_tree_drilldown/00_phase.md

Phase 94 — ls becomes a drill-down tree with sync-time folder summaries

Source: TODO.md L4 — "Brain of reese can end up indexing thousands of files. When the llm calls list it can end up filling the context entirely with just the document lists. I think list should work like a filesystem tree where it shows only the current "folder" (or at the top level, the list of synced projects) and then allow the LLM to drill down into the tree structure as it needs. We should add a level of "smart" to this by summarizing what each folder contains (at sync time) so when the LLM calls list it gets a summary of what's in each directory. Run tests against this new architecture and ensure accuracy and performance aren't too badly impacted." Story: n/a (owner TODO item — agent tooling on 37_agent_document_tools / 45_agent_unlimited_tools / 68_search_tool / 70_harness_aligned_tools). Context: Today ls (the harness-aligned surface, phase 70 — app/rag/agent.py AGENT_TOOLS) lists every indexed document as source: X | path: Y | title: Z lines (or one source's documents when path is a source name) — with a 1,340-document KB that is the entire context in one call. read/grep take the combined source/path identity; grep is the whole-KB locator. The one-shot lite machinery exists: document summaries (app/rag/summarizer.py, SUMMARY_MODE) and the KB overview (app/rag/overview.py, KB_OVERVIEW_MODE) — both change-gated at import, best-effort/fail-soft, with a marker the deterministic E2E mock (tests/e2e/mock_llm.py) keys on. Sync paths: scripts/import_docs.py (_run(), the overview hook) and POST /api/sync (app/api/sync.py, step 5). The controlled accuracy/performance methodology is TOOL_CALLING_TESTING.md + uv run python -m scripts.agent_realmodel_check --restore --mode fixture (the 8-document, 2-source, folder-structured fixture KB in tests/fixtures/test_kb.dump.sql: deployments/{ansible,ci,quadlet} + homelab/{backups,containers,networking}).

Objective

Rebuild ls as a filesystem-style tree the LLM drills through: top level lists the synced projects (sources) with a per-source summary; ls of a source lists its folders (each with a sync-time summary) + top-level files; ls of a folder lists its subfolders + files. Folder summaries are generated at sync time by the lite model (change-gated, fail-soft). The controlled tool-calling battery then proves accuracy and performance are not badly impacted.

Dependencies

• 93_theme_semantic_completion (todo) — queue order only; no code dependency (different subsystems).

Owner-permitted decision recorded here (PLAN.md is being redone by the owner)

• Tool-surface revision (owner permission 2026-09-10, TODO.md L4): the ls RESULT format and path semantics change (tree drill-down); the ls NAME and the read/grep contract (combined source/path) are untouched. This extends the phase-70 harness-aligned surface per the owner's explicit request; the prompt teaching (TOOLS_SECTION) is rewritten to match in the same change.

Design (shared by all tasks — the executor reads this, not the chat)

Tree shape (task 03)

Document paths are already tree-like (deployments/ansible/lab-inventory.md); a folder is a path prefix. Three ls levels, one argument:

Call	Result
ls() (no path)	every registered source (registry order, list_source_names): name — N documents + an indented summary line when one is stored (folders with no docs still list as 0 documents)
ls(path=<source>)	the source's root folder: each subfolder <folder>/ — N documents[: summary] (count = documents with path == folder or path startswith folder + "/"), then the root's own file lines `source: X
ls(path=<source>/<folder>[/…])	that folder's subfolders (same shape, paths relative to the source) + the folder's own file lines

• File-line cap: a folder's own files list at most 50 lines (path order — catalog order), then one deterministic note: …and {N−50} more documents in this folder — use grep (pattern) to find a specific one. (folds the old whole-KB flood into one bounded level; grep stays the locator — no new tool).
• Summaries: shown for the SOURCE (top level) and for each SUBFOLDER, when stored; absent → the count line only (no placeholder).
• Refusals (adapted from the phase-70/72 teaching lines, same style — one line, argument echoed, counts in nothing, consumes a round): unknown first segment → the existing no-source refusal; a source/… argument whose folder matches no indexed prefix → a NOT-A-FOLDER line teaching the drill-down (echo the argument, list the parent folder's subfolders so the model can self-correct in the next round). ls of a registered source with no documents keeps the 0 documents: behavior.
• holder.tool_calls increments on every successful listing (top/root/folder); refusals count in nothing (unchanged house rule).

Folder summaries (tasks 01, 02)

• Storage: new table folder_summaries — PK (source, folder_path) (column types mirror Document.source / Document.path), summary Text, updated_at timestamptz. folder_path = "" is the SOURCE ROOT (the top-level source summary); a folder is any other path prefix.
• Generation (app/rag/folder_summaries.py, mirroring app/rag/overview.py): marker FOLDER_SUMMARY_MODE (the E2E mock keys on it — tests/e2e/mock_llm.py gains the canned branch); input per folder = its recursive subtree — every document whose path equals the folder or starts with folder + "/" (the same set the ls count shows; folder_path "" = the whole source), each as path + title + first summary line — capped like the overview; lite one-shot via LLMClient.chat; the instruction asks for a 1–3 sentence plain-text, grounded description of what the folder's documents cover (shorter than a document summary — there can be hundreds of folders); non-empty validation, else LLMError → fail-soft (log, keep the previous summary — an old outline is better than none, the phase-31 rule).
• Gate: regenerate only when the import changed the KB (the overview's change gate: added/updated > 0) or the table is empty; --limit debug runs skip (mirror the overview); a lite failure never fails the sync (log + continue — the overview's overview=failed status token is NOT extended: folder summaries are auxiliary, the KB is the product).
• Scope rules: summarize folders with ≥ 2 documents (a single-document folder is fully described by its one file line); after a changed sync, DELETE rows for folders that no longer have ≥ 2 documents (pruned/renamed folders — a 3→1 doc folder's summary goes stale and is dropped); rows for untouched folders persist (an unchanged folder's summary is still true).

Accuracy & performance gate (task 05)

TOOL_CALLING_TESTING.md's controlled loop: uv run python -m scripts.agent_realmodel_check --restore --mode fixture — the 10-turn fixture battery against the live model through the real grounded path. The verdict line + per-turn wall times + the comparison against the recorded phase-70/72 baseline go into TOOL_CALLING_TESTING.md (dated section). Accuracy (contract/verdict) is the gate; wall time is reported. If a turn regresses, iterate on the ls copy levers (output format / TOOLS_SECTION teaching) with the micro-loop (--turns 3) per the methodology, then re-run the full battery.

Tasks

1. 01_folder_summary_model.md — migration 0017 + FolderSummary model + app/rag/folder_summaries.py generator (marker, fail-soft, prune) + mock-LLM branch
2. 02_sync_wiring.md — change-gated generation in both sync paths (scripts/import_docs.py, app/api/sync.py)
3. 03_ls_tree.md — the three-level ls rewrite (app/rag/agent.py) + AGENT_TOOLS description + TOOLS_SECTION teaching
4. 04_e2e_drilldown.md — dedicated Playwright suite: scripted drill-down turns, tree shapes, cap + note
5. 05_realmodel_battery.md — controlled 10-turn battery vs baseline, verdict recorded in TOOL_CALLING_TESTING.md

Testing & Quality

• Unit/integration: folder extraction + grouping; summary prompt building + capped input + fail-soft + prune; generator unit (mock LLM); sync wiring integration (both paths, change-gated, fail-soft, same-transaction convention); ls tree output for every level/edge (empty source, unknown source, unknown folder, root vs nested, single-doc folder, 50+ file cap + note); holder.tool_calls accounting; read/grep untouched.
• Coverage: >90% on new/modified code (uv run pytest --cov=app --cov-report=term-missing).
• E2E: uv run pytest tests/e2e/test_ls_tree_drilldown.py -v --no-cov in isolation; the existing agent-tool suites (test_agent_document_tools.py, test_agent_unlimited_tools.py, test_harness_aligned_tools.py, test_search_tool.py, test_grep_regex_teaching.py, test_response_to_docs.py) stay green in isolation.
• Lint/types: uv run ruff check . && uv run pyright.

Completion Criteria

• ls() lists sources with summaries; ls(source) / ls(source/folder) drill down; file lines capped at 50 + grep pointer
• folder summaries exist in the DB after a changed sync and are visible in ls output; a lite outage leaves old summaries intact and the sync green
• read/grep behavior byte-identical for their contracts (existing suites green)
• the 10-turn fixture battery verdict is recorded in TOOL_CALLING_TESTING.md and accuracy is at or above the phase-70/72 baseline (or an owner-acknowledged tradeoff is written down)
• test suite green, coverage >90%, ruff + pyright clean
• no behavior change in completed phases; one atomic Conventional Commit, --no-gpg-sign (e.g. feat(agent): make ls a drill-down tree with sync-time folder summaries)