phase: 94_ls_tree_drilldown

All green. Verification complete.

**Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)**

- Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal
- Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths
- Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met
- `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched)
- Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed
- Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol)

**Next pending phase:** `95_read_truncation_cap`

app/api/chat.py

@@ -618,10 +618,10 @@ async def chat(
                 )
                 return
             except Exception:  # noqa: BLE001 — a tool call hit the DB mid-stream
-                # Phase 37: tool execution (list_catalog / find_document)
-                # runs inside the stream now; a mid-turn DB failure gets
-                # the same structured ``error`` event as the pre-stream
-                # retrieval path.
+                # Phase 37: tool execution (the drill-down ``ls`` /
+                # ``read`` / ``grep`` lookups) runs inside the stream
+                # now; a mid-turn DB failure gets the same structured
+                # ``error`` event as the pre-stream retrieval path.
                 logger.exception(
                     "chat: tool execution failed question=%r total_ms=%d",
                     request.message,

app/api/sync.py

@@ -41,7 +41,16 @@ decisions):
    (pruning covers the union; the CLI's no-prune default is unchanged);
 5. when the import changed the KB (added + updated > 0),
    ``regenerate_overview`` refreshes the single ``kb_overview`` row
-   (phase 31 trigger, best-effort inside);
+   (phase 31 trigger, best-effort inside) — and ``generate_
+   folder_summaries`` (phase 94, task 02) regenerates the stored folder
+   summaries (the drill-down ``ls``'s per-level descriptions): its gate
+   is the same change trigger **plus** an empty ``folder_summaries``
+   table (the first sync after migration 0017 — the KB may predate the
+   table). It is per-folder fail-soft (a ``lite`` outage keeps the
+   failed folders' previous rows and never flips the run to
+   ``failed``) and only flushes — this run's own short-lived session
+   commits (the phase-53 convention), so a folder failure never blocks
+   step 6's bump;
 6. when the import changed the KB (added + updated + pruned > 0 — the
    saved-chat invalidation gate, phase 53 task 02: a pruned document
    can invalidate a saved answer that cited it, deliberately broader
@@ -79,6 +88,7 @@ from app.config import get_settings
 from app.core.auth import require_admin
 from app.core.errors import sanitize_error as _sanitize_error
 from app.db import SessionLocal
+from app.rag.folder_summaries import folder_summary_table_empty, generate_folder_summaries
 from app.rag.git_sources import effective_sources
 from app.rag.importer import ImportSummary, import_sources
 from app.rag.llm import LLMClient, check_models
@@ -251,6 +261,26 @@ async def _run_sync() -> None:
         overview = False
         if summary.added + summary.updated > 0:
             overview = await regenerate_overview(llm)
+        # Phase 94 (task 02): the folder summaries — the drill-down
+        # ls's per-level descriptions. Same change gate as the overview
+        # (added + updated > 0), plus the table-empty first-run trigger
+        # (the first sync after migration 0017 — the KB may have been
+        # imported by the CLI before the table landed). The generator is
+        # per-folder fail-soft (a lite outage never flips the run to
+        # failed) and only flushes: this run's own short-lived session
+        # commits (the phase-53 convention), so the step-6 bump stays
+        # change-gated on the KB, not on the summaries. No status-surface
+        # change: the stats are log-only (the detail shape is untouched).
+        fs_db = SessionLocal()
+        try:
+            if summary.added + summary.updated > 0 or folder_summary_table_empty(fs_db):
+                folder_stats = await generate_folder_summaries(fs_db, llm)
+                fs_db.commit()
+                logger.info("sync: folder_summaries stats=%s", folder_stats)
+            else:
+                logger.info("sync: folder_summaries skipped (KB unchanged)")
+        finally:
+            fs_db.close()
         # Phase 53 (task 02): a sync that changed the KB advances the
         # sources version exactly once — the saved-chat invalidation
         # marker (task 03 stamps rows against it). The gate is

app/config.py

@@ -135,6 +135,14 @@ class Settings(BaseSettings):
     #: ``app.rag.overview``). Overflow is cut at the cap and the shared
     #: ``[…truncated…]`` marker is appended (summarizer convention).
     overview_input_max_chars: int = 40_000
+    #: Cap on the folder document list (path/title/first summary line per
+    #: row) sent to the ``lite`` folder-summary model in ONE call
+    #: (phase 94, ``app.rag.folder_summaries``). Overflow is cut at the
+    #: cap and the shared ``[…truncated…]`` marker is appended
+    #: (summarizer convention). Smaller than the KB-overview cap on
+    #: purpose: a folder sees its own subtree only — there can be
+    #: hundreds of folders, each summarized separately at sync time.
+    folder_summary_input_max_chars: int = 8_000
     #: Hard cap on the agent tool rounds per grounded turn (phase 45,
     #: revising phase 37's per-tool budgets — owner permission
     #: 2026-08-27, TODO L8: "allow the LLM to make as many tool calls

app/models.py

@@ -63,6 +63,10 @@ Data model — see ``.agents/PLAN.md`` §Data Model:
                       (``id = 1``); every column NULL = "use the
                       default" (env value for the strings, the built-in
                       palette for the colors — task 01).
+* ``folder_summaries`` — one row per folder with ≥ 2 documents:
+                      the sync-time ``lite`` summary the drill-down
+                      ``ls`` shows next to each folder (phase 94;
+                      ``folder_path = ""`` = the source root).
 """
 from __future__ import annotations
 
@@ -446,3 +450,51 @@ class UiSettings(Base):
     accent_bg: Mapped[str | None] = mapped_column(String(7), nullable=True)
     accent_ink: Mapped[str | None] = mapped_column(String(7), nullable=True)
     accent_line: Mapped[str | None] = mapped_column(String(7), nullable=True)
+
+
+class FolderSummary(Base):
+    """One sync-time folder summary (phase 94, task 01).
+
+    ``ls`` is a drill-down tree (phase 94, ``00_phase.md``): the top
+    level lists the synced projects, each with its stored source-root
+    summary; drilling into a source lists its folders, each with its
+    stored folder summary. This table holds those summaries:
+
+    * PK ``(source, folder_path)`` — ``source`` mirrors
+      ``documents.source`` (String(120)); ``folder_path`` mirrors
+      ``documents.path`` (String(1000)) and is the source-relative
+      folder prefix. ``folder_path = ""`` is the SOURCE ROOT — the
+      top-level source summary (the whole source's recursive subtree).
+    * Rows exist only for folders with ≥ 2 documents (recursive count
+      — the same set the ``ls`` count rule counts): a
+      single-document folder is fully described by its one file line,
+      so no ``lite`` burn. After a changed sync, rows whose folder
+      dropped below 2 documents are pruned (a pruned/renamed folder's
+      summary would otherwise go stale); rows for folders that still
+      have ≥ 2 documents persist (an unchanged folder's summary is
+      still true). Both rules are generator policy (``app.rag.
+      folder_summaries``), not schema constraints.
+    * ``summary`` — the 1–3 sentence plain-text description the aipi
+      ``lite`` model wrote at sync time (``FOLDER_SUMMARY_MODE``,
+      ``app.rag.folder_summaries`` — change-gated and fail-soft like
+      the KB overview: an old summary is better than none).
+
+    Chat turns only READ these rows (the agent's ``ls`` output, phase
+    94 task 03) — generation happens at sync time only (task 02).
+    """
+
+    __tablename__ = "folder_summaries"
+
+    #: Mirrors ``documents.source`` (String(120)) — PK part 1.
+    source: Mapped[str] = mapped_column(String(120), primary_key=True)
+    #: Mirrors ``documents.path`` (String(1000)) — the source-relative
+    #: folder prefix; ``""`` = the source root. PK part 2.
+    folder_path: Mapped[str] = mapped_column(String(1000), primary_key=True)
+    #: The lite-written 1–3 sentence description — never empty (the
+    #: generator validates before storing, ``app.rag.folder_summaries``).
+    summary: Mapped[str] = mapped_column(Text)
+    #: Fresh UTC stamp on every upsert (the ``kb_overview.updated_at``
+    #: precedent; the generator always sets it explicitly).
+    updated_at: Mapped[datetime] = mapped_column(
+        DateTime(timezone=True), server_default=func.now(), onupdate=func.now()
+    )

app/rag/agent.py

@@ -1,5 +1,6 @@
 """Agent loop: the grounded-turn document tools (phase 37, task 03; the
-harness-aligned ``ls``/``read``/``grep`` surface, phase 70).
+harness-aligned ``ls``/``read``/``grep`` surface, phase 70; the
+drill-down tree ``ls`` + sync-time folder summaries, phase 94).
 
 Probe verdict (task 01 — ``uv run python -m scripts.llm_probe --tools``
 run live against aipi): **``probe: turbo tool_calls=supported 2026-08-26``**
@@ -48,14 +49,41 @@ task 04):
    split (with its self-correction and "teach the split" refusals) is
    gone — the model's combined form is now simply correct. The
    phase-68 A5 match/output contract rides along under the new name.
+   Phase 94 revision (owner permission 2026-09-10, ``TODO.md`` L4 — the
+   tool-surface revision recorded in the phase 94 overview ``00_phase.md``;
+   PLAN.md is being redone by the owner): the ``ls`` RESULT format and
+   ``path`` semantics changed — ``ls`` is now a filesystem-style
+   drill-down tree (no path: the synced sources with counts + stored
+   source-root summaries; a source name: its top-level folders + files;
+   a ``source/folder`` path: that folder's subfolders + files — the
+   model drills one level per call instead of flooding the whole
+   catalog into one result), while the tool NAME and the
+   ``read``/``grep`` contract (combined ``source/path``) are untouched.
 2. Each tool call the model emits is executed server-side against
-   Postgres only (no LLM, no network): ``ls`` returns the indexed
-   catalog — one ``source: X | path: Y | title: Z`` line per document
-   (phase 63: labeled fields — unambiguous for LLM parsing),
-   ``GET /api/docs`` order (uncapped in v1; the UI never shows it, only
-   the model does) — optionally scoped to one source name (a ``path``
-   argument matching no source name is a refusal; a registered source
-   with no indexed documents lists as ``0 documents:`` and counts) —
+   Postgres only (no LLM, no network): ``ls`` lists ONE level of the
+   drill-down tree (phase 94, task 03) — no ``path``: every registered
+   source (registry order; a 0-document source still lists) as
+   ``{source} — {n} documents`` plus the indented stored source-root
+   summary line when one is in ``folder_summaries``; a ``path`` that is
+   a registered source name (no ``/``): that source's root folder —
+   each direct subfolder ``  {sub}/ — {m} documents[: {summary}]``
+   (the count is the subfolder's recursive subtree — every document
+   whose path equals the folder or starts with ``folder + "/"``, the
+   same set the sync-time folder summary describes — and the file
+   lines ``source: X | path: Y | title: Z`` (the canonical
+   ``read``/``grep`` identity — the phase-63 labeled format,
+   unchanged) in path order (``GET /api/docs`` order), capped at
+   :data:`LS_MAX_FILE_LINES` lines + one deterministic grep-pointer
+   note for the rest (a 500-file folder costs 50 lines, never 500);
+   a ``source/folder`` ``path``: that folder's subfolders + own file
+   lines (the same template, ``identity = source + "/" + folder``);
+   a registered source with no documents lists its header line alone
+   (``… — 0 documents, 0 folders:`` — the old ``0 documents:``
+   behavior preserved in spirit) — every successful listing (top/root/
+   folder) counts; a ``source/…`` argument whose first segment names no
+   registered source is the no-source refusal (below, the segment
+   echoed), and a folder segment matching no indexed prefix is the
+   :data:`NOT_A_FOLDER` teaching (below) —
    ``read`` takes the combined ``source/path`` string, splits it at the
    FIRST ``'/'`` (source names are directory basenames — they can never
    contain ``'/'``), and returns the document's **full** content
@@ -82,15 +110,19 @@ task 04):
    blank or non-string) → ``"read requires a string argument
    'path'."``; a ``grep`` without a usable ``pattern`` (missing, blank
    or non-string) → ``"grep requires a string argument
-   'pattern'."``; a scoped ``ls`` whose (stripped) ``path`` contains a
-   ``/`` — a document path where a source name belongs (source names
-   are directory basenames and can never contain one; the 2026-09-03
-   incident's ``ls(path='app/rag/importer.py')``) →
-   :data:`LS_PATH_NOT_A_SOURCE`, the document-path teaching line with
-   the argument echoed; a scoped ``ls`` whose ``path`` names no
-   registered source (no ``/`` — the incident's ``ls(path='.')``)
-   → :data:`NO_SOURCE_NOT_A_DIRECTORY`, the no-source refusal with the
-   teaching parenthetical appended; a document
+   'pattern'."``; a scoped ``ls`` whose ``path``'s FIRST segment (split
+   at the first ``/``) names no registered source — a bare unknown
+   name (the 2026-09-03 incident's ``ls(path='.')``) or the source
+   segment of a ``source/…`` argument (phase 94: a ``/`` now names a
+   folder, so the phase-72 document-path teaching refusal is deleted)
+   → :data:`NO_SOURCE_NOT_A_DIRECTORY`, the
+   no-source refusal with the teaching parenthetical appended, the
+   segment echoed; a ``source/…`` argument whose folder segment
+   matches no indexed prefix (the phase-94 existence rule — some
+   indexed path of the source starts with ``folder + "/"``) →
+   :data:`NOT_A_FOLDER`, the drill-down teaching with the argument
+   echoed and the deepest existing ancestor's direct subfolders
+   listed, so the model self-corrects in the next round; a document
    already in context (seed or previously read) →
    :data:`ALREADY_IN_CONTEXT` (phase 72, task 05 gate iteration:
    the line names the correct action — answer from the text already
@@ -164,25 +196,30 @@ needs none either (the tool ran) — the policy keys on the no-calls
 exit only. No model participates in detection or repair: the
 registry + the fixed retry policy are the whole guardrail.
 
-The DB accessors (:func:`list_catalog`, :func:`list_source_names`,
-:func:`find_document`, :func:`all_documents`) and the
-:func:`grep_document` line matcher are module-level functions so unit
-tests can monkeypatch them without a database.
+The DB accessors — the drill-down ``ls`` (:func:`ls_top`,
+:func:`ls_folder`; the pure grouping :func:`group_folder_listing` and
+the pure renderers :func:`render_ls_top` /
+:func:`render_folder_listing` sit next to them),
+:func:`list_source_names`, :func:`find_document`,
+:func:`all_documents` — and the :func:`grep_document` line matcher are
+module-level functions so unit tests can monkeypatch them without a
+database.
 """
 from __future__ import annotations
 
 import json
 import logging
 import re
-from collections.abc import AsyncIterator, Sequence
+from collections.abc import AsyncIterator, Mapping, Sequence
 from dataclasses import dataclass, field
 from typing import Any
 
-from sqlalchemy import select
+from sqlalchemy import func, select
 from sqlalchemy.orm import Session
 
 from app.config import Settings
-from app.models import Document
+from app.models import Document, FolderSummary
+from app.rag.folder_summaries import folder_of
 from app.rag.git_sources import effective_sources
 from app.rag.llm import (
     LLMClient,
@@ -212,9 +249,17 @@ AGENT_TOOLS: list[dict[str, Any]] = [
         "function": {
             "name": "ls",
             "description": (
-                "List the indexed documents as `source: X | path: Y | "
-                "title: Z` lines. Call one tool at a time — wait for "
-                "this result before your next call."
+                "List the knowledge base as a tree, one level at a "
+                "time. With no path: the synced sources — each with "
+                "its document count and a summary of its contents. "
+                "With a source name (no '/'): that source's top-level "
+                "folders and files. With a `source/folder` path: that "
+                "folder's subfolders and files. Folder lines carry a "
+                "summary of what the folder contains. File lines are "
+                "`source: X | path: Y | title: Z` — use the combined "
+                "`source/path` with `read` and `grep`. Call one tool "
+                "at a time — wait for this result before your next "
+                "call."
             ),
             "parameters": {
                 "type": "object",
@@ -222,13 +267,11 @@ AGENT_TOOLS: list[dict[str, Any]] = [
                     "path": {
                         "type": "string",
                         "description": (
-                            "Source name to list one source's documents "
-                            "(e.g. 'homelab') — a source name, not a "
-                            "file or directory path; omit to list "
-                            "every document. This is the only tool "
-                            "whose `path` is a source name — for "
-                            "`read` and `grep` it must be a document's "
-                            "combined `source/path`."
+                            "Optional — a source name (e.g. "
+                            "'homelab') to list its top level, or a "
+                            "`source/folder` path to drill down "
+                            "(e.g. 'homelab/active'). Omit it to list "
+                            "every source."
                         ),
                     }
                 },
@@ -338,21 +381,6 @@ UNKNOWN_TOOL = "Unknown tool."
 MISSING_READ_ARGS = "read requires a string argument 'path'."
 MISSING_SEARCH_ARGS = "grep requires a string argument 'pattern'."
 
-#: Teaching refusal for a scoped ``ls`` whose stripped ``path``
-#: contains a ``/`` (phase 72): a source name is a directory basename
-#: and can never contain one, so the argument is a document path passed
-#: where a source name belongs (the 2026-09-03 incident's
-#: ``ls(path='app/rag/importer.py')``). One ``{path}`` field — the
-#: argument echoed; a fixed template states the correct contract
-#: instead of the terse pre-phase-72 line, so the harness-prior misuse
-#: self-corrects in one round.
-LS_PATH_NOT_A_SOURCE = (
-    "'{path}' looks like a document path, not a source name. The "
-    "'path' argument of ls filters by source name (e.g. 'homelab') — "
-    "omit it to list every document, or read a document by its "
-    "combined 'source/path' string."
-)
-
 #: The no-source ``ls`` refusal with the teaching parenthetical
 #: appended (phase 72): used when a stripped scope has no ``/`` and
 #: matches no registered source (the incident's ``ls(path='.')``). The
@@ -394,6 +422,29 @@ NO_DOCUMENT_DID_YOU_MEAN_MANY = (
 #: dropped).
 SUGGESTION_LIMIT = 3
 
+#: The drill-down ``ls`` file-line cap (phase 94, task 03): a folder's
+#: own files list at most this many ``source: X | path: Y | title: Z``
+#: lines (path order), then one deterministic grep-pointer note — a
+#: 500-file folder costs the model 50 lines + the note, never 500.
+#: Pinned module constant (no env var — the phase-94 TODO asks for a
+#: shape change, not a knob; the constant lives next to
+#: :data:`SEARCH_MAX_MATCHES`).
+LS_MAX_FILE_LINES = 50
+
+#: The NOT-A-FOLDER ``ls`` teaching refusal (phase 94, task 03):
+#: a ``source/…`` argument whose folder segment matches no indexed
+#: prefix (the ``00_phase.md`` existence rule — a folder exists iff
+#: some indexed path starts with ``folder + "/"``; a document's own
+#: path is never a folder). Phase-72 teaching style: one line, the
+#: argument echoed (``{arg}``), the DEEPEST existing ancestor's name
+#: (``{parent}`` — the source for a top-level miss, ``source/folder``
+#: for a nested one) and its direct subfolders (``{subfolders}`` —
+#: space-joined ``name/`` entries in path order, so the model
+#: self-corrects in the next round; ``none`` when the ancestor has no
+#: subfolders). Still a refusal: it counts in nothing and consumes a
+#: round (no silent argument normalization).
+NOT_A_FOLDER = "'{arg}' is not a folder — {parent} has: {subfolders}"
+
 #: The harness-owned recovery line (phase 71, task 03) — folded into the
 #: ORIGINAL single system message of the one bounded recovery request
 #: (``system_prompt + "\n" + CORRECTION_INSTRUCTION``; provider-safe,
@@ -507,20 +558,6 @@ def plain_form(pattern: str) -> str:
     return p.strip()
 
 
-def list_catalog(db: Session) -> list[tuple[str, str, str]]:
-    """Every indexed document as ``(source, path, title)``.
-
-    Ordered by ``(source, path)`` — the same order as ``GET /api/docs``.
-    Module-level (not a method) so unit tests can monkeypatch it.
-    """
-    rows = db.execute(
-        select(Document.source, Document.path, Document.title).order_by(
-            Document.source, Document.path
-        )
-    ).all()
-    return [(source, path, title) for source, path, title in rows]
-
-
 def list_source_names(db: Session) -> list[str]:
     """Every registered source name, deduped, in registry order.
 
@@ -546,6 +583,286 @@ def list_source_names(db: Session) -> list[str]:
     return names
 
 
+#: The drill-down ``ls`` fetchers (phase 94, task 03) — module-level so
+#: unit tests can monkeypatch them without a database (the house style:
+#: :func:`ls_top` / :func:`ls_folder` compose them; the pure grouping
+#: :func:`group_folder_listing` and renderers
+#: :func:`render_ls_top` / :func:`render_folder_listing` sit below).
+
+
+def _source_document_counts(db: Session) -> list[tuple[str, int]]:
+    """``(source, document count)`` per indexed source (one grouped
+    query — the top-level ``ls`` counts, phase 94 task 03)."""
+    return [
+        (source, count)
+        for source, count in db.execute(
+            select(Document.source, func.count()).group_by(Document.source)
+        )
+    ]
+
+
+def _source_root_summaries(db: Session) -> list[tuple[str, str]]:
+    """The stored source-root summaries ``(source, summary)``
+    (``folder_path = ""`` — the top-level ``ls`` indented lines, phase
+    94 task 03; absent when never generated)."""
+    return [
+        (source, summary)
+        for source, summary in db.execute(
+            select(FolderSummary.source, FolderSummary.summary).where(
+                FolderSummary.folder_path == ""
+            )
+        )
+    ]
+
+
+def _source_document_rows(db: Session, source: str) -> list[tuple[str, str]]:
+    """``(path, title)`` of every document under *source*, ordered by
+    ``path`` — the one bounded fetch a folder drill level lists (phase
+    94 task 03; one source's paths, not the whole KB)."""
+    return [
+        (path, title)
+        for path, title in db.execute(
+            select(Document.path, Document.title)
+            .where(Document.source == source)
+            .order_by(Document.path)
+        )
+    ]
+
+
+def _source_folder_summaries(db: Session, source: str) -> dict[str, str]:
+    """The stored folder summaries ``{folder_path: summary}`` of one
+    source (phase 94 task 03; includes the ``""`` source-root row when
+    stored)."""
+    return {
+        folder_path: summary
+        for folder_path, summary in db.execute(
+            select(FolderSummary.folder_path, FolderSummary.summary).where(
+                FolderSummary.source == source
+            )
+        )
+    }
+
+
+def ls_top(db: Session) -> list[tuple[str, int, str | None]]:
+    """The top level of the drill-down ``ls`` (phase 94, task 03).
+
+    Every registered source in :func:`list_source_names` order (the
+    registry is the source of truth — a registered source with 0
+    indexed documents still lists, the phase-70/72 invariant) as
+    ``(name, recursive_document_count, source_root_summary)``: the
+    count is the source's whole subtree (all of its documents — the
+    same set its stored summary describes) and the summary is the
+    stored ``folder_summaries`` row for ``(source, "")`` (the source
+    root, phase 94 task 01) or ``None`` when absent. Module-level so
+    unit tests can monkeypatch the fetchers.
+    """
+    names = list_source_names(db)
+    if not names:
+        return []
+    counts = dict(_source_document_counts(db))
+    summaries = dict(_source_root_summaries(db))
+    return [(name, counts.get(name, 0), summaries.get(name)) for name in names]
+
+
+def group_folder_listing(
+    source: str,
+    folder: str,
+    rows: Sequence[tuple[str, str]],
+    summaries: Mapping[str, str],
+) -> tuple[list[tuple[str, int, str | None]], list[tuple[str, str, str]], int]:
+    """One level of the drill-down tree (phase 94, task 03) — pure.
+
+    Given *rows* — the source's ``(path, title)`` pairs in catalog
+    (path) order — and *summaries* (the source's stored
+    ``folder_summaries`` rows: ``folder_path → summary``), the folder
+    level *folder* (source-relative; ``""`` = the source root):
+
+    * **(a) direct subfolders** — the folders whose parent is exactly
+      *folder*, in path order, each
+      ``(sub_path_relative_to_source, recursive_count,
+      summary_or_None)``. A folder is a slash-boundary prefix of at
+      least one indexed path (the ``00_phase.md`` existence rule: a
+      folder ``F`` exists ⟺ some path starts with ``F + "/"`` — a
+      document's OWN path is never a folder); the count is the
+      folder's recursive subtree — every path equal to the folder or
+      starting with ``folder + "/"`` (the same set the sync-time
+      folder summary describes, phase 94 task 01 — one concept end to
+      end).
+    * **(b) direct file lines** — the documents whose folder (the
+      prefix before the last ``/`` —
+      :func:`app.rag.folder_summaries.folder_of`, the shared notion) IS
+      *folder*, in path order (catalog order — the same order
+      ``GET /api/docs`` serves), as ``(source, path, title)`` triples
+      — the canonical ``read``/``grep`` identity, capped at
+      :data:`LS_MAX_FILE_LINES` (the rest fold into the renderer's
+      note; a 500-file folder never costs 500 lines).
+    * **(c) the TOTAL direct-file count** — pre-cap, for the note.
+
+    Pure (no I/O) — unit tests drive the grouping without a database;
+    :func:`ls_folder` is the DB-composing wrapper.
+    """
+    # The source's existing folders: every slash-boundary prefix of an
+    # indexed path (the existence rule's candidate set — a folder is
+    # present iff at least one path starts with ``folder + "/"``).
+    folders: set[str] = set()
+    for path, _title in rows:
+        f = folder_of(path)
+        while f:
+            folders.add(f)
+            f = folder_of(f)
+    # The recursive count per folder — the ``path == folder`` arm (a
+    # document sharing a folder's name) plus the ``startswith
+    # folder + "/"`` arm (the folder's true descendants), one pass per
+    # document.
+    counts: dict[str, int] = {f: 0 for f in folders}
+    for path, _title in rows:
+        if path in folders:
+            counts[path] += 1
+        f = folder_of(path)
+        while f:
+            counts[f] += 1
+            f = folder_of(f)
+    subfolders = [
+        (g, counts[g], summaries.get(g))
+        for g in sorted(g for g in folders if folder_of(g) == folder)
+    ]
+    files = [
+        (source, path, title)
+        for path, title in rows
+        if folder_of(path) == folder
+    ]
+    return subfolders, files[:LS_MAX_FILE_LINES], len(files)
+
+
+def ls_folder(
+    db: Session, source: str, folder: str
+) -> tuple[list[tuple[str, int, str | None]], list[tuple[str, str, str]], int]:
+    """One folder level of the drill-down ``ls`` (phase 94, task 03).
+
+    The source's document rows (:func:`_source_document_rows`) and
+    stored folder summaries (:func:`_source_folder_summaries`) through
+    :func:`group_folder_listing` — the pure grouping the unit tests
+    drive directly. ``folder = ""`` is the source root. Module-level
+    so unit tests can monkeypatch the fetchers.
+    """
+    return group_folder_listing(
+        source,
+        folder,
+        _source_document_rows(db, source),
+        _source_folder_summaries(db, source),
+    )
+
+
+def _folder_exists_in(rows: Sequence[tuple[str, str]], folder: str) -> bool:
+    """The phase-94 folder-existence rule (``00_phase.md``), pure.
+
+    Folder *folder* (source-relative) under a registered source
+    exists ⟺ ``folder == ""`` OR some indexed path starts with
+    ``folder + "/"`` — a document's OWN path is never a folder
+    (nothing starts with ``path + "/"``), so ``ls`` of a file path
+    refuses with :data:`NOT_A_FOLDER` rather than listing.
+    """
+    if not folder:
+        return True
+    prefix = folder + "/"
+    return any(path.startswith(prefix) for path, _title in rows)
+
+
+def _deepest_existing_ancestor(
+    rows: Sequence[tuple[str, str]], folder: str
+) -> str:
+    """The deepest EXISTING folder prefix of a missing *folder* (pure).
+
+    The :data:`NOT_A_FOLDER` refusal's teaching context: the argument's
+    segments are walked from the top; the walk stops at the first
+    segment that is no folder, so the returned prefix is the deepest
+    existing ancestor (``""`` = the source root when the first segment
+    is already the miss) and its direct subfolders are the bounded
+    self-correction list the refusal prints.
+    """
+    parent = ""
+    for part in folder.split("/"):
+        candidate = f"{parent}/{part}" if parent else part
+        if not _folder_exists_in(rows, candidate):
+            break
+        parent = candidate
+    return parent
+
+
+def render_ls_top(entries: Sequence[tuple[str, int, str | None]]) -> str:
+    """The top-level ``ls`` result (phase 94, task 03) — the pinned
+    template.
+
+    ``{N} sources:`` — the line alone when the registry is empty (the
+    old ``0 documents:`` behavior preserved in spirit) — then, when at
+    least one source is registered, a blank line and one block per
+    source in registry order: ``{source} — {n} documents`` plus the
+    indented ``  {summary}`` line ONLY when the source-root summary is
+    stored (absent → the count line alone, no placeholder) — with NO
+    blank line between blocks.
+    """
+    lines = [f"{len(entries)} sources:"]
+    if entries:
+        lines.append("")
+        for source, count, summary in entries:
+            lines.append(f"{source} — {count} documents")
+            if summary:
+                lines.append(f"  {summary}")
+    return "\n".join(lines)
+
+
+def render_folder_listing(
+    identity: str,
+    subfolders: Sequence[tuple[str, int, str | None]],
+    files: Sequence[tuple[str, str, str]],
+    total_files: int,
+) -> str:
+    """One folder level of the drill-down ``ls`` (phase 94, task 03) —
+    the pinned template.
+
+    The header ``{identity} — {n_files} documents, {n_folders}
+    folders:`` (``n_files`` = *total_files*, the PRE-cap direct-file
+    count; ``identity`` is the source name at the root level and
+    ``source/folder`` below it), then — when the level carries anything
+    below the header — a blank line, the 2-space-indented subfolder
+    lines ``  {sub}/ — {m} documents`` in path order (``: {summary}``
+    appended ONLY when the subfolder's summary is stored), a blank
+    line, the file lines in EXACTLY the existing
+    ``source: X | path: Y | title: Z`` format (the canonical
+    ``read``/``grep`` identity — unchanged), and the cap note
+    ``…and {hidden} more documents in this folder — use grep
+    (pattern) to find a specific one.`` ONLY when the folder's own
+    files outnumber :data:`LS_MAX_FILE_LINES` (*files* arrives capped;
+    *total_files* carries the pre-cap count). A header with no
+    subfolders and no files — a registered source with no documents —
+    is the header line alone (``… — 0 documents, 0 folders:``).
+    """
+    header = f"{identity} — {total_files} documents, {len(subfolders)} folders:"
+    if not subfolders and not files:
+        return header
+    body: list[str] = []
+    if subfolders:
+        body.append("")
+        for sub, count, summary in subfolders:
+            line = f"  {sub}/ — {count} documents"
+            if summary:
+                line += f": {summary}"
+            body.append(line)
+    if files or total_files > len(files):
+        body.append("")
+        body.extend(
+            f"source: {source} | path: {path} | title: {title}"
+            for source, path, title in files
+        )
+        hidden = total_files - len(files)
+        if hidden > 0:
+            body.append(
+                f"…and {hidden} more documents in this folder — use grep "
+                "(pattern) to find a specific one."
+            )
+    return "\n".join([header, *body])
+
+
 def find_document(db: Session, source: str, path: str) -> Document | None:
     """The indexed document at ``(source, path)``, or ``None``.
 
@@ -704,30 +1021,61 @@ def _execute_tool(
     phase 70).
     """
     if call.name == "ls":
+        # Phase 94 (task 03): the drill-down tree — one level per call
+        # (the owner-permitted tool-surface revision; the old
+        # whole-catalog listing is gone). A successful listing at ANY
+        # level (top/root/folder) counts; a refusal counts in nothing
+        # and consumes a round like every refusal.
         raw_path = call.arguments.get("path")
         scope = raw_path.strip() if isinstance(raw_path, str) else ""
-        rows = list_catalog(db)
-        if scope:
-            if "/" in scope:
-                # A source name (a directory basename) can never
-                # contain '/' — this is a document path where a source
-                # name belongs (phase 72): teach the contract; no
-                # registry lookup needed, counts in nothing, consumes
-                # a round like every refusal.
-                return LS_PATH_NOT_A_SOURCE.format(path=scope)
-            if scope not in list_source_names(db):
-                # The no-source refusal with the teaching parenthetical
-                # (phase 72) — the prefix byte-identical to the
-                # pre-phase-72 line; counts in nothing, consumes a
-                # round like every refusal.
-                return NO_SOURCE_NOT_A_DIRECTORY.format(scope=scope)
-            rows = [row for row in rows if row[0] == scope]
-        listing = f"{len(rows)} documents:\n" + "\n".join(
-            f"source: {source} | path: {path} | title: {title}"
-            for source, path, title in rows
-        )
+        if not scope:
+            # The top level: the synced sources, registry order, each
+            # with its recursive document count and its stored
+            # source-root summary (``None`` → no indented line).
+            holder.tool_calls += 1
+            return render_ls_top(ls_top(db))
+        source, _, rest = scope.partition("/")
+        if source not in list_source_names(db):
+            # The no-source refusal with the teaching parenthetical
+            # (phase 72, the prefix byte-identical to the pre-phase-72
+            # line): the FIRST segment is the source candidate — a bare
+            # unknown name (the incident's ``ls(path='.')``) or the
+            # source segment of a ``source/…`` argument (phase 94: a
+            # ``/`` now names a folder, so the phase-72 document-path
+            # teaching is deleted) — the segment echoed; counts in
+            # nothing, consumes a round like every refusal.
+            return NO_SOURCE_NOT_A_DIRECTORY.format(scope=source)
+        if not rest:
+            # The source's ROOT folder: subfolders + own file lines
+            # (capped + note) — the pinned template; a registered
+            # source with no documents lists its header line alone
+            # (``… — 0 documents, 0 folders:`` — the old
+            # ``0 documents:`` behavior preserved in spirit).
+            subfolders, files, total = ls_folder(db, source, "")
+            holder.tool_calls += 1
+            return render_folder_listing(source, subfolders, files, total)
+        rows = _source_document_rows(db, source)
+        summaries = _source_folder_summaries(db, source)
+        if not _folder_exists_in(rows, rest):
+            # The NOT-A-FOLDER teaching (phase 94, task 03): the
+            # argument echoed, the DEEPEST existing ancestor's name and
+            # its direct subfolders listed (bounded — the parent's own
+            # listing, so no new flood path), so the model
+            # self-corrects in the next round; counts in nothing,
+            # consumes a round like every refusal.
+            parent = _deepest_existing_ancestor(rows, rest)
+            parent_subs = group_folder_listing(source, parent, rows, summaries)[0]
+            return NOT_A_FOLDER.format(
+                arg=scope,
+                parent=source if not parent else f"{source}/{parent}",
+                subfolders=" ".join(f"{sub}/" for sub, _c, _s in parent_subs)
+                or "none",
+            )
+        # The folder level: the same template as the root, identity =
+        # source + "/" + folder.
+        subfolders, files, total = group_folder_listing(source, rest, rows, summaries)
         holder.tool_calls += 1
-        return listing
+        return render_folder_listing(f"{source}/{rest}", subfolders, files, total)
     if call.name == "read":
         raw_path = call.arguments.get("path")
         arg = raw_path.strip() if isinstance(raw_path, str) else ""

app/rag/folder_summaries.py

@@ -0,0 +1,391 @@
+"""Folder summary generator (phase 94, task 01).
+
+``ls`` becomes a drill-down tree (phase 94, ``00_phase.md``): the LLM
+lists the synced projects, then drills into folders. Each level's
+listing shows the folder's **subtree summary** — a 1–3 sentence
+plain-text description of what the folder's documents cover, generated
+at SYNC time by the aipi ``lite`` model (the ``KB_OVERVIEW_MODE``
+contract of ``app.rag.overview``: change-gated by the caller,
+fail-soft — an old summary is better than none — and E2E-mockable via
+the ``FOLDER_SUMMARY_MODE`` system-prompt marker).
+
+ONE concept end to end (:func:`group_by_folder`): a folder row's
+documents are its **recursive subtree** — every document whose path
+equals the folder or starts with ``folder + "/"`` — exactly the set the
+``ls`` count rule (``00_phase.md``) counts. A document under ``a/b/``
+therefore contributes to the ``a``, ``a/b``, and ``""`` (source root)
+groups alike: each level's listing shows its own accurate subtree
+summary, and the number next to a folder is the number of documents its
+summary describes.
+
+Storage: ``folder_summaries`` (migration 0017) — PK
+``(source, folder_path)``; ``folder_path = ""`` is the SOURCE ROOT
+(the top-level source summary). Rows exist only for folders with
+≥ 2 documents (the :data:`MIN_DOCS_PER_FOLDER` rule — a
+single-document folder is fully described by its one file line, so no
+``lite`` burn); after a changed sync, rows whose folder dropped below
+2 documents are pruned (a pruned/renamed folder's summary would
+otherwise go stale), while rows for folders that still have
+≥ 2 documents persist (an unchanged folder's summary is still true).
+
+Chat turns never generate folder summaries — the agent's ``ls`` output
+(phase 94, task 03) only reads the stored rows. Generation is the
+caller's job at sync time (phase 94, task 02), and :func:`generate_
+folder_summaries` only flushes — the sync path owns the transaction
+(the phase-53 ``bump_sources_version`` convention).
+"""
+from __future__ import annotations
+
+import logging
+from collections.abc import Sequence
+from datetime import UTC, datetime
+from typing import Protocol
+
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+
+from app.config import Settings, get_settings
+from app.models import Document, FolderSummary
+from app.rag.llm import LLMError
+from app.rag.retriever import TRUNCATION_MARKER
+
+logger = logging.getLogger("app.rag.folder_summaries")
+
+#: System-prompt marker for folder-summary generation — the E2E mock
+#: LLM keys on it (same convention as ``SUMMARY_MODE`` /
+#: ``KB_OVERVIEW_MODE``, PLAN §6).
+FOLDER_SUMMARY_MODE = "FOLDER_SUMMARY_MODE"
+
+#: Locked instruction for the ``lite`` model (phase 94): the folder
+#: summary is the drill-down ``ls``'s per-level picture, so it must be
+#: a short plain-text description of what the folder's documents
+#: cover — 1–3 sentences (a folder is a skim, not a read: the document
+#: summary's 3–6 sentences would blow up a 20-folder listing),
+#: strictly grounded in the listed titles/paths/summary lines.
+FOLDER_SUMMARY_INSTRUCTION = (
+    "From the document list below, write a 1-3 sentence plain-text "
+    "summary of what this folder's documents cover, in natural "
+    "language. Do not use markdown. Do not invent anything that is not "
+    "in the list."
+)
+
+#: Full system prompt: marker first (the mock's key), then the
+#: instruction (the ``app.rag.overview.SYSTEM_PROMPT`` shape).
+SYSTEM_PROMPT = f"{FOLDER_SUMMARY_MODE}: {FOLDER_SUMMARY_INSTRUCTION}"
+
+#: First line of the user message — the folder the summary describes:
+#: ``<source>`` for the source root, ``<source>/<folder_path>`` for a
+#: folder. The deterministic E2E mock keys on it to name the folder in
+#: its canned reply (``tests/e2e/mock_llm.py``), so the stored row is
+#: a pure function of the request.
+FOLDER_HEADER_PREFIX = "Folder: "
+
+#: A folder is summarized only while its recursive subtree holds at
+#: least this many documents — a single-document folder is fully
+#: described by its one file line, so no ``lite`` burn (phase 94
+#: ``00_phase.md`` scope rule; the prune rule applies the same count).
+MIN_DOCS_PER_FOLDER = 2
+
+#: One document row for the grouping/prompting:
+#: ``(source, path, title, summary)`` — the ``app.rag.overview``
+#: ``build_overview_prompt`` row shape (``summary`` is the stored
+#: lite-written text or ``None`` for markdown docs / the fail-soft
+#: path).
+DocRow = tuple[str, str, str, str | None]
+
+
+class FolderSummaryLLM(Protocol):
+    """The one-shot chat surface the folder-summary generator needs.
+
+    :class:`app.rag.llm.LLMClient` satisfies it; unit tests pass a
+    duck-typed fake (``chat`` + ``settings``) instead — same pattern as
+    the overview's ``OverviewLLM`` protocol.
+    """
+
+    settings: Settings
+
+    async def chat(
+        self, messages: list[dict[str, str]], model: str | None = None
+    ) -> str: ...
+
+
+def folder_of(path: str) -> str:
+    """The directory prefix before the last ``/`` (``""`` for root-level files).
+
+    The single notion of "which folder owns this document" used by the
+    whole module: ``folder_of("a.md") == ""``,
+    ``folder_of("a/b.md") == "a"``, ``folder_of("a/b/c.md") == "a/b"``.
+    Iterating :func:`folder_of` over its own result walks a path's
+    folder prefixes from nearest to farthest, ending at ``""``.
+    """
+    idx = path.rfind("/")
+    return path[:idx] if idx >= 0 else ""
+
+
+def group_by_folder(rows: Sequence[DocRow]) -> dict[tuple[str, str], list[DocRow]]:
+    """Group document rows by the folders their recursive subtree fills.
+
+    The ONE concept of this module, documented in the module docstring:
+    a folder row's documents are its **recursive subtree** — every
+    document whose path equals the folder or starts with
+    ``folder + "/"`` — exactly the set the ``ls`` count rule counts.
+    Each row therefore lands in the ``""`` (source root) group AND in
+    the group of every folder prefix of its path: a doc under
+    ``a/b/`` contributes to the ``a``, ``a/b``, and ``""`` groups.
+    Per source the candidate rows are thus ``""`` (all of the source's
+    documents — the top-level source summary) plus every distinct
+    folder prefix of an indexed path.
+
+    *rows* are ``(source, path, title, summary)`` tuples (the
+    :data:`DocRow` shape, e.g. straight from the ``documents``
+    catalogue query). Returns ``{(source, folder_path): [rows]}`` —
+    group lists keep the input (catalogue) order, so downstream prompt
+    building is deterministic. Groups of ANY size (≥ 1) are returned;
+    the ≥ 2 :data:`MIN_DOCS_PER_FOLDER` rule is applied by
+    :func:`generate_folder_summaries`, not here.
+
+    """
+    # Pass 1: the distinct TRUE folder prefixes of the catalogue — a
+    # folder is a slash-boundary prefix of at least one indexed path
+    # (the ``00_phase.md`` candidate definition: ``""`` + every distinct
+    # folder prefix, per source).
+    folders: set[tuple[str, str]] = set()
+    for row in rows:
+        source, path = row[0], row[1]
+        folder = folder_of(path)
+        while folder:
+            folders.add((source, folder))
+            folder = folder_of(folder)
+
+    # Pass 2: every row lands in the source-root group, in the group of
+    # every ancestor folder (the ``startswith folder + "/"`` arm of the
+    # count rule), and — when its path IS one of the source's folder
+    # prefixes (a file sharing its name with a directory) — in that
+    # folder's group too (the ``path == folder`` arm). A row therefore
+    # belongs to its folder's group iff its path equals the folder or
+    # starts with ``folder + "/"`` — exactly the set the ``ls`` count
+    # rule counts, for EVERY group.
+    groups: dict[tuple[str, str], list[DocRow]] = {}
+    for row in rows:
+        source, path = row[0], row[1]
+        groups.setdefault((source, ""), []).append(row)
+        if (source, path) in folders:
+            groups.setdefault((source, path), []).append(row)
+        folder = folder_of(path)
+        while folder:
+            groups.setdefault((source, folder), []).append(row)
+            folder = folder_of(folder)
+    return groups
+
+
+def _first_summary_line(summary: str | None) -> str:
+    """First line of a stored summary, stripped; ``''`` when absent.
+
+    The stored summary (phase 30) ends in the code-appended
+    ``Source: …`` pointer line, so its first line is the model-written
+    lead sentence — the best one-line picture of the document for the
+    folder summary. Blank/whitespace-only summaries yield ``''`` as
+    well (the ``app.rag.overview`` helper, mirrored locally so each
+    lite-mode module stays self-contained).
+    """
+    if not summary:
+        return ""
+    for line in summary.splitlines():
+        stripped = line.strip()
+        if stripped:
+            return stripped
+    return ""
+
+
+def _folder_label(source: str, folder_path: str) -> str:
+    """The user-message folder identifier (the ``FOLDER_HEADER_PREFIX`` tail).
+
+    ``<source>`` for the source root (``folder_path = ""``),
+    ``<source>/<folder_path>`` for a folder — the canonical folder
+    identity the E2E mock echoes into its canned reply.
+    """
+    return source if not folder_path else f"{source}/{folder_path}"
+
+
+def build_folder_summary_prompt(
+    source: str,
+    folder_path: str,
+    docs: Sequence[DocRow],
+    max_chars: int | None = None,
+) -> tuple[str, str]:
+    """The ``(system, user)`` message pair for one folder-summary call.
+
+    * ``system`` — :data:`SYSTEM_PROMPT`: the ``FOLDER_SUMMARY_MODE``
+      marker + the locked instruction.
+    * ``user`` — the folder header line
+      (``FOLDER_HEADER_PREFIX + <source>[/<folder_path>]`` — the line
+      the E2E mock parses to name the folder), then one line per
+      document, ``path — title — {first line of summary}``, joined
+      with newlines, in the given (catalogue) order. The summary field
+      is omitted when the document has none (markdown docs and the
+      fail-soft path — no dangling dash, the overview convention). The
+      whole message is capped at *max_chars* (default
+      ``BOR_FOLDER_SUMMARY_INPUT_MAX_CHARS``): overflow is cut exactly
+      at the cap and the shared ``[…truncated…]`` marker is appended on
+      its own line, so the model never sees more than the cap and the
+      cut is visible (summarizer convention, phase 30).
+    """
+    lines = [FOLDER_HEADER_PREFIX + _folder_label(source, folder_path)]
+    for _source, path, title, summary in docs:
+        line = f"{path} — {title}"
+        first = _first_summary_line(summary)
+        if first:
+            line += f" — {first}"
+        lines.append(line)
+    user = "\n".join(lines)
+    limit = (
+        max_chars
+        if max_chars is not None
+        else get_settings().folder_summary_input_max_chars
+    )
+    if len(user) > limit:
+        user = user[:limit] + "\n" + TRUNCATION_MARKER
+    return SYSTEM_PROMPT, user
+
+
+async def summarize_folder(
+    source: str, folder_path: str, docs: Sequence[DocRow], llm: FolderSummaryLLM
+) -> str:
+    """One-shot ``lite`` summary of one folder's recursive subtree.
+
+    Builds the prompt from the folder's documents (each ``path``,
+    ``title``, first summary line), makes ONE :meth:`LLMClient.chat`
+    call against ``llm.settings.llm_summary_model`` (the ``lite``
+    model — no new model management), and returns the model's text
+    trimmed. Raises :class:`LLMError` when the reply is empty after
+    trimming (the client already rejects empty content; re-asserted
+    defensively, the summarizer's rule — a silent summary must never be
+    stored), and propagates any :class:`LLMError` the client raises
+    (the generator's fail-soft path catches it per folder).
+    """
+    system, user = build_folder_summary_prompt(source, folder_path, docs)
+    raw = await llm.chat(
+        [{"role": "system", "content": system}, {"role": "user", "content": user}],
+        model=llm.settings.llm_summary_model,
+    )
+    summary = raw.strip()
+    if not summary:
+        label = _folder_label(source, folder_path)
+        raise LLMError(
+            f"folder summary model returned empty content for {label} — "
+            "refusing to store a silent summary"
+        )
+    return summary
+
+
+def _upsert(db: Session, source: str, folder_path: str, summary: str) -> None:
+    """Insert or update the row for one folder (fresh UTC stamp)."""
+    now = datetime.now(UTC)
+    row = db.get(FolderSummary, (source, folder_path))
+    if row is None:
+        db.add(
+            FolderSummary(
+                source=source, folder_path=folder_path, summary=summary, updated_at=now
+            )
+        )
+    else:
+        row.summary = summary
+        row.updated_at = now
+
+
+def folder_summary_table_empty(db: Session) -> bool:
+    """Whether ``folder_summaries`` holds no rows (the sync-path gate).
+
+    Phase 94 (task 02): the ``_overview_row_exists`` pattern
+    (``scripts.import_docs``) extended to a table-emptiness check —
+    after an unchanged re-sync, an EMPTY table (the first full sync
+    after migration 0017, or after a ``--limit`` debug walk that
+    skipped generation) still gets a fresh batch, while a populated
+    table is left untouched until the KB actually changes. One
+    bounded ``LIMIT 1`` probe, never a count scan.
+    """
+    return db.execute(select(FolderSummary.source).limit(1)).first() is None
+
+
+async def generate_folder_summaries(
+    db: Session, llm: FolderSummaryLLM, *, skip: bool = False
+) -> dict[str, int]:
+    """Regenerate the stored folder summaries for the current catalogue.
+
+    The sync-path orchestrator (phase 94, task 02 calls it
+    change-gated, like the KB overview; ``--limit`` debug runs pass
+    ``skip=True``). Steps, in order:
+
+    1. ``skip=True`` → a no-op: the zero stats dict is returned, the
+       LLM is never called, and no rows are touched.
+    2. Group the document catalogue by :func:`group_by_folder` (the
+       recursive-subtree concept) and keep the candidate folders —
+       the ones whose recursive subtree holds
+       :data:`MIN_DOCS_PER_FOLDER` (≥ 2) documents. Single-document
+       folders get no row (their one file line IS their summary).
+    3. For each candidate (deterministic ``(source, folder_path)``
+       order) call :func:`summarize_folder` and UPSERT — fail-soft PER
+       FOLDER: one folder's :class:`LLMError` is logged and counted,
+       its previous row (if any) is kept, and the remaining folders
+       still land (a ``lite`` outage must never fail the sync — the KB
+       is the product, the summaries are auxiliary).
+    4. DELETE rows whose folder no longer has ≥ 2 documents — a
+       pruned/renamed folder's summary goes stale and is dropped.
+       Rows for folders that still qualify persist (regenerated in
+       step 3 — an unchanged folder's summary is still true).
+
+    Only flushes — the CALLER commits (the phase-53
+    ``bump_sources_version`` convention: the sync path owns the
+    transaction, so a failed sync rolls the summaries back with it).
+
+    Returns the small stats dict ``{"generated", "failed", "pruned"}``
+    for the caller's summary-line logging (PLAN §9 ample logging).
+    """
+    stats = {"generated": 0, "failed": 0, "pruned": 0}
+    if skip:
+        return stats
+
+    result = db.execute(
+        select(Document.source, Document.path, Document.title, Document.summary)
+        .order_by(Document.source, Document.path)
+    ).all()
+    rows: list[DocRow] = [
+        (source, path, title, summary)
+        for source, path, title, summary in result
+    ]
+    groups = group_by_folder(rows)
+    candidates = {
+        key: docs for key, docs in groups.items() if len(docs) >= MIN_DOCS_PER_FOLDER
+    }
+
+    for source, folder_path in sorted(candidates):
+        docs = candidates[(source, folder_path)]
+        try:
+            summary = await summarize_folder(source, folder_path, docs, llm)
+        except LLMError as e:
+            stats["failed"] += 1
+            logger.error(
+                "folder summary failed for %s/%s — %s", source, folder_path, e
+            )
+            continue
+        _upsert(db, source, folder_path, summary)
+        stats["generated"] += 1
+
+    existing = db.execute(
+        select(FolderSummary.source, FolderSummary.folder_path)
+    ).all()
+    for source, folder_path in existing:
+        if (source, folder_path) not in candidates:
+            row = db.get(FolderSummary, (source, folder_path))
+            if row is not None:
+                db.delete(row)
+                stats["pruned"] += 1
+
+    db.flush()
+    logger.info(
+        "folder_summaries: generated=%d failed=%d pruned=%d",
+        stats["generated"],
+        stats["failed"],
+        stats["pruned"],
+    )
+    return stats

app/rag/prompts.py

@@ -31,7 +31,15 @@ name*, not a directory or file path, and ``read``/``grep`` take the
 combined ``source/path`` string *including the source name* (a bare
 document path will not resolve) — the same two things the phase-72
 teaching refusals in :mod:`app.rag.agent` re-state after the fact, so
-the model carries the contract before it calls a tool): the **HIGH**
+the model carries the contract before it calls a tool; phase 94,
+task 03: the ``ls`` clause is rewritten to the drill-down tree
+contract — no ``path``: the synced sources with counts + summaries, a
+source name: its top-level folders + files, a ``source/folder`` path:
+one level deeper — a listing shows only that level's subfolders + own
+files (never the whole KB in one call), the folder summaries say
+what's in a folder before drilling, and ``grep`` stays the locator for
+finding one document without listing; the ``read``/``grep`` clauses
+and the discipline rules are byte-identical): the **HIGH**
 prompt only carries a ``<tools>`` section after the ``<documents>``
 body — the grounded turn may extend its context through the three
 server-side tools (round-capped, see :mod:`app.rag.agent`; the cap is
@@ -103,7 +111,17 @@ _KB_INTRO = (
 #: ``path`` is a *source name* (not a directory or file path) and
 #: ``read``/``grep`` take the combined ``source/path`` string *including
 #: the source name* (a bare document path will not resolve) — the same
-#: two things the phase-72 teaching refusals re-state after the fact):
+#: two things the phase-72 teaching refusals re-state after the fact;
+#: phase 94, task 03 — the owner-permitted tool-surface revision,
+#: recorded in the phase 94 overview: the ``ls`` clause is rewritten
+#: to the drill-down tree contract — no ``path``: the synced sources
+#: with counts + summaries, a source name: its top-level folders +
+#: files, a ``source/folder`` path: one level deeper — a listing shows
+#: only that level's subfolders + own files, never the whole KB in one
+#: call, the folder summaries say what's in a folder before drilling,
+#: and ``grep`` stays the locator for finding one document without
+#: listing; the ``read``/``grep`` clauses and the discipline rules are
+#: byte-identical):
 #: a grounded turn may extend its context through the three server-side
 #: tools (round cap: ``BOR_AGENT_MAX_ROUNDS`` — the cap is the bound and
 #: this section does not re-state it, phase 45). Appended after the mode
@@ -139,23 +157,30 @@ _KB_INTRO = (
 TOOLS_SECTION: str = (
     "<tools>\n"
     "You may extend your context with three tools. `ls` lists the "
-    "indexed documents as `source: X | path: Y | title: Z` lines; its "
-    "optional `path` argument is a source name (e.g. 'homelab'), not a "
-    "directory or file path — omit it to list every document. `read` "
-    "pulls in one document by its combined `source/path` string, "
-    "exactly as shown in the `ls` output — including the source name — "
-    "adding its full content to your context. Do not call `read` for a "
-    "document already shown in the <documents> section, even when the "
-    "user asks you to open or read it — its full text is already in "
-    "your prompt; answer directly from it. For `read`, a bare document "
-    "path (without the source name) will not resolve. `grep` locates an "
-    "exact string (case-insensitive) in the indexed documents and "
-    "returns up to 20 matching `source/path:line: text` lines — a "
-    "locator, not a context-adder: read the winner with `read`. A grep "
-    "pattern is a plain substring, NEVER a regex — '.*' and '\\.' are "
-    "literal text there; if such a pattern returns no matches, retry "
-    "with the plain text you expect to see. For a normal search pass "
-    "only `pattern` — its optional `path` argument "
+    "knowledge base as a tree, one level at a time: with no `path` it "
+    "lists every synced source with its document count and a summary "
+    "of its contents; with a source name (e.g. 'homelab') it lists "
+    "that source's top-level folders and files; with a `source/folder` "
+    "path it drills one level deeper. A listing shows only that "
+    "level's subfolders and its own files — never the whole knowledge "
+    "base in one call — and each folder line's summary says what the "
+    "folder contains before you drill into it. File lines are "
+    "`source: X | path: Y | title: Z`; to find one specific document "
+    "without listing, use `grep`. `read` pulls in one document by its "
+    "combined `source/path` string, exactly as shown in the `ls` "
+    "output — including the source name — adding its full content to "
+    "your context. Do not call `read` for a document already shown in "
+    "the <documents> section, even when the user asks you to open or "
+    "read it — its full text is already in your prompt; answer "
+    "directly from it. For `read`, a bare document path (without the "
+    "source name) will not resolve. `grep` locates an exact string "
+    "(case-insensitive) in the indexed documents and returns up to 20 "
+    "matching `source/path:line: text` lines — a locator, not a "
+    "context-adder: read the winner with `read`. A grep pattern is a "
+    "plain substring, NEVER a regex — '.*' and '\\.' are literal text "
+    "there; if such a pattern returns no matches, retry with the plain "
+    "text you expect to see. For a normal search pass only `pattern` — "
+    "its optional `path` argument "
     "limits the search to one document you already know, by the same "
     "combined `source/path` string; never a source name — a bare "
     "document path (without the source name) will not resolve there "