phase: 94_ls_tree_drilldown

All green. Verification complete.

**Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)**

- Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal
- Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths
- Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met
- `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched)
- Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed
- Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol)

**Next pending phase:** `95_read_truncation_cap`

tests/integration/test_agent_tools.py

@@ -1,19 +1,28 @@
 """Integration: the agent DB accessors against real Postgres (phase 37;
-the harness-aligned ``ls``/``read``/``grep`` surface, phase 70).
+the harness-aligned ``ls``/``read``/``grep`` surface, phase 70; the
+drill-down tree ``ls``, phase 94).
 
-``list_catalog`` must order rows by ``(source, path)`` — the same order
-as ``GET /api/docs`` — ``list_source_names`` must resolve the
-registered source names (the scoped ``ls`` join), and ``find_document``
+``_source_document_rows`` must order a source's rows by ``path`` (the
+file lines' catalog order), ``list_source_names`` must resolve the
+registered source names (the registry join), and ``find_document``
 must resolve a hit to the full document row (content included, for the
 never-truncated read) and return ``None`` for unknown pairs. Phase 70:
 the ``ls``/``read``/``grep`` tools are pinned here too — the locked
 parameter shape in ``AGENT_TOOLS``, and scripted ``ToolCallPiece``s
-executed through ``run_agent`` against the real DB: ``ls`` scoped to a
-registered source name (unknown name → refusal), ``read`` on the
-canonical combined ``source/path`` form (first-slash split; a bare
-source name and an unknown identity get the no-document refusal), and
-``grep`` (``all_documents`` for a whole-KB search, ``find_document`` for
-a scoped one).
+executed through ``run_agent`` against the real DB. Phase 94: the
+drill-down ``ls`` against the REAL tables — ``ls()`` lists the
+registered sources (registry order, recursive counts, stored
+source-root summaries from ``folder_summaries``), ``ls(source)`` /
+``ls(source/folder)`` list one folder level (the SQL prefix logic:
+subfolders = slash-boundary prefixes, counts = the recursive subtree,
+file lines in catalog order, capped at 50 + the grep-pointer note),
+and the refusals (unknown source segment → the no-source refusal; an
+unknown folder → NOT-A_FOLDER with the parent's subfolders).
+``read`` runs on the canonical combined ``source/path`` form
+(first-slash split; a bare source name and an unknown identity get the
+no-document refusal), and ``grep`` (``all_documents`` for a whole-KB
+search, ``find_document`` for a scoped one) — both byte-identical
+across the phase-94 change.
 
 Requires:  podman compose up -d db
 """
@@ -30,7 +39,7 @@ from sqlalchemy import delete, text
 from sqlalchemy.orm import Session
 
 from app.config import Settings
-from app.models import Document, GitSource
+from app.models import Document, FolderSummary, GitSource
 from app.rag import agent
 from app.rag.agent import AGENT_TOOLS, AgentHolder, run_agent
 from app.rag.llm import LLMClient, RetryPiece, StreamPiece, ToolCallPiece
@@ -55,11 +64,30 @@ def _doc(db: Session, source: str, path: str, title: str, content: str) -> Docum
 
 @pytest.fixture()
 def kb(db) -> Iterator[None]:
-    """Fresh documents table (chunks first — the FK) for these accessors."""
-    db.execute(text("TRUNCATE chunks, documents"))
+    """Fresh documents + folder_summaries tables (chunks first — the FK)
+    for these accessors (phase 94: the drill-down ``ls`` reads the
+    stored summaries too)."""
+    db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
     db.commit()
     yield
-    db.execute(text("TRUNCATE chunks, documents"))
+    db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
+    db.commit()
+
+
+@pytest.fixture()
+def registry(db) -> Iterator[None]:
+    """A FRESH two-source registry (phase 94): the drill-down ``ls``
+    top level IS the registry, so the table is truncated and re-seeded
+    around the tests in a controlled ``(added_at, id)`` order —
+    ``Deployments`` before ``Homelab`` (the top-level listing order)."""
+    db.execute(text("TRUNCATE git_sources"))
+    db.commit()
+    db.add(GitSource(url="https://github.com/reese/Deployments.git", kind="git"))
+    db.commit()
+    db.add(GitSource(url="https://github.com/reese/Homelab.git", kind="git"))
+    db.commit()
+    yield
+    db.execute(text("TRUNCATE git_sources"))
     db.commit()
 
 
@@ -76,21 +104,24 @@ def src(db) -> Iterator[GitSource]:
     db.commit()
 
 
-def test_list_catalog_orders_by_source_then_path(kb, db) -> None:
+def test_source_document_rows_order_by_path_within_the_source(kb, db) -> None:
+    """Phase 94: the file lines' order — the source's rows in ``path``
+    order (the old ``list_catalog``'s per-source ordering, now the
+    ``ls`` folder-level accessor's contract; a different source's rows
+    never leak in)."""
     _doc(db, "Zeta", "b/second.md", "Zeta B", "ZB")
     _doc(db, "Zeta", "a/first.md", "Zeta A", "ZA")
     _doc(db, "Alpha", "c/third.md", "Alpha C", "AC")
     db.commit()
 
-    assert agent.list_catalog(db) == [
-        ("Alpha", "c/third.md", "Alpha C"),
-        ("Zeta", "a/first.md", "Zeta A"),
-        ("Zeta", "b/second.md", "Zeta B"),
+    assert agent._source_document_rows(db, "Zeta") == [
+        ("a/first.md", "Zeta A"),
+        ("b/second.md", "Zeta B"),
     ]
 
 
-def test_list_catalog_is_empty_without_rows(kb, db) -> None:
-    assert agent.list_catalog(db) == []
+def test_source_document_rows_is_empty_without_rows(kb, db) -> None:
+    assert agent._source_document_rows(db, "Zeta") == []
 
 
 def test_list_source_names_resolves_registry_rows(db) -> None:
@@ -238,30 +269,156 @@ async def _consume(
     return out
 
 
-# ---------- ls (scoped through the real registry) ----------
+# ---------- ls (the drill-down tree, phase 94 — the real registry + DB) ----------
 
 
-def test_ls_scoped_to_registered_source_through_run_agent(kb, src, db) -> None:
-    _doc(db, "Homelab", "a.md", "A", "A-CONTENT")
-    _doc(db, "Other", "b.md", "B", "B-CONTENT")
+def test_ls_top_level_lists_registered_sources_through_run_agent(
+    kb, registry, db
+) -> None:
+    """No path: the TOP level against the real tables — registry order
+    (``(added_at, id)`` — Deployments before Homelab), recursive counts
+    (all of a source's documents), the stored source-root summary
+    (``folder_path = ''``) shown only when stored."""
+    _doc(db, "Deployments", "a/one.md", "A1", "A1-CONTENT")
+    _doc(db, "Homelab", "x.md", "X", "X-CONTENT")
+    _doc(db, "Homelab", "y/z.md", "Z", "Z-CONTENT")
+    db.add(FolderSummary(source="Homelab", folder_path="", summary="Home lab notes."))
     db.commit()
 
-    holder, llm = _run_call(db, "ls", {"path": "Homelab"})
+    holder, llm = _run_call(db, "ls", {})
 
-    # Offered: the first request carries AGENT_TOOLS (the 3-tool list).
     assert llm.requests[0][1] == AGENT_TOOLS
-    # Executed against the real DB: the listing filtered to the source.
     assert llm.requests[1][0][3]["content"] == (
-        "1 documents:\nsource: Homelab | path: a.md | title: A"
+        "2 sources:\n"
+        "\n"
+        "Deployments — 1 documents\n"
+        "Homelab — 2 documents\n"
+        "  Home lab notes."
     )
     assert holder.tool_calls == 1
     assert holder.read_docs == []
 
 
+def test_ls_top_level_empty_registry_through_run_agent(
+    kb, db, monkeypatch: pytest.MonkeyPatch
+) -> None:
+    """No registered sources: the top level is the header line alone
+    (``0 sources:`` — the old ``0 documents:`` behavior preserved in
+    spirit), still counted. The env fallback (``BOR_GIT_SOURCES`` — the
+    operator's ``.env`` may name sources) is emptied for the test, so
+    the registry is genuinely empty."""
+    import app.rag.git_sources as git_sources_mod
+
+    db.execute(text("TRUNCATE git_sources"))
+    db.commit()
+    monkeypatch.setattr(
+        git_sources_mod,
+        "get_settings",
+        lambda: _settings(git_sources=""),
+    )
+    _doc(db, "Orphan", "a.md", "A", "A-CONTENT")  # indexed but unregistered
+    db.commit()
+
+    holder, llm = _run_call(db, "ls", {})
+    assert llm.requests[1][0][3]["content"] == "0 sources:"
+    assert holder.tool_calls == 1
+
+
+def test_ls_source_scope_lists_root_folder_through_run_agent(kb, registry, db) -> None:
+    """A registered source name: the source's ROOT folder — the direct
+    subfolders (path order, recursive counts, stored summaries attached)
+    + the root's own file lines in catalog order — against the real
+    tables; a registered source with no documents lists its header
+    line alone."""
+    _doc(db, "Homelab", "backups/cron.md", "Cron", "CRON")
+    _doc(db, "Homelab", "backups/restic.md", "Restic", "RESTIC")
+    _doc(db, "Homelab", "networking/lan.md", "LAN", "LAN")
+    _doc(db, "Homelab", "readme.md", "Readme", "README")
+    db.add(
+        FolderSummary(
+            source="Homelab", folder_path="backups", summary="Backup notes."
+        )
+    )
+    db.commit()
+
+    holder, llm = _run_call(db, "ls", {"path": "Homelab"})
+
+    assert llm.requests[1][0][3]["content"] == (
+        "Homelab — 1 documents, 2 folders:\n"
+        "\n"
+        "  backups/ — 2 documents: Backup notes.\n"
+        "  networking/ — 1 documents\n"
+        "\n"
+        "source: Homelab | path: readme.md | title: Readme"
+    )
+    assert holder.tool_calls == 1
+    assert holder.read_docs == []
+
+    # A registered source with no documents: the header line alone.
+    holder0, llm0 = _run_call(db, "ls", {"path": "Deployments"})
+    assert llm0.requests[1][0][3]["content"] == "Deployments — 0 documents, 0 folders:"
+    assert holder0.tool_calls == 1
+
+
+def test_ls_nested_folder_scope_drills_one_level_through_run_agent(
+    kb, registry, db
+) -> None:
+    """A ``source/folder`` path: that folder's subfolders + own file
+    lines (identity = ``source/folder``) — the drill-down against the
+    real tables."""
+    _doc(db, "Homelab", "networking/lan/a.md", "A", "A")
+    _doc(db, "Homelab", "networking/lan/b.md", "B", "B")
+    _doc(db, "Homelab", "networking/vpn/c.md", "C", "C")
+    db.commit()
+
+    holder, llm = _run_call(db, "ls", {"path": "Homelab/networking"})
+
+    assert llm.requests[1][0][3]["content"] == (
+        "Homelab/networking — 0 documents, 2 folders:\n"
+        "\n"
+        "  networking/lan/ — 2 documents\n"
+        "  networking/vpn/ — 1 documents"
+    )
+    assert holder.tool_calls == 1
+    assert holder.read_docs == []
+
+    # One level deeper.
+    holder2, llm2 = _run_call(db, "ls", {"path": "Homelab/networking/lan"})
+    assert llm2.requests[1][0][3]["content"] == (
+        "Homelab/networking/lan — 2 documents, 0 folders:\n"
+        "\n"
+        "source: Homelab | path: networking/lan/a.md | title: A\n"
+        "source: Homelab | path: networking/lan/b.md | title: B"
+    )
+    assert holder2.tool_calls == 1
+
+
+def test_ls_folder_file_cap_through_run_agent(kb, registry, db) -> None:
+    """The cap end-to-end: 51 direct files in one folder cost 50 file
+    lines + the deterministic grep-pointer note, never 51."""
+    for i in range(51):
+        _doc(db, "Homelab", f"big/f{i:03d}.md", f"T{i}", "BODY")
+    db.commit()
+
+    holder, llm = _run_call(db, "ls", {"path": "Homelab/big"})
+
+    content = llm.requests[1][0][3]["content"]
+    lines = content.splitlines()
+    assert lines[0] == "Homelab/big — 51 documents, 0 folders:"
+    assert lines[2] == "source: Homelab | path: big/f000.md | title: T0"
+    assert lines[51] == "source: Homelab | path: big/f049.md | title: T49"
+    assert lines[52] == (
+        "…and 1 more documents in this folder — use grep (pattern) to "
+        "find a specific one."
+    )
+    assert len(lines) == 53
+    assert holder.tool_calls == 1
+
+
 def test_ls_scoped_unknown_source_refused_through_run_agent(kb, src, db) -> None:
-    """Phase 72: the no-source refusal now carries the teaching
-    parenthetical — the prefix byte-identical to the pre-phase-72 line;
-    still not counted."""
+    """A ``path`` without ``/`` matching no source name is a refusal —
+    the extended line with the teaching parenthetical (phase 72, the
+    prefix byte-identical to the pre-phase-72 line); still not counted."""
     _doc(db, "Homelab", "a.md", "A", "A-CONTENT")
     db.commit()
 
@@ -274,11 +431,11 @@ def test_ls_scoped_unknown_source_refused_through_run_agent(kb, src, db) -> None
     assert holder.tool_calls == 0 and holder.read_docs == []
 
 
-def test_ls_path_like_scope_teaching_refusal_through_run_agent(kb, src, db) -> None:
-    """Phase 72: a ``/``-containing ``path`` is a document path, not a
-    source name — the ``LS_PATH_NOT_A_SOURCE`` teaching line (no
-    registry lookup needed), not counted, the tools stay offered on the
-    next request."""
+def test_ls_path_like_scope_unknown_source_gets_no_source_refusal(kb, src, db) -> None:
+    """Phase 94: a ``/`` now names a folder — the phase-72 document-path
+    teaching is DELETED; a ``source/…`` argument whose FIRST segment
+    names no registered source gets the no-source refusal (the segment
+    echoed), not counted, the tools stay offered."""
     _doc(db, "Homelab", "a.md", "A", "A-CONTENT")
     db.commit()
 
@@ -286,7 +443,30 @@ def test_ls_path_like_scope_teaching_refusal_through_run_agent(kb, src, db) -> N
 
     assert (
         llm.requests[1][0][3]["content"]
-        == agent.LS_PATH_NOT_A_SOURCE.format(path="app/rag/importer.py")
+        == agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="app")
+    )
+    assert holder.tool_calls == 0 and holder.read_docs == []
+    assert llm.requests[1][1] == AGENT_TOOLS  # rejected → tools stay offered
+
+
+def test_ls_unknown_folder_gets_not_a_folder_with_parents_subfolders(
+    kb, src, db,
+) -> None:
+    """Phase 94: a folder segment matching no indexed prefix gets the
+    NOT-A_FOLDER teaching — the argument echoed, the source named, its
+    DIRECT subfolders listed (the self-correction list), not counted,
+    the tools stay offered."""
+    _doc(db, "Homelab", "backups/cron.md", "Cron", "CRON")
+    _doc(db, "Homelab", "containers/caddy.md", "Caddy", "CADDY")
+    _doc(db, "Homelab", "networking/lan.md", "LAN", "LAN")
+    _doc(db, "Homelab", "readme.md", "Readme", "README")
+    db.commit()
+
+    holder, llm = _run_call(db, "ls", {"path": "Homelab/netwoking"})
+
+    assert llm.requests[1][0][3]["content"] == (
+        "'Homelab/netwoking' is not a folder — Homelab has: "
+        "backups/ containers/ networking/"
     )
     assert holder.tool_calls == 0 and holder.read_docs == []
     assert llm.requests[1][1] == AGENT_TOOLS  # rejected → tools stay offered