phase: 94_ls_tree_drilldown
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Build and Push Containers / build-and-push-db (push) Successful in 25s

All green. Verification complete.

**Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)**

- Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal
- Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths
- Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met
- `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched)
- Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed
- Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol)

**Next pending phase:** `95_read_truncation_cap`
This commit is contained in:
2026-09-11 00:59:35 -04:00
parent 9188be259b
commit d4943b4822
61 changed files with 6289 additions and 666 deletions
+480 -128
View File
@@ -2,15 +2,20 @@
harness-aligned ``ls``/``read``/``grep`` surface, phase 70).
A scripted fake LLM (canned stream sequences) + monkeypatched
``list_catalog`` / ``list_source_names`` / ``find_document`` /
``ls_top`` / ``ls_folder`` / ``list_source_names`` / ``find_document`` /
``all_documents`` — no database, no network. Covers the loop mechanics:
the ls → read (combined ``source/path``) → answer happy path (event
order, holder state, the tools staying offered on every request —
phase 45 removed the per-tool budgets, the assistant/tool message
history), the ``ls`` scoping (no-arg full catalog in the phase-63
labeled-field format, a one-source scope, a known source with 0
documents → ``0 documents:`` counted, an unknown-source refusal that
counts nothing), ``read`` on the canonical combined form (split at the
history), the phase-94 drill-down ``ls`` (no-arg top level = the
registered sources with counts + stored summaries in the pinned
``{N} sources:`` template, a source scope = its root folder —
subfolders + capped file lines in the pinned folder template —, a
``source/folder`` scope = one level deeper, a registered source with 0
documents → the ``… — 0 documents, 0 folders:`` header counted, an
unknown-source refusal that counts nothing, the NOT-A-FOLDER teaching
with the parent's subfolders, the 50-file cap + grep-pointer note),
``read`` on the canonical combined form (split at the
FIRST slash, full content, the bare-source-name refusal, the
already-in-context dedupe, missing-args refusals), the phase-68 ``grep``
contract under its new name (the locked A5 pins: fixed substring,
@@ -149,28 +154,35 @@ def test_agent_tools_names_and_parameters() -> None:
# Task 05 (live gate iteration 2): the one-call-at-a-time discipline
# clause (the harness prior batches calls; the loop executes one
# per round — the extras count as unexecuted in the gate).
# Phase 94 (task 03): the description is the drill-down tree
# contract (pinned copy — the tool-surface revision, owner
# permission 2026-09-10, ``TODO.md`` L4): one level per call,
# sources at the top, folders + files below, the file-line format
# and the combined-identity handoff to read/grep intact.
assert ls["description"] == (
"List the indexed documents as `source: X | path: Y | "
"title: Z` lines. Call one tool at a time — wait for "
"this result before your next call."
"List the knowledge base as a tree, one level at a time. "
"With no path: the synced sources — each with its document "
"count and a summary of its contents. With a source name (no "
"'/'): that source's top-level folders and files. With a "
"`source/folder` path: that folder's subfolders and files. "
"Folder lines carry a summary of what the folder contains. "
"File lines are `source: X | path: Y | title: Z` — use the "
"combined `source/path` with `read` and `grep`. Call one tool "
"at a time — wait for this result before your next call."
)
ls_params = ls["parameters"]
assert ls_params["type"] == "object"
assert ls_params["required"] == [] # path is optional
assert set(ls_params["properties"]) == {"path"}
assert ls_params["properties"]["path"]["type"] == "string"
# Phase 72: the description states the contract up front — the
# 'path' argument is a source name, not a file or directory path.
# Task 05 (live gate iteration 5): the cross-tool contrast clause
# (ls is the ONLY tool whose path is a source name — the model
# kept transferring that scope to grep's document identity).
# Phase 94 (task 03): the 'path' argument teaches the drill-down
# semantics — a source name lists that source's top level, a
# `source/folder` path drills one level deeper, omitted lists
# every source (pinned copy).
assert ls_params["properties"]["path"]["description"] == (
"Source name to list one source's documents (e.g. 'homelab') — "
"a source name, not a file or directory path; omit to list "
"every document. This is the only tool "
"whose `path` is a source name — for "
"`read` and `grep` it must be a document's "
"combined `source/path`."
"Optional — a source name (e.g. 'homelab') to list its top "
"level, or a `source/folder` path to drill down (e.g. "
"'homelab/active'). Omit it to list every source."
)
read = by_name["read"]["function"]
# Tool-calling fast loop (2026-09-04, controlled fixture gate):
@@ -285,16 +297,11 @@ def test_refusal_constants_are_harness_aligned() -> None:
assert agent.UNKNOWN_TOOL == "Unknown tool."
assert agent.MISSING_READ_ARGS == "read requires a string argument 'path'."
assert agent.MISSING_SEARCH_ARGS == "grep requires a string argument 'pattern'."
# Phase 72: the ls teaching-refusal templates, pinned byte-for-byte
# (task 01 — the read/grep suggestion templates below, task 02).
assert agent.LS_PATH_NOT_A_SOURCE == (
"'{path}' looks like a document path, not a source name. The "
"'path' argument of ls filters by source name (e.g. 'homelab') — "
"omit it to list every document, or read a document by its "
"combined 'source/path' string."
)
# The pre-phase-72 no-source line is the byte-identical prefix of
# the extended line — only the teaching parenthetical was appended.
# Phase 94 (task 03): the phase-72 document-path teaching refusal is
# DELETED (a ``/`` now names a folder — the drill-down contract);
# the no-source refusal stays byte-identical (the task's "existing
# refusal, teaching parenthetical intact" pin).
assert not hasattr(agent, "LS_PATH_NOT_A_SOURCE")
assert agent.NO_SOURCE_NOT_A_DIRECTORY.startswith(
"No source named '{scope}' — check the ls output."
)
@@ -303,6 +310,11 @@ def test_refusal_constants_are_harness_aligned() -> None:
"argument is a source name, not a directory — omit it to list "
"every document.)"
)
# Phase 94 (task 03): the NOT-A-FOLDER drill-down teaching template,
# pinned byte-for-byte (argument echoed, parent's subfolders
# listed), and the pinned file-line cap constant.
assert agent.NOT_A_FOLDER == "'{arg}' is not a folder — {parent} has: {subfolders}"
assert agent.LS_MAX_FILE_LINES == 50
# Phase 72 (task 02): the read/grep "did you mean …?" suggestion
# templates, pinned byte-for-byte, and the suggestion cap.
assert agent.NO_DOCUMENT_DID_YOU_MEAN == (
@@ -348,11 +360,12 @@ def test_list_source_names_empty_registry(monkeypatch: pytest.MonkeyPatch) -> No
def test_ls_then_read_then_answer(monkeypatch: pytest.MonkeyPatch) -> None:
catalog = [
("Deployments", "backups.md", "Backup Strategy"),
("Homelab", "aws-route53.md", "AWS Route53 Records"),
]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
# Phase 94 (task 03): the no-arg ``ls`` is the drill-down TOP level
# (the registered sources, registry order) — monkeypatched the way
# the phase-70 full-catalog listing used to be.
monkeypatch.setattr(
agent, "ls_top", lambda db: [("Deployments", 1, None), ("Homelab", 1, None)]
)
target = _doc("Homelab", "aws-route53.md", "AWS Route53 Records", "R53-CONTENT")
calls: list[tuple[str, str]] = []
@@ -423,11 +436,7 @@ def test_ls_then_read_then_answer(monkeypatch: pytest.MonkeyPatch) -> None:
assert msgs[3] == {
"role": "tool",
"tool_call_id": "call_1",
"content": (
"2 documents:\n"
"source: Deployments | path: backups.md | title: Backup Strategy\n"
"source: Homelab | path: aws-route53.md | title: AWS Route53 Records"
),
"content": "2 sources:\n\nDeployments — 1 documents\nHomelab — 1 documents",
}
# The second follow-up request carries the read call + the FULL text.
msgs = llm.requests[2][0]
@@ -448,7 +457,7 @@ def test_content_and_tool_call_in_one_stream_keeps_both(
) -> None:
"""Rare stream with content AND a tool call: the content stays (it was
already emitted) and the tool still runs."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
monkeypatch.setattr(agent, "ls_top", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[
@@ -460,7 +469,7 @@ def test_content_and_tool_call_in_one_stream_keeps_both(
pieces = asyncio.run(_run(llm, holder, _settings()))
assert [type(p) for p in pieces] == [StreamPiece, ToolCallPiece, StreamPiece]
assert holder.tool_calls == 1 # the tool ran despite the content
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
assert llm.requests[1][0][3]["content"] == "0 sources:"
# ---------- phase 74: client history between system and user ----------
@@ -519,7 +528,7 @@ def test_run_agent_history_survives_a_tool_round(
"""The tool rounds append assistant/tool messages to the SAME
``messages`` list — the prior history stays in place between the
system prompt and the current question on the SECOND request too."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
monkeypatch.setattr(agent, "ls_top", lambda db: [])
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
[StreamPiece("content", "the answer")],
@@ -534,22 +543,25 @@ def test_run_agent_history_survives_a_tool_round(
]
# ---------- ls: full catalog + scoping ----------
# ---------- ls: the drill-down tree (phase 94, task 03) ----------
def test_ls_full_catalog_format(monkeypatch: pytest.MonkeyPatch) -> None:
"""No argument: the full catalog in the phase-63 labeled-field format
(``source: X | path: Y | title: Z``) — counted; no registry lookup."""
catalog = [
("Deployments", "backups.md", "Backup Strategy"),
("Homelab", "aws-route53.md", "AWS Route53 Records"),
]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
def _boom_sources(*_a: Any, **_k: Any) -> None:
raise AssertionError("no registry lookup for an unscoped ls")
monkeypatch.setattr(agent, "list_source_names", _boom_sources)
def test_ls_top_level_lists_sources_with_summaries(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""No argument: the TOP level — the registered sources, registry
order, each ``{source} — {n} documents`` + the indented summary line
only when stored — the pinned template, counted; the registry IS
consulted (unlike the phase-70 full catalog, the top level is the
registry itself)."""
monkeypatch.setattr(
agent,
"ls_top",
lambda db: [
("Deployments", 0, None),
("Homelab", 1, "The homelab notes."),
],
)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
@@ -557,22 +569,27 @@ def test_ls_full_catalog_format(monkeypatch: pytest.MonkeyPatch) -> None:
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == (
"2 documents:\n"
"source: Deployments | path: backups.md | title: Backup Strategy\n"
"source: Homelab | path: aws-route53.md | title: AWS Route53 Records"
"2 sources:\n"
"\n"
"Deployments — 0 documents\n"
"Homelab — 1 documents\n"
" The homelab notes."
)
assert holder.tool_calls == 1
def test_ls_empty_catalog_says_zero_documents(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
def test_ls_empty_registry_says_zero_sources(monkeypatch: pytest.MonkeyPatch) -> None:
"""No registered sources: the top level is the header line alone —
``0 sources:`` (the old ``0 documents:`` behavior preserved in
spirit), still a counted result."""
monkeypatch.setattr(agent, "ls_top", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
assert llm.requests[1][0][3]["content"] == "0 sources:"
assert holder.tool_calls == 1
@@ -583,13 +600,12 @@ def test_ls_empty_catalog_says_zero_documents(monkeypatch: pytest.MonkeyPatch) -
({"path": 7}, "non-string path"),
],
)
def test_ls_blank_path_lists_full_catalog(
def test_ls_blank_path_lists_top_level(
monkeypatch: pytest.MonkeyPatch, arguments: dict[str, Any], label: str
) -> None:
"""A blank (or non-string) ``path`` is treated as omitted — the full
catalog, counted (no refusal for an empty scope)."""
catalog = [("S", "a.md", "A")]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
"""A blank (or non-string) ``path`` is treated as omitted — the top
level (the sources), counted (no refusal for an empty scope)."""
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -597,21 +613,25 @@ def test_ls_blank_path_lists_full_catalog(
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert (
llm.requests[1][0][3]["content"] == "1 documents:\nsource: S | path: a.md | title: A"
)
assert llm.requests[1][0][3]["content"] == "1 sources:\n\nS — 1 documents"
assert holder.tool_calls == 1
def test_ls_scoped_to_known_source(monkeypatch: pytest.MonkeyPatch) -> None:
"""A known source name: the same listing filtered to that source —
def test_ls_source_scope_lists_root_folder(monkeypatch: pytest.MonkeyPatch) -> None:
"""A registered source name (no ``/``): the source's ROOT folder —
subfolders (2-space-indented, path order, ``: {summary}`` only when
stored) + the root's own file lines in EXACTLY the
``source: X | path: Y | title: Z`` format — the pinned template,
counted."""
catalog = [
("Deployments", "backups.md", "Backup Strategy"),
("Homelab", "a.md", "A"),
("Homelab", "b.md", "B"),
]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
monkeypatch.setattr(
agent,
"ls_folder",
lambda db, source, folder: (
[("backups", 2, "Backup notes."), ("networking", 1, None)],
[("Homelab", "readme.md", "Readme")],
1,
),
)
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Deployments", "Homelab"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -620,9 +640,50 @@ def test_ls_scoped_to_known_source(monkeypatch: pytest.MonkeyPatch) -> None:
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == (
"2 documents:\n"
"source: Homelab | path: a.md | title: A\n"
"source: Homelab | path: b.md | title: B"
"Homelab — 1 documents, 2 folders:\n"
"\n"
" backups/ — 2 documents: Backup notes.\n"
" networking/ — 1 documents\n"
"\n"
"source: Homelab | path: readme.md | title: Readme"
)
assert holder.tool_calls == 1
def test_ls_nested_folder_scope_lists_one_level_deeper(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A ``source/folder`` path: that folder's subfolders + own file
lines, identity = ``source/folder`` (the same template as the
root), counted; the fetchers are the source-scoped ones."""
def _rows(db: Any, source: str) -> list[tuple[str, str]]:
assert (source, db) == ("Homelab", None)
return [
("networking/lan.md", "LAN"),
("networking/vpn.md", "VPN"),
]
monkeypatch.setattr(agent, "_source_document_rows", _rows)
monkeypatch.setattr(
agent, "_source_folder_summaries", lambda db, source: {"networking": "Network notes."}
)
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab"])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1", name="ls", arguments={"path": "Homelab/networking"}
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == (
"Homelab/networking — 2 documents, 0 folders:\n"
"\n"
"source: Homelab | path: networking/lan.md | title: LAN\n"
"source: Homelab | path: networking/vpn.md | title: VPN"
)
assert holder.tool_calls == 1
@@ -631,9 +692,13 @@ def test_ls_scoped_known_source_with_zero_docs_counts(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A registered source with no indexed documents is KNOWN (the
registry is the source of truth, not the catalog): it lists as
``0 documents:`` — a valid, counted result, not a refusal."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("Other", "a.md", "A")])
registry is the source of truth, not the catalog): it lists its
header line alone (``… — 0 documents, 0 folders:`` — the old
``0 documents:`` behavior preserved in spirit) — a valid, counted
result, not a refusal."""
monkeypatch.setattr(
agent, "ls_folder", lambda db, source, folder: ([], [], 0)
)
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab", "Other"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -641,7 +706,7 @@ def test_ls_scoped_known_source_with_zero_docs_counts(
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
assert llm.requests[1][0][3]["content"] == "Homelab — 0 documents, 0 folders:"
assert holder.tool_calls == 1 # an executed ls, not a refusal
assert llm.requests[1][1] == AGENT_TOOLS
@@ -650,7 +715,6 @@ def test_ls_scoped_unknown_source_refused(monkeypatch: pytest.MonkeyPatch) -> No
"""A ``path`` without ``/`` matching no source name is a refusal —
the extended line with the teaching parenthetical (phase 72), not
counted, the round cap bounds its repetition."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -666,20 +730,14 @@ def test_ls_scoped_unknown_source_refused(monkeypatch: pytest.MonkeyPatch) -> No
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_ls_path_like_scope_gets_document_path_teaching_refusal(
def test_ls_path_like_scope_unknown_source_gets_no_source_refusal(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 72: a stripped scope containing ``/`` looks like a document
path (the incident's ``ls(path='app/rag/importer.py')``) — a source
name is a directory basename and can never contain one, so this gets
the ``LS_PATH_NOT_A_SOURCE`` teaching line with the argument echoed;
no registry lookup, counts in nothing, tools stay offered."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
def _boom_sources(*_a: Any, **_k: Any) -> None:
raise AssertionError("no registry lookup for a path-like scope")
monkeypatch.setattr(agent, "list_source_names", _boom_sources)
"""Phase 94: a ``/`` now names a folder, so the phase-72
document-path teaching is DELETED — a ``source/…`` argument whose
FIRST segment names no registered source gets the no-source refusal
(the segment echoed), counted in nothing, tools stay offered."""
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
[
@@ -695,7 +753,7 @@ def test_ls_path_like_scope_gets_document_path_teaching_refusal(
assert holder.tool_calls == 0 # a refusal counts in nothing
assert (
llm.requests[1][0][3]["content"]
== agent.LS_PATH_NOT_A_SOURCE.format(path="app/rag/importer.py")
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="app")
)
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
@@ -706,8 +764,7 @@ def test_ls_dot_scope_gets_not_a_directory_teaching_refusal(
"""Phase 72: ``ls(path='.')`` (the incident's second round — no
``/``, no matching source) gets the extended no-source refusal with
the teaching parenthetical, ``'.'`` echoed — not counted, tools stay
offered."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
offered (unchanged by phase 94)."""
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -723,6 +780,312 @@ def test_ls_dot_scope_gets_not_a_directory_teaching_refusal(
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_ls_unknown_top_level_folder_gets_not_a_folder_teaching(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 94: a ``source/…`` argument whose TOP-LEVEL folder segment
matches no indexed prefix gets the NOT-A-FOLDER teaching — the
argument echoed, the source named, its direct subfolders listed so
the model self-corrects in the next round; not counted, tools stay
offered."""
monkeypatch.setattr(
agent,
"_source_document_rows",
lambda db, source: [
("backups/cron.md", "Cron"),
("containers/caddy.md", "Caddy"),
("networking/lan.md", "LAN"),
],
)
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab"])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1", name="ls", arguments={"path": "Homelab/netwoking"}
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 0 # a refusal counts in nothing
assert llm.requests[1][0][3]["content"] == (
"'Homelab/netwoking' is not a folder — Homelab has: "
"backups/ containers/ networking/"
)
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_ls_unknown_nested_folder_gets_not_a_folder_with_nested_parent(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 94: a nested miss names the DEEPEST existing ancestor —
``source/folder`` — and lists ITS direct subfolders (bounded: the
parent's own listing, no new flood path)."""
monkeypatch.setattr(
agent,
"_source_document_rows",
lambda db, source: [
("networking/lan/a.md", "A"),
("networking/vpn/b.md", "B"),
("readme.md", "Readme"),
],
)
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab"])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1", name="ls", arguments={"path": "Homelab/networking/lan/x"}
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 0
assert llm.requests[1][0][3]["content"] == (
"'Homelab/networking/lan/x' is not a folder — "
"Homelab/networking/lan has: none"
)
def test_ls_file_path_scope_gets_not_a_folder(monkeypatch: pytest.MonkeyPatch) -> None:
"""Phase 94: a document's OWN path is never a folder (nothing starts
with ``path + '/'``) — ``ls`` of a file path refuses with the
NOT-A-FOLDER teaching (the parent's subfolders listed)."""
monkeypatch.setattr(
agent,
"_source_document_rows",
lambda db, source: [("notes.md", "Notes"), ("a/b.md", "B")],
)
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(id="call_1", name="ls", arguments={"path": "S/notes.md"})
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 0
assert llm.requests[1][0][3]["content"] == (
"'S/notes.md' is not a folder — S has: a/"
)
# ---------- ls_top / ls_folder: the drill-down accessors (pure + composed) ----------
def test_ls_top_registry_order_zero_docs_and_summaries(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""``ls_top``: registry order (not catalog order), a 0-document
source still lists, the summary is the stored ``(source, "")`` row
or ``None`` when absent; an empty registry → ``[]``."""
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Zeta", "Alpha"])
monkeypatch.setattr(
agent, "_source_document_counts", lambda db: [("Zeta", 3), ("Beta", 1)]
)
monkeypatch.setattr(
agent, "_source_root_summaries", lambda db: [("Zeta", "Zeta stuff.")]
)
assert agent.ls_top(cast("Session", object())) == [
("Zeta", 3, "Zeta stuff."),
("Alpha", 0, None), # 0 docs (no count row) + no stored summary
]
def test_ls_top_empty_registry_is_empty(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setattr(agent, "list_source_names", lambda db: [])
def _boom(*_a: Any, **_k: Any) -> None:
raise AssertionError("no fetches for an empty registry")
monkeypatch.setattr(agent, "_source_document_counts", _boom)
monkeypatch.setattr(agent, "_source_root_summaries", _boom)
assert agent.ls_top(cast("Session", object())) == []
def test_ls_folder_composes_the_fetchers(monkeypatch: pytest.MonkeyPatch) -> None:
"""``ls_folder`` = the source's document rows + stored summaries
through the pure :func:`group_folder_listing` (the fetchers are the
monkeypatch surface)."""
seen: list[tuple[str, str, str]] = []
def _rows(db: Any, source: str) -> list[tuple[str, str]]:
seen.append(("rows", source, ""))
return [("a/b.md", "B"), ("a.md", "A")]
def _summaries(db: Any, source: str) -> dict[str, str]:
seen.append(("summaries", source, ""))
return {"a": "A stuff."}
monkeypatch.setattr(agent, "_source_document_rows", _rows)
monkeypatch.setattr(agent, "_source_folder_summaries", _summaries)
assert agent.ls_folder(cast("Session", object()), "S", "") == (
[("a", 1, "A stuff.")],
[("S", "a.md", "A")],
1,
)
assert seen == [("rows", "S", ""), ("summaries", "S", "")]
def test_group_folder_listing_subfolder_recursion_and_counts() -> None:
"""The recursive count per subfolder — every path equal to the
folder or starting with ``folder + '/'`` (a doc under ``a/b/``
counts for BOTH ``a`` and ``a/b``), path order, the stored summary
attached or ``None``."""
rows = [
("a/b/c.md", "C"),
("a/b/d.md", "D"),
("a/e.md", "E"),
("f.md", "F"),
]
sub, files, total = agent.group_folder_listing(
"S", "", rows, {"a": "A subtree.", "a/b": "B subtree."}
)
# ROOT level: the direct subfolders of "" are the TOP-LEVEL folders
# only (a/b is nested under a, not direct) — a's count is its whole
# recursive subtree (a/e.md + a/b/c.md + a/b/d.md), the stored
# summary attached.
assert sub == [("a", 3, "A subtree.")]
assert files == [("S", "f.md", "F")]
assert total == 1
# One level down: a/b is a's direct subfolder with its own count.
sub2, _files2, _total2 = agent.group_folder_listing("S", "a", rows, {"a/b": "B subtree."})
assert sub2 == [("a/b", 2, "B subtree.")]
def test_group_folder_listing_nested_level_counts_and_membership() -> None:
"""One level down: ``a``'s direct subfolder is ``a/b`` (count 2),
its own direct file is ``a/e.md`` (``a/b/c.md`` is NOT a direct
file of ``a``) — membership is the folder_of rule, order is path
order."""
rows = [
("a/b/c.md", "C"),
("a/b/d.md", "D"),
("a/e.md", "E"),
]
sub, files, total = agent.group_folder_listing("S", "a", rows, {})
assert sub == [("a/b", 2, None)]
assert files == [("S", "a/e.md", "E")]
assert total == 1
def test_group_folder_listing_file_path_is_not_a_folder() -> None:
"""A document whose path is a prefix of NO other path is a file,
never a folder: ``ls`` of it must not list a subfolder (and the
``path == folder`` count arm only fires for TRUE folders — a doc
sharing a real folder's name counts for that folder, the existence
rule intact)."""
rows = [
("a.md", "A"), # a file at the root, and a folder name? NO —
("b/x.md", "X"), # nothing starts with "a.md/"
]
sub, files, total = agent.group_folder_listing("S", "", rows, {})
assert sub == [("b", 1, None)] # "a.md" is NOT a subfolder
assert files == [("S", "a.md", "A")] # b/x.md is NOT a direct root file
assert total == 1
# The path == folder arm: a doc named "a" under a real folder "a/".
rows2 = [("a", "FileA"), ("a/c.md", "C")]
sub2, files2, total2 = agent.group_folder_listing("S", "", rows2, {})
assert sub2 == [("a", 2, None)] # the file "a" counts for folder "a"
assert files2 == [("S", "a", "FileA")] # …and is a direct ROOT file
assert total2 == 1
def test_group_folder_listing_caps_files_at_fifty_keeps_the_total() -> None:
"""The cap: 51 direct files → 50 file lines + the PRE-cap total (51)
for the renderer's note; 50 files → 50 lines, no note material.
A 500-file folder costs 50 lines, never 500."""
rows51 = [(f"big/f{i:03d}.md", f"T{i}") for i in range(51)]
sub, files, total = agent.group_folder_listing("S", "big", rows51, {})
assert sub == []
assert total == 51
assert len(files) == 50
assert files[0] == ("S", "big/f000.md", "T0")
assert files[-1] == ("S", "big/f049.md", "T49")
rows50 = [(f"big/f{i:03d}.md", f"T{i}") for i in range(50)]
_sub, files50, total50 = agent.group_folder_listing("S", "big", rows50, {})
assert total50 == 50 and len(files50) == 50
# ---------- the pinned drill-down templates (byte-for-byte) ----------
def test_render_ls_top_template() -> None:
assert (
agent.render_ls_top(
[("Deployments", 3, None), ("Homelab", 5, "Home lab notes.")]
)
== "2 sources:\n\n"
"Deployments — 3 documents\n"
"Homelab — 5 documents\n"
" Home lab notes."
)
assert agent.render_ls_top([]) == "0 sources:"
def test_render_folder_listing_root_template() -> None:
assert (
agent.render_folder_listing(
"Homelab",
[("backups", 2, "Backup notes."), ("networking", 1, None)],
[("Homelab", "readme.md", "Readme")],
1,
)
== "Homelab — 1 documents, 2 folders:\n"
"\n"
" backups/ — 2 documents: Backup notes.\n"
" networking/ — 1 documents\n"
"\n"
"source: Homelab | path: readme.md | title: Readme"
)
def test_render_folder_listing_empty_level_is_header_alone() -> None:
"""A registered source with no documents: the header line alone —
the old ``0 documents:`` behavior preserved in spirit."""
assert agent.render_folder_listing("Homelab", [], [], 0) == (
"Homelab — 0 documents, 0 folders:"
)
def test_render_folder_listing_subfolders_only_no_blank_trailer() -> None:
"""Subfolders but no own files: header + blank + subfolder lines —
no trailing blank line, no file section."""
assert (
agent.render_folder_listing("S", [("a", 1, None)], [], 0)
== "S — 0 documents, 1 folders:\n\n a/ — 1 documents"
)
def test_render_folder_listing_cap_note_only_past_fifty() -> None:
"""The note appears ONLY when the folder's own files outnumber the
cap: 51 → 50 lines + the deterministic grep-pointer note (the
``…and 1 more…`` shape — unpluralized, the house pin); 50 → no
note."""
files51 = [("S", f"f{i:03d}.md", f"T{i}") for i in range(51)]
capped = files51[:50]
rendered = agent.render_folder_listing("S/big", [], capped, 51)
lines = rendered.splitlines()
assert lines[0] == "S/big — 51 documents, 0 folders:"
assert len(lines) == 1 + 1 + 50 + 1 # header, blank, 50 lines, note
assert lines[-1] == (
"…and 1 more documents in this folder — use grep (pattern) to "
"find a specific one."
)
files50 = [("S", f"f{i:03d}.md", f"T{i}") for i in range(50)]
rendered50 = agent.render_folder_listing("S/big", [], files50, 50)
assert rendered50.splitlines()[-1] == "source: S | path: f049.md | title: T49"
assert "more documents" not in rendered50
# ---------- read: the canonical combined source/path form ----------
@@ -771,7 +1134,6 @@ def test_read_bare_source_name_refused_without_db(monkeypatch: pytest.MonkeyPatc
no-document refusal (the argument echoed as passed), no DB lookup
(NOT even the phase-72 candidate lookup — ``all_documents`` must
not run either), nothing counted."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("Homelab", "a.md", "A")])
def _boom(*_a: Any, **_k: Any) -> None:
raise AssertionError(
@@ -1158,7 +1520,6 @@ def test_reading_an_already_read_doc_is_deduped(monkeypatch: pytest.MonkeyPatch)
def test_unknown_tool_name_refused(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="delete_universe", arguments={"x": 1})],
@@ -1702,13 +2063,11 @@ def test_grep_counts_but_never_adds_context(monkeypatch: pytest.MonkeyPatch) ->
def test_relist_executes_and_counts(monkeypatch: pytest.MonkeyPatch) -> None:
"""Re-lists execute — a second ``ls`` in one turn returns the catalog
again and counts in ``tool_calls`` (no budget to exhaust)."""
catalog = [
("Deployments", "backups.md", "Backup Strategy"),
("Homelab", "aws-route53.md", "AWS Route53 Records"),
]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
"""Re-lists execute — a second ``ls`` in one turn returns the top
level again and counts in ``tool_calls`` (no budget to exhaust)."""
monkeypatch.setattr(
agent, "ls_top", lambda db: [("Deployments", 1, None), ("Homelab", 1, None)]
)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
@@ -1717,12 +2076,8 @@ def test_relist_executes_and_counts(monkeypatch: pytest.MonkeyPatch) -> None:
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 2 # both re-lists executed and counted
listing = (
"2 documents:\n"
"source: Deployments | path: backups.md | title: Backup Strategy\n"
"source: Homelab | path: aws-route53.md | title: AWS Route53 Records"
)
# The answer request carries the catalog a second time as a tool result.
listing = "2 sources:\n\nDeployments — 1 documents\nHomelab — 1 documents"
# The answer request carries the listing a second time as a tool result.
assert llm.requests[2][0][3]["content"] == listing # first listing
assert llm.requests[2][0][5]["content"] == listing # the re-list
assert llm.requests[2][1] == AGENT_TOOLS # still offered (no budgets)
@@ -1758,8 +2113,8 @@ def test_always_ls_bounded_by_round_cap(monkeypatch: pytest.MonkeyPatch) -> None
"""A model that keeps calling ``ls`` gets exactly
``agent_max_rounds`` tool rounds, then one forced ``tools=None``
request streams the answer — the cap is the only forced exit."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
listing = "1 documents:\nsource: S | path: a.md | title: A"
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
listing = "1 sources:\n\nS — 1 documents"
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
@@ -1923,7 +2278,7 @@ def test_round_retried_before_first_piece(
same messages: the stream carries a RetryPiece BEFORE the tool call,
the tool executes, the final answer streams, and the per-call log line
is still emitted exactly once (retries are invisible to the loop)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = FailingLLM(
[
@@ -1961,7 +2316,6 @@ def test_round_failure_after_first_piece_is_terminal(monkeypatch: pytest.MonkeyP
the LLMError propagates out of ``run_agent``, no RetryPiece, no
sleep, no second request, and the holder is untouched (the tool
never ran)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = FailingLLM(
[([StreamPiece("content", "partial ")], LLMError("mid-stream drop"))]
@@ -1994,7 +2348,7 @@ def test_forced_final_no_tools_call_is_retried(monkeypatch: pytest.MonkeyPatch)
"""The forced final request (round cap reached) goes through the same
retry rule: a failure before its first piece yields a RetryPiece and
restarts with ``tools=None``; the answer from the retry streams."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = FailingLLM(
[
@@ -2026,7 +2380,6 @@ def test_zero_retries_is_one_plain_attempt(monkeypatch: pytest.MonkeyPatch) -> N
"""The kill-switch path (``llm_retries=0``): a dead round raises
immediately — one request, no RetryPiece, no sleep (pre-phase-67
behavior)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = FailingLLM([([], LLMError("connection refused"))])
sleeps = _record_sleeps(monkeypatch)
@@ -2065,7 +2418,6 @@ def test_abandon_mid_retry_sleep_leaks_nothing(monkeypatch: pytest.MonkeyPatch)
await asyncio.Event().wait() # park until the abandon arrives
monkeypatch.setattr(asyncio, "sleep", parking_sleep)
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = FailingLLM(
[([], LLMError("endpoint down")), ([StreamPiece("content", "never")], None)]
@@ -2111,7 +2463,7 @@ def test_retries_are_invisible_to_the_round_cap(
2, the retried first round and the second tool round fill the cap —
the forced final follows the SECOND call, and the log lines read
round=1/2 and round=2/2."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = FailingLLM(
[
@@ -2364,7 +2716,7 @@ def test_scaffolding_round_with_tool_calls_needs_no_recovery(
ran, and the policy keys on the no-calls exit only — no recovery (the
next round is a normal tools-offered round carrying the tool
history)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = ScriptedLLM(
[
@@ -2398,7 +2750,7 @@ def test_recovery_after_tool_rounds_keeps_the_history(
keeps the SINGLE (folded) system message at the front and the tool
history intact behind it — no second system message, no duplicated
correction."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
@@ -2423,7 +2775,7 @@ def test_recovery_after_tool_rounds_keeps_the_history(
assert recovered[3] == {
"role": "tool",
"tool_call_id": "call_1",
"content": "1 documents:\nsource: S | path: a.md | title: A",
"content": "1 sources:\n\nS — 1 documents",
}
assert sum(1 for m in recovered if m["role"] == "system") == 1
assert holder.scaffold_stripped == len(_INCIDENT_SPAN)
@@ -2436,7 +2788,7 @@ def test_forced_final_scaffolding_only_settles_malformed(
scaffolding-only forced answer never reaches the user raw — the turn
settles with :class:`MalformedReplyError` (the same terminal
semantics; this turn used no recovery, so nothing is doubled up)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],