Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 94_ls_tree_drilldown
Build and Push Containers / build-and-push-app (push) Successful in 1m45s
Details
Build and Push Containers / build-and-push-db (push) Successful in 25s
Details

Browse Source 
All green. Verification complete.

**Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)**

- Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal
- Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths
- Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met
- `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %)
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched)
- Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed
- Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol)

**Next pending phase:** `95_read_truncation_cap`
This commit is contained in:
ducoterra
2026-09-11 00:59:35 -04:00
parent 9188be259b
commit d4943b4822
61 changed files with 6289 additions and 666 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/unit/test_agent.py
+480 -128
View File
@@ -2,15 +2,20 @@
harness-aligned ``ls``/``read``/``grep`` surface, phase 70).
A scripted fake LLM (canned stream sequences) + monkeypatched
``list_catalog`` / ``list_source_names`` / ``find_document`` /
``ls_top`` / ``ls_folder`` / ``list_source_names`` / ``find_document`` /
``all_documents`` — no database, no network. Covers the loop mechanics:
the ls → read (combined ``source/path``) → answer happy path (event
order, holder state, the tools staying offered on every request —
phase 45 removed the per-tool budgets, the assistant/tool message
history), the ``ls`` scoping (no-arg full catalog in the phase-63
labeled-field format, a one-source scope, a known source with 0
documents → ``0 documents:`` counted, an unknown-source refusal that
counts nothing), ``read`` on the canonical combined form (split at the
history), the phase-94 drill-down ``ls`` (no-arg top level = the
registered sources with counts + stored summaries in the pinned
``{N} sources:`` template, a source scope = its root folder —
subfolders + capped file lines in the pinned folder template —, a
``source/folder`` scope = one level deeper, a registered source with 0
documents → the ``… — 0 documents, 0 folders:`` header counted, an
unknown-source refusal that counts nothing, the NOT-A-FOLDER teaching
with the parent's subfolders, the 50-file cap + grep-pointer note),
``read`` on the canonical combined form (split at the
FIRST slash, full content, the bare-source-name refusal, the
already-in-context dedupe, missing-args refusals), the phase-68 ``grep``
contract under its new name (the locked A5 pins: fixed substring,
@@ -149,28 +154,35 @@ def test_agent_tools_names_and_parameters() -> None:
# Task 05 (live gate iteration 2): the one-call-at-a-time discipline
# clause (the harness prior batches calls; the loop executes one
# per round — the extras count as unexecuted in the gate).
# Phase 94 (task 03): the description is the drill-down tree
# contract (pinned copy — the tool-surface revision, owner
# permission 2026-09-10, ``TODO.md`` L4): one level per call,
# sources at the top, folders + files below, the file-line format
# and the combined-identity handoff to read/grep intact.
assert ls["description"] == (
"List the indexed documents as `source: X | path: Y | "
"title: Z` lines. Call one tool at a time — wait for "
"this result before your next call."
"List the knowledge base as a tree, one level at a time. "
"With no path: the synced sources — each with its document "
"count and a summary of its contents. With a source name (no "
"'/'): that source's top-level folders and files. With a "
"`source/folder` path: that folder's subfolders and files. "
"Folder lines carry a summary of what the folder contains. "
"File lines are `source: X | path: Y | title: Z` — use the "
"combined `source/path` with `read` and `grep`. Call one tool "
"at a time — wait for this result before your next call."
)
ls_params = ls["parameters"]
assert ls_params["type"] == "object"
assert ls_params["required"] == [] # path is optional
assert set(ls_params["properties"]) == {"path"}
assert ls_params["properties"]["path"]["type"] == "string"
# Phase 72: the description states the contract up front — the
# 'path' argument is a source name, not a file or directory path.
# Task 05 (live gate iteration 5): the cross-tool contrast clause
# (ls is the ONLY tool whose path is a source name — the model
# kept transferring that scope to grep's document identity).
# Phase 94 (task 03): the 'path' argument teaches the drill-down
# semantics — a source name lists that source's top level, a
# `source/folder` path drills one level deeper, omitted lists
# every source (pinned copy).
assert ls_params["properties"]["path"]["description"] == (
"Source name to list one source's documents (e.g. 'homelab') — "
"a source name, not a file or directory path; omit to list "
"every document. This is the only tool "
"whose `path` is a source name — for "
"`read` and `grep` it must be a document's "
"combined `source/path`."
"Optional — a source name (e.g. 'homelab') to list its top "
"level, or a `source/folder` path to drill down (e.g. "
"'homelab/active'). Omit it to list every source."
)
read = by_name["read"]["function"]
# Tool-calling fast loop (2026-09-04, controlled fixture gate):
@@ -285,16 +297,11 @@ def test_refusal_constants_are_harness_aligned() -> None:
assert agent.UNKNOWN_TOOL == "Unknown tool."
assert agent.MISSING_READ_ARGS == "read requires a string argument 'path'."
assert agent.MISSING_SEARCH_ARGS == "grep requires a string argument 'pattern'."
# Phase 72: the ls teaching-refusal templates, pinned byte-for-byte
# (task 01 — the read/grep suggestion templates below, task 02).
assert agent.LS_PATH_NOT_A_SOURCE == (
"'{path}' looks like a document path, not a source name. The "
"'path' argument of ls filters by source name (e.g. 'homelab') — "
"omit it to list every document, or read a document by its "
"combined 'source/path' string."
)
# The pre-phase-72 no-source line is the byte-identical prefix of
# the extended line — only the teaching parenthetical was appended.
# Phase 94 (task 03): the phase-72 document-path teaching refusal is
# DELETED (a ``/`` now names a folder — the drill-down contract);
# the no-source refusal stays byte-identical (the task's "existing
# refusal, teaching parenthetical intact" pin).
assert not hasattr(agent, "LS_PATH_NOT_A_SOURCE")
assert agent.NO_SOURCE_NOT_A_DIRECTORY.startswith(
"No source named '{scope}' — check the ls output."
)
@@ -303,6 +310,11 @@ def test_refusal_constants_are_harness_aligned() -> None:
"argument is a source name, not a directory — omit it to list "
"every document.)"
)
# Phase 94 (task 03): the NOT-A-FOLDER drill-down teaching template,
# pinned byte-for-byte (argument echoed, parent's subfolders
# listed), and the pinned file-line cap constant.
assert agent.NOT_A_FOLDER == "'{arg}' is not a folder — {parent} has: {subfolders}"
assert agent.LS_MAX_FILE_LINES == 50
# Phase 72 (task 02): the read/grep "did you mean …?" suggestion
# templates, pinned byte-for-byte, and the suggestion cap.
assert agent.NO_DOCUMENT_DID_YOU_MEAN == (
@@ -348,11 +360,12 @@ def test_list_source_names_empty_registry(monkeypatch: pytest.MonkeyPatch) -> No
def test_ls_then_read_then_answer(monkeypatch: pytest.MonkeyPatch) -> None:
catalog = [
("Deployments", "backups.md", "Backup Strategy"),
("Homelab", "aws-route53.md", "AWS Route53 Records"),
]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
# Phase 94 (task 03): the no-arg ``ls`` is the drill-down TOP level
# (the registered sources, registry order) — monkeypatched the way
# the phase-70 full-catalog listing used to be.
monkeypatch.setattr(
agent, "ls_top", lambda db: [("Deployments", 1, None), ("Homelab", 1, None)]
)
target = _doc("Homelab", "aws-route53.md", "AWS Route53 Records", "R53-CONTENT")
calls: list[tuple[str, str]] = []
@@ -423,11 +436,7 @@ def test_ls_then_read_then_answer(monkeypatch: pytest.MonkeyPatch) -> None:
assert msgs[3] == {
"role": "tool",
"tool_call_id": "call_1",
"content": (
"2 documents:\n"
"source: Deployments | path: backups.md | title: Backup Strategy\n"
"source: Homelab | path: aws-route53.md | title: AWS Route53 Records"
),
"content": "2 sources:\n\nDeployments — 1 documents\nHomelab — 1 documents",
}
# The second follow-up request carries the read call + the FULL text.
msgs = llm.requests[2][0]
@@ -448,7 +457,7 @@ def test_content_and_tool_call_in_one_stream_keeps_both(
) -> None:
"""Rare stream with content AND a tool call: the content stays (it was
already emitted) and the tool still runs."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
monkeypatch.setattr(agent, "ls_top", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[
@@ -460,7 +469,7 @@ def test_content_and_tool_call_in_one_stream_keeps_both(
pieces = asyncio.run(_run(llm, holder, _settings()))
assert [type(p) for p in pieces] == [StreamPiece, ToolCallPiece, StreamPiece]
assert holder.tool_calls == 1 # the tool ran despite the content
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
assert llm.requests[1][0][3]["content"] == "0 sources:"
# ---------- phase 74: client history between system and user ----------
@@ -519,7 +528,7 @@ def test_run_agent_history_survives_a_tool_round(
"""The tool rounds append assistant/tool messages to the SAME
``messages`` list — the prior history stays in place between the
system prompt and the current question on the SECOND request too."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
monkeypatch.setattr(agent, "ls_top", lambda db: [])
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
[StreamPiece("content", "the answer")],
@@ -534,22 +543,25 @@ def test_run_agent_history_survives_a_tool_round(
]
# ---------- ls: full catalog + scoping ----------
# ---------- ls: the drill-down tree (phase 94, task 03) ----------
def test_ls_full_catalog_format(monkeypatch: pytest.MonkeyPatch) -> None:
"""No argument: the full catalog in the phase-63 labeled-field format
(``source: X | path: Y | title: Z``) — counted; no registry lookup."""
catalog = [
("Deployments", "backups.md", "Backup Strategy"),
("Homelab", "aws-route53.md", "AWS Route53 Records"),
]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
def _boom_sources(*_a: Any, **_k: Any) -> None:
raise AssertionError("no registry lookup for an unscoped ls")
monkeypatch.setattr(agent, "list_source_names", _boom_sources)
def test_ls_top_level_lists_sources_with_summaries(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""No argument: the TOP level — the registered sources, registry
order, each ``{source} — {n} documents`` + the indented summary line
only when stored — the pinned template, counted; the registry IS
consulted (unlike the phase-70 full catalog, the top level is the
registry itself)."""
monkeypatch.setattr(
agent,
"ls_top",
lambda db: [
("Deployments", 0, None),
("Homelab", 1, "The homelab notes."),
],
)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
@@ -557,22 +569,27 @@ def test_ls_full_catalog_format(monkeypatch: pytest.MonkeyPatch) -> None:
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == (
"2 documents:\n"
"source: Deployments | path: backups.md | title: Backup Strategy\n"
"source: Homelab | path: aws-route53.md | title: AWS Route53 Records"
"2 sources:\n"
"\n"
"Deployments — 0 documents\n"
"Homelab — 1 documents\n"
" The homelab notes."
)
assert holder.tool_calls == 1
def test_ls_empty_catalog_says_zero_documents(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
def test_ls_empty_registry_says_zero_sources(monkeypatch: pytest.MonkeyPatch) -> None:
"""No registered sources: the top level is the header line alone —
``0 sources:`` (the old ``0 documents:`` behavior preserved in
spirit), still a counted result."""
monkeypatch.setattr(agent, "ls_top", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
assert llm.requests[1][0][3]["content"] == "0 sources:"
assert holder.tool_calls == 1
@@ -583,13 +600,12 @@ def test_ls_empty_catalog_says_zero_documents(monkeypatch: pytest.MonkeyPatch) -
({"path": 7}, "non-string path"),
],
)
def test_ls_blank_path_lists_full_catalog(
def test_ls_blank_path_lists_top_level(
monkeypatch: pytest.MonkeyPatch, arguments: dict[str, Any], label: str
) -> None:
"""A blank (or non-string) ``path`` is treated as omitted — the full
catalog, counted (no refusal for an empty scope)."""
catalog = [("S", "a.md", "A")]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
"""A blank (or non-string) ``path`` is treated as omitted — the top
level (the sources), counted (no refusal for an empty scope)."""
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -597,21 +613,25 @@ def test_ls_blank_path_lists_full_catalog(
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert (
llm.requests[1][0][3]["content"] == "1 documents:\nsource: S | path: a.md | title: A"
)
assert llm.requests[1][0][3]["content"] == "1 sources:\n\nS — 1 documents"
assert holder.tool_calls == 1
def test_ls_scoped_to_known_source(monkeypatch: pytest.MonkeyPatch) -> None:
"""A known source name: the same listing filtered to that source —
def test_ls_source_scope_lists_root_folder(monkeypatch: pytest.MonkeyPatch) -> None:
"""A registered source name (no ``/``): the source's ROOT folder —
subfolders (2-space-indented, path order, ``: {summary}`` only when
stored) + the root's own file lines in EXACTLY the
``source: X | path: Y | title: Z`` format — the pinned template,
counted."""
catalog = [
("Deployments", "backups.md", "Backup Strategy"),
("Homelab", "a.md", "A"),
("Homelab", "b.md", "B"),
]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
monkeypatch.setattr(
agent,
"ls_folder",
lambda db, source, folder: (
[("backups", 2, "Backup notes."), ("networking", 1, None)],
[("Homelab", "readme.md", "Readme")],
1,
),
)
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Deployments", "Homelab"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -620,9 +640,50 @@ def test_ls_scoped_to_known_source(monkeypatch: pytest.MonkeyPatch) -> None:
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == (
"2 documents:\n"
"source: Homelab | path: a.md | title: A\n"
"source: Homelab | path: b.md | title: B"
"Homelab — 1 documents, 2 folders:\n"
"\n"
" backups/ — 2 documents: Backup notes.\n"
" networking/ — 1 documents\n"
"\n"
"source: Homelab | path: readme.md | title: Readme"
)
assert holder.tool_calls == 1
def test_ls_nested_folder_scope_lists_one_level_deeper(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A ``source/folder`` path: that folder's subfolders + own file
lines, identity = ``source/folder`` (the same template as the
root), counted; the fetchers are the source-scoped ones."""
def _rows(db: Any, source: str) -> list[tuple[str, str]]:
assert (source, db) == ("Homelab", None)
return [
("networking/lan.md", "LAN"),
("networking/vpn.md", "VPN"),
]
monkeypatch.setattr(agent, "_source_document_rows", _rows)
monkeypatch.setattr(
agent, "_source_folder_summaries", lambda db, source: {"networking": "Network notes."}
)
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab"])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1", name="ls", arguments={"path": "Homelab/networking"}
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == (
"Homelab/networking — 2 documents, 0 folders:\n"
"\n"
"source: Homelab | path: networking/lan.md | title: LAN\n"
"source: Homelab | path: networking/vpn.md | title: VPN"
)
assert holder.tool_calls == 1
@@ -631,9 +692,13 @@ def test_ls_scoped_known_source_with_zero_docs_counts(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A registered source with no indexed documents is KNOWN (the
registry is the source of truth, not the catalog): it lists as
``0 documents:`` — a valid, counted result, not a refusal."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("Other", "a.md", "A")])
registry is the source of truth, not the catalog): it lists its
header line alone (``… — 0 documents, 0 folders:`` — the old
``0 documents:`` behavior preserved in spirit) — a valid, counted
result, not a refusal."""
monkeypatch.setattr(
agent, "ls_folder", lambda db, source, folder: ([], [], 0)
)
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab", "Other"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -641,7 +706,7 @@ def test_ls_scoped_known_source_with_zero_docs_counts(
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert llm.requests[1][0][3]["content"] == "0 documents:\n"
assert llm.requests[1][0][3]["content"] == "Homelab — 0 documents, 0 folders:"
assert holder.tool_calls == 1 # an executed ls, not a refusal
assert llm.requests[1][1] == AGENT_TOOLS
@@ -650,7 +715,6 @@ def test_ls_scoped_unknown_source_refused(monkeypatch: pytest.MonkeyPatch) -> No
"""A ``path`` without ``/`` matching no source name is a refusal —
the extended line with the teaching parenthetical (phase 72), not
counted, the round cap bounds its repetition."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -666,20 +730,14 @@ def test_ls_scoped_unknown_source_refused(monkeypatch: pytest.MonkeyPatch) -> No
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_ls_path_like_scope_gets_document_path_teaching_refusal(
def test_ls_path_like_scope_unknown_source_gets_no_source_refusal(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 72: a stripped scope containing ``/`` looks like a document
path (the incident's ``ls(path='app/rag/importer.py')``) — a source
name is a directory basename and can never contain one, so this gets
the ``LS_PATH_NOT_A_SOURCE`` teaching line with the argument echoed;
no registry lookup, counts in nothing, tools stay offered."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
def _boom_sources(*_a: Any, **_k: Any) -> None:
raise AssertionError("no registry lookup for a path-like scope")
monkeypatch.setattr(agent, "list_source_names", _boom_sources)
"""Phase 94: a ``/`` now names a folder, so the phase-72
document-path teaching is DELETED — a ``source/…`` argument whose
FIRST segment names no registered source gets the no-source refusal
(the segment echoed), counted in nothing, tools stay offered."""
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
[
@@ -695,7 +753,7 @@ def test_ls_path_like_scope_gets_document_path_teaching_refusal(
assert holder.tool_calls == 0 # a refusal counts in nothing
assert (
llm.requests[1][0][3]["content"]
== agent.LS_PATH_NOT_A_SOURCE.format(path="app/rag/importer.py")
== agent.NO_SOURCE_NOT_A_DIRECTORY.format(scope="app")
)
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
@@ -706,8 +764,7 @@ def test_ls_dot_scope_gets_not_a_directory_teaching_refusal(
"""Phase 72: ``ls(path='.')`` (the incident's second round — no
``/``, no matching source) gets the extended no-source refusal with
the teaching parenthetical, ``'.'`` echoed — not counted, tools stay
offered."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
offered (unchanged by phase 94)."""
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
@@ -723,6 +780,312 @@ def test_ls_dot_scope_gets_not_a_directory_teaching_refusal(
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_ls_unknown_top_level_folder_gets_not_a_folder_teaching(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 94: a ``source/…`` argument whose TOP-LEVEL folder segment
matches no indexed prefix gets the NOT-A-FOLDER teaching — the
argument echoed, the source named, its direct subfolders listed so
the model self-corrects in the next round; not counted, tools stay
offered."""
monkeypatch.setattr(
agent,
"_source_document_rows",
lambda db, source: [
("backups/cron.md", "Cron"),
("containers/caddy.md", "Caddy"),
("networking/lan.md", "LAN"),
],
)
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab"])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1", name="ls", arguments={"path": "Homelab/netwoking"}
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 0 # a refusal counts in nothing
assert llm.requests[1][0][3]["content"] == (
"'Homelab/netwoking' is not a folder — Homelab has: "
"backups/ containers/ networking/"
)
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
def test_ls_unknown_nested_folder_gets_not_a_folder_with_nested_parent(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 94: a nested miss names the DEEPEST existing ancestor —
``source/folder`` — and lists ITS direct subfolders (bounded: the
parent's own listing, no new flood path)."""
monkeypatch.setattr(
agent,
"_source_document_rows",
lambda db, source: [
("networking/lan/a.md", "A"),
("networking/vpn/b.md", "B"),
("readme.md", "Readme"),
],
)
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Homelab"])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1", name="ls", arguments={"path": "Homelab/networking/lan/x"}
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 0
assert llm.requests[1][0][3]["content"] == (
"'Homelab/networking/lan/x' is not a folder — "
"Homelab/networking/lan has: none"
)
def test_ls_file_path_scope_gets_not_a_folder(monkeypatch: pytest.MonkeyPatch) -> None:
"""Phase 94: a document's OWN path is never a folder (nothing starts
with ``path + '/'``) — ``ls`` of a file path refuses with the
NOT-A-FOLDER teaching (the parent's subfolders listed)."""
monkeypatch.setattr(
agent,
"_source_document_rows",
lambda db, source: [("notes.md", "Notes"), ("a/b.md", "B")],
)
monkeypatch.setattr(agent, "_source_folder_summaries", lambda db, source: {})
monkeypatch.setattr(agent, "list_source_names", lambda db: ["S"])
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(id="call_1", name="ls", arguments={"path": "S/notes.md"})
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 0
assert llm.requests[1][0][3]["content"] == (
"'S/notes.md' is not a folder — S has: a/"
)
# ---------- ls_top / ls_folder: the drill-down accessors (pure + composed) ----------
def test_ls_top_registry_order_zero_docs_and_summaries(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""``ls_top``: registry order (not catalog order), a 0-document
source still lists, the summary is the stored ``(source, "")`` row
or ``None`` when absent; an empty registry → ``[]``."""
monkeypatch.setattr(agent, "list_source_names", lambda db: ["Zeta", "Alpha"])
monkeypatch.setattr(
agent, "_source_document_counts", lambda db: [("Zeta", 3), ("Beta", 1)]
)
monkeypatch.setattr(
agent, "_source_root_summaries", lambda db: [("Zeta", "Zeta stuff.")]
)
assert agent.ls_top(cast("Session", object())) == [
("Zeta", 3, "Zeta stuff."),
("Alpha", 0, None), # 0 docs (no count row) + no stored summary
]
def test_ls_top_empty_registry_is_empty(
monkeypatch: pytest.MonkeyPatch,
) -> None:
monkeypatch.setattr(agent, "list_source_names", lambda db: [])
def _boom(*_a: Any, **_k: Any) -> None:
raise AssertionError("no fetches for an empty registry")
monkeypatch.setattr(agent, "_source_document_counts", _boom)
monkeypatch.setattr(agent, "_source_root_summaries", _boom)
assert agent.ls_top(cast("Session", object())) == []
def test_ls_folder_composes_the_fetchers(monkeypatch: pytest.MonkeyPatch) -> None:
"""``ls_folder`` = the source's document rows + stored summaries
through the pure :func:`group_folder_listing` (the fetchers are the
monkeypatch surface)."""
seen: list[tuple[str, str, str]] = []
def _rows(db: Any, source: str) -> list[tuple[str, str]]:
seen.append(("rows", source, ""))
return [("a/b.md", "B"), ("a.md", "A")]
def _summaries(db: Any, source: str) -> dict[str, str]:
seen.append(("summaries", source, ""))
return {"a": "A stuff."}
monkeypatch.setattr(agent, "_source_document_rows", _rows)
monkeypatch.setattr(agent, "_source_folder_summaries", _summaries)
assert agent.ls_folder(cast("Session", object()), "S", "") == (
[("a", 1, "A stuff.")],
[("S", "a.md", "A")],
1,
)
assert seen == [("rows", "S", ""), ("summaries", "S", "")]
def test_group_folder_listing_subfolder_recursion_and_counts() -> None:
"""The recursive count per subfolder — every path equal to the
folder or starting with ``folder + '/'`` (a doc under ``a/b/``
counts for BOTH ``a`` and ``a/b``), path order, the stored summary
attached or ``None``."""
rows = [
("a/b/c.md", "C"),
("a/b/d.md", "D"),
("a/e.md", "E"),
("f.md", "F"),
]
sub, files, total = agent.group_folder_listing(
"S", "", rows, {"a": "A subtree.", "a/b": "B subtree."}
)
# ROOT level: the direct subfolders of "" are the TOP-LEVEL folders
# only (a/b is nested under a, not direct) — a's count is its whole
# recursive subtree (a/e.md + a/b/c.md + a/b/d.md), the stored
# summary attached.
assert sub == [("a", 3, "A subtree.")]
assert files == [("S", "f.md", "F")]
assert total == 1
# One level down: a/b is a's direct subfolder with its own count.
sub2, _files2, _total2 = agent.group_folder_listing("S", "a", rows, {"a/b": "B subtree."})
assert sub2 == [("a/b", 2, "B subtree.")]
def test_group_folder_listing_nested_level_counts_and_membership() -> None:
"""One level down: ``a``'s direct subfolder is ``a/b`` (count 2),
its own direct file is ``a/e.md`` (``a/b/c.md`` is NOT a direct
file of ``a``) — membership is the folder_of rule, order is path
order."""
rows = [
("a/b/c.md", "C"),
("a/b/d.md", "D"),
("a/e.md", "E"),
]
sub, files, total = agent.group_folder_listing("S", "a", rows, {})
assert sub == [("a/b", 2, None)]
assert files == [("S", "a/e.md", "E")]
assert total == 1
def test_group_folder_listing_file_path_is_not_a_folder() -> None:
"""A document whose path is a prefix of NO other path is a file,
never a folder: ``ls`` of it must not list a subfolder (and the
``path == folder`` count arm only fires for TRUE folders — a doc
sharing a real folder's name counts for that folder, the existence
rule intact)."""
rows = [
("a.md", "A"), # a file at the root, and a folder name? NO —
("b/x.md", "X"), # nothing starts with "a.md/"
]
sub, files, total = agent.group_folder_listing("S", "", rows, {})
assert sub == [("b", 1, None)] # "a.md" is NOT a subfolder
assert files == [("S", "a.md", "A")] # b/x.md is NOT a direct root file
assert total == 1
# The path == folder arm: a doc named "a" under a real folder "a/".
rows2 = [("a", "FileA"), ("a/c.md", "C")]
sub2, files2, total2 = agent.group_folder_listing("S", "", rows2, {})
assert sub2 == [("a", 2, None)] # the file "a" counts for folder "a"
assert files2 == [("S", "a", "FileA")] # …and is a direct ROOT file
assert total2 == 1
def test_group_folder_listing_caps_files_at_fifty_keeps_the_total() -> None:
"""The cap: 51 direct files → 50 file lines + the PRE-cap total (51)
for the renderer's note; 50 files → 50 lines, no note material.
A 500-file folder costs 50 lines, never 500."""
rows51 = [(f"big/f{i:03d}.md", f"T{i}") for i in range(51)]
sub, files, total = agent.group_folder_listing("S", "big", rows51, {})
assert sub == []
assert total == 51
assert len(files) == 50
assert files[0] == ("S", "big/f000.md", "T0")
assert files[-1] == ("S", "big/f049.md", "T49")
rows50 = [(f"big/f{i:03d}.md", f"T{i}") for i in range(50)]
_sub, files50, total50 = agent.group_folder_listing("S", "big", rows50, {})
assert total50 == 50 and len(files50) == 50
# ---------- the pinned drill-down templates (byte-for-byte) ----------
def test_render_ls_top_template() -> None:
assert (
agent.render_ls_top(
[("Deployments", 3, None), ("Homelab", 5, "Home lab notes.")]
)
== "2 sources:\n\n"
"Deployments — 3 documents\n"
"Homelab — 5 documents\n"
" Home lab notes."
)
assert agent.render_ls_top([]) == "0 sources:"
def test_render_folder_listing_root_template() -> None:
assert (
agent.render_folder_listing(
"Homelab",
[("backups", 2, "Backup notes."), ("networking", 1, None)],
[("Homelab", "readme.md", "Readme")],
1,
)
== "Homelab — 1 documents, 2 folders:\n"
"\n"
" backups/ — 2 documents: Backup notes.\n"
" networking/ — 1 documents\n"
"\n"
"source: Homelab | path: readme.md | title: Readme"
)
def test_render_folder_listing_empty_level_is_header_alone() -> None:
"""A registered source with no documents: the header line alone —
the old ``0 documents:`` behavior preserved in spirit."""
assert agent.render_folder_listing("Homelab", [], [], 0) == (
"Homelab — 0 documents, 0 folders:"
)
def test_render_folder_listing_subfolders_only_no_blank_trailer() -> None:
"""Subfolders but no own files: header + blank + subfolder lines —
no trailing blank line, no file section."""
assert (
agent.render_folder_listing("S", [("a", 1, None)], [], 0)
== "S — 0 documents, 1 folders:\n\n a/ — 1 documents"
)
def test_render_folder_listing_cap_note_only_past_fifty() -> None:
"""The note appears ONLY when the folder's own files outnumber the
cap: 51 → 50 lines + the deterministic grep-pointer note (the
``…and 1 more…`` shape — unpluralized, the house pin); 50 → no
note."""
files51 = [("S", f"f{i:03d}.md", f"T{i}") for i in range(51)]
capped = files51[:50]
rendered = agent.render_folder_listing("S/big", [], capped, 51)
lines = rendered.splitlines()
assert lines[0] == "S/big — 51 documents, 0 folders:"
assert len(lines) == 1 + 1 + 50 + 1 # header, blank, 50 lines, note
assert lines[-1] == (
"…and 1 more documents in this folder — use grep (pattern) to "
"find a specific one."
)
files50 = [("S", f"f{i:03d}.md", f"T{i}") for i in range(50)]
rendered50 = agent.render_folder_listing("S/big", [], files50, 50)
assert rendered50.splitlines()[-1] == "source: S | path: f049.md | title: T49"
assert "more documents" not in rendered50
# ---------- read: the canonical combined source/path form ----------
@@ -771,7 +1134,6 @@ def test_read_bare_source_name_refused_without_db(monkeypatch: pytest.MonkeyPatc
no-document refusal (the argument echoed as passed), no DB lookup
(NOT even the phase-72 candidate lookup — ``all_documents`` must
not run either), nothing counted."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("Homelab", "a.md", "A")])
def _boom(*_a: Any, **_k: Any) -> None:
raise AssertionError(
@@ -1158,7 +1520,6 @@ def test_reading_an_already_read_doc_is_deduped(monkeypatch: pytest.MonkeyPatch)
def test_unknown_tool_name_refused(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="delete_universe", arguments={"x": 1})],
@@ -1702,13 +2063,11 @@ def test_grep_counts_but_never_adds_context(monkeypatch: pytest.MonkeyPatch) ->
def test_relist_executes_and_counts(monkeypatch: pytest.MonkeyPatch) -> None:
"""Re-lists execute — a second ``ls`` in one turn returns the catalog
again and counts in ``tool_calls`` (no budget to exhaust)."""
catalog = [
("Deployments", "backups.md", "Backup Strategy"),
("Homelab", "aws-route53.md", "AWS Route53 Records"),
]
monkeypatch.setattr(agent, "list_catalog", lambda db: catalog)
"""Re-lists execute — a second ``ls`` in one turn returns the top
level again and counts in ``tool_calls`` (no budget to exhaust)."""
monkeypatch.setattr(
agent, "ls_top", lambda db: [("Deployments", 1, None), ("Homelab", 1, None)]
)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
@@ -1717,12 +2076,8 @@ def test_relist_executes_and_counts(monkeypatch: pytest.MonkeyPatch) -> None:
)
asyncio.run(_run(llm, holder, _settings()))
assert holder.tool_calls == 2 # both re-lists executed and counted
listing = (
"2 documents:\n"
"source: Deployments | path: backups.md | title: Backup Strategy\n"
"source: Homelab | path: aws-route53.md | title: AWS Route53 Records"
)
# The answer request carries the catalog a second time as a tool result.
listing = "2 sources:\n\nDeployments — 1 documents\nHomelab — 1 documents"
# The answer request carries the listing a second time as a tool result.
assert llm.requests[2][0][3]["content"] == listing # first listing
assert llm.requests[2][0][5]["content"] == listing # the re-list
assert llm.requests[2][1] == AGENT_TOOLS # still offered (no budgets)
@@ -1758,8 +2113,8 @@ def test_always_ls_bounded_by_round_cap(monkeypatch: pytest.MonkeyPatch) -> None
"""A model that keeps calling ``ls`` gets exactly
``agent_max_rounds`` tool rounds, then one forced ``tools=None``
request streams the answer — the cap is the only forced exit."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
listing = "1 documents:\nsource: S | path: a.md | title: A"
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
listing = "1 sources:\n\nS — 1 documents"
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
@@ -1923,7 +2278,7 @@ def test_round_retried_before_first_piece(
same messages: the stream carries a RetryPiece BEFORE the tool call,
the tool executes, the final answer streams, and the per-call log line
is still emitted exactly once (retries are invisible to the loop)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = FailingLLM(
[
@@ -1961,7 +2316,6 @@ def test_round_failure_after_first_piece_is_terminal(monkeypatch: pytest.MonkeyP
the LLMError propagates out of ``run_agent``, no RetryPiece, no
sleep, no second request, and the holder is untouched (the tool
never ran)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = FailingLLM(
[([StreamPiece("content", "partial ")], LLMError("mid-stream drop"))]
@@ -1994,7 +2348,7 @@ def test_forced_final_no_tools_call_is_retried(monkeypatch: pytest.MonkeyPatch)
"""The forced final request (round cap reached) goes through the same
retry rule: a failure before its first piece yields a RetryPiece and
restarts with ``tools=None``; the answer from the retry streams."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = FailingLLM(
[
@@ -2026,7 +2380,6 @@ def test_zero_retries_is_one_plain_attempt(monkeypatch: pytest.MonkeyPatch) -> N
"""The kill-switch path (``llm_retries=0``): a dead round raises
immediately — one request, no RetryPiece, no sleep (pre-phase-67
behavior)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = FailingLLM([([], LLMError("connection refused"))])
sleeps = _record_sleeps(monkeypatch)
@@ -2065,7 +2418,6 @@ def test_abandon_mid_retry_sleep_leaks_nothing(monkeypatch: pytest.MonkeyPatch)
await asyncio.Event().wait() # park until the abandon arrives
monkeypatch.setattr(asyncio, "sleep", parking_sleep)
monkeypatch.setattr(agent, "list_catalog", lambda db: [])
holder = AgentHolder()
llm = FailingLLM(
[([], LLMError("endpoint down")), ([StreamPiece("content", "never")], None)]
@@ -2111,7 +2463,7 @@ def test_retries_are_invisible_to_the_round_cap(
2, the retried first round and the second tool round fill the cap —
the forced final follows the SECOND call, and the log lines read
round=1/2 and round=2/2."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = FailingLLM(
[
@@ -2364,7 +2716,7 @@ def test_scaffolding_round_with_tool_calls_needs_no_recovery(
ran, and the policy keys on the no-calls exit only — no recovery (the
next round is a normal tools-offered round carrying the tool
history)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = ScriptedLLM(
[
@@ -2398,7 +2750,7 @@ def test_recovery_after_tool_rounds_keeps_the_history(
keeps the SINGLE (folded) system message at the front and the tool
history intact behind it — no second system message, no duplicated
correction."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
@@ -2423,7 +2775,7 @@ def test_recovery_after_tool_rounds_keeps_the_history(
assert recovered[3] == {
"role": "tool",
"tool_call_id": "call_1",
"content": "1 documents:\nsource: S | path: a.md | title: A",
"content": "1 sources:\n\nS — 1 documents",
}
assert sum(1 for m in recovered if m["role"] == "system") == 1
assert holder.scaffold_stripped == len(_INCIDENT_SPAN)
@@ -2436,7 +2788,7 @@ def test_forced_final_scaffolding_only_settles_malformed(
scaffolding-only forced answer never reaches the user raw — the turn
settles with :class:`MalformedReplyError` (the same terminal
semantics; this turn used no recovery, so nothing is doubled up)."""
monkeypatch.setattr(agent, "list_catalog", lambda db: [("S", "a.md", "A")])
monkeypatch.setattr(agent, "ls_top", lambda db: [("S", 1, None)])
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="ls", arguments={})],
tests/unit/test_folder_summaries.py
+628
View File
@@ -0,0 +1,628 @@
"""Unit: folder summary storage + generator (phase 94, task 01).
The prompt/grouping tests are pure (no DB): ``FOLDER_SUMMARY_MODE``
system prompt, the ``folder_of`` / ``group_by_folder`` recursive-subtree
concept, and the user-message cap with the shared ``[…truncated…]``
marker. The generator tests run against the local compose Postgres
(preferred — real upsert/prune on the ``folder_summaries`` table),
skipping with clear instructions when the stack is not up — same
pattern as ``test_overview.py``.
"""
from __future__ import annotations
import asyncio
import logging
import uuid
from datetime import UTC, datetime
from typing import Any
import pytest
from sqlalchemy import text
from sqlalchemy.orm import Session
from app.config import Settings, get_settings
from app.db import SessionLocal
from app.models import Document, FolderSummary
from app.rag.folder_summaries import (
FOLDER_HEADER_PREFIX,
FOLDER_SUMMARY_INSTRUCTION,
FOLDER_SUMMARY_MODE,
MIN_DOCS_PER_FOLDER,
SYSTEM_PROMPT,
build_folder_summary_prompt,
folder_of,
folder_summary_table_empty,
generate_folder_summaries,
group_by_folder,
summarize_folder,
)
from app.rag.llm import LLMError
from app.rag.retriever import TRUNCATION_MARKER
from tests.e2e.mock_llm import compose_answer
REPLY = "Covers lab automation runbooks: inventories, playbooks, and schedules."
class _FakeLLM:
"""Duck-typed stand-in for ``LLMClient`` (``chat`` + ``settings``).
Records each ``(system, user)`` request and the ``model`` kwarg;
returns the canned reply, or raises — either a fixed exception or a
per-folder failure keyed on the user message's ``Folder: …`` header
(the per-folder fail-soft tests).
"""
def __init__(
self,
reply: str = REPLY,
fail_folders: tuple[str, ...] = (),
fail: Exception | None = None,
) -> None:
self._reply = reply
self._fail_folders = tuple(fail_folders)
self._fail = fail
self.settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
self.calls = 0
self.model: str | None = None
self.requests: list[tuple[str, str]] = []
async def chat(
self, messages: list[dict[str, str]], model: str | None = None
) -> str:
self.calls += 1
self.model = model
system = messages[0]["content"]
user = messages[-1]["content"]
self.requests.append((system, user))
for folder in self._fail_folders:
if FOLDER_HEADER_PREFIX + folder in user:
raise LLMError(f"simulated lite-model failure for {folder}")
if self._fail is not None:
raise self._fail
return self._reply
# ---------- folder_of ----------
def test_folder_of_root_level_file_is_empty() -> None:
assert folder_of("a.md") == ""
def test_folder_of_one_level() -> None:
assert folder_of("a/b.md") == "a"
def test_folder_of_deep_path() -> None:
assert folder_of("a/b/c/d.md") == "a/b/c"
def test_folder_of_iterating_walks_prefixes_to_root() -> None:
"""Iterating ``folder_of`` over its own result walks the folder
prefixes nearest-first, ending at the root (the grouping walk)."""
folder = folder_of("a/b/c.md")
chain: list[str] = []
while folder:
chain.append(folder)
folder = folder_of(folder)
assert chain == ["a/b", "a"] # plus the "" root the grouping adds
# ---------- group_by_folder ----------
def test_group_by_folder_root_file_lands_only_in_source_root() -> None:
rows = [("S", "top.md", "T", None)]
groups = group_by_folder(rows)
assert set(groups) == {("S", "")}
assert groups[("S", "")] == rows
def test_group_by_folder_nested_multi_source_recursive_subtree() -> None:
"""A doc under ``a/b/`` is present in the ``a``, ``a/b``, and ``""``
groups (recursive subtree — the ``ls`` count scope, one concept);
per source the candidates are ``""`` + every distinct folder
prefix; group lists keep the input (catalogue) order."""
rows = [
("S", "a/b/c.md", "C", None),
("S", "a/b/d.md", "D", None),
("S", "a/x.md", "X", None),
("S", "top.md", "T", None),
("T", "a/b/e.md", "E", None),
]
groups = group_by_folder(rows)
assert set(groups) == {
("S", ""),
("S", "a"),
("S", "a/b"),
("T", ""),
("T", "a"),
("T", "a/b"),
}
# The recursive-subtree concept: a/b/ docs in the a/, a/b/, and ""
# groups alike — exactly the set each level's ls count shows.
assert [r[1] for r in groups[("S", "a/b")]] == ["a/b/c.md", "a/b/d.md"]
assert [r[1] for r in groups[("S", "a")]] == ["a/b/c.md", "a/b/d.md", "a/x.md"]
assert [r[1] for r in groups[("S", "")]] == [
"a/b/c.md",
"a/b/d.md",
"a/x.md",
"top.md",
]
# Multi-source: the same folder prefix under another source is a
# separate group (PK is (source, folder_path)).
assert [r[1] for r in groups[("T", "a/b")]] == ["a/b/e.md"]
assert [r[1] for r in groups[("T", "")]] == ["a/b/e.md"]
# Input (catalogue) order is preserved inside each group.
assert [r[2] for r in groups[("S", "")]] == ["C", "D", "X", "T"]
def test_group_by_folder_single_doc_folder_is_a_group_too() -> None:
"""Grouping is pure subtree membership (≥ 1 docs): the ≥ 2 rule is
the GENERATOR's (the recursive count below the minimum yields no
row — pinned by the generator tests, not the grouping)."""
rows = [("S", "a/only.md", "O", None)]
groups = group_by_folder(rows)
assert len(groups[("S", "a")]) == 1 # present, but below the minimum
def test_group_by_folder_doc_path_equal_to_a_folder_prefix_counts_for_it() -> None:
"""The count rule's ``path == folder`` arm: a document whose path
IS one of the source's folder prefixes (a file sharing its name
with a directory) belongs to that folder's group too — the grouping
stays EXACTLY the set the ``ls`` count rule counts (path equal or
starting with ``folder + "/"``), while the returned keys remain the
true folder prefixes only (no file-path keys)."""
rows = [
("S", "a/b", "B", None), # a file named "b" ... (its path is a folder prefix)
("S", "a/b/c.md", "C", None), # ... and a real folder "a/b/" holding a doc
("S", "a/x.md", "X", None),
]
groups = group_by_folder(rows)
assert set(groups) == {("S", ""), ("S", "a"), ("S", "a/b")}, (
"the keys stay the true folder prefixes — the file's own path adds no key"
)
assert [r[1] for r in groups[("S", "a/b")]] == ["a/b", "a/b/c.md"]
assert [r[1] for r in groups[("S", "a")]] == ["a/b", "a/b/c.md", "a/x.md"]
assert [r[1] for r in groups[("S", "")]] == ["a/b", "a/b/c.md", "a/x.md"]
def test_group_by_folder_plain_file_path_is_not_a_group_key() -> None:
"""A file path that is NO folder prefix (no doc under it) adds no
group key of its own — the ``path == folder`` arm only fires when
the path really is a prefix of the catalogue."""
rows = [("S", "top.md", "T", None), ("S", "a/one.md", "O", None)]
groups = group_by_folder(rows)
assert set(groups) == {("S", ""), ("S", "a")}
assert "top.md" not in [folder for _source, folder in groups]
# ---------- build_folder_summary_prompt: system ----------
def test_system_prompt_has_marker_and_locked_instruction() -> None:
assert SYSTEM_PROMPT.startswith(FOLDER_SUMMARY_MODE)
assert FOLDER_SUMMARY_INSTRUCTION in SYSTEM_PROMPT
for fragment in (
"1-3 sentence",
"plain-text summary",
"natural language",
"Do not use markdown",
"not in the list",
):
assert fragment in SYSTEM_PROMPT
system, _ = build_folder_summary_prompt("S", "a/b", [])
assert system == SYSTEM_PROMPT
assert FOLDER_SUMMARY_MODE in system # the marker the E2E mock keys on
# ---------- build_folder_summary_prompt: user ----------
def test_user_prompt_header_names_the_folder() -> None:
"""The first line is the ``FOLDER_HEADER_PREFIX`` header the E2E
mock parses: ``<source>`` for the root, ``<source>/<folder_path>``
for a folder."""
_, user = build_folder_summary_prompt("Homelab", "deployments/ansible", [])
assert user == FOLDER_HEADER_PREFIX + "Homelab/deployments/ansible"
_, user = build_folder_summary_prompt("Homelab", "", [])
assert user == FOLDER_HEADER_PREFIX + "Homelab"
def test_user_lines_carry_path_title_and_first_summary_line() -> None:
docs = [
("S", "a/b/one.md", "One", "First lead.\nSecond line.\nSource: S/a/b/one.md"),
("S", "a/b/two.md", "Two", None),
]
system, user = build_folder_summary_prompt("S", "a/b", docs)
assert system == SYSTEM_PROMPT
assert user == (
"Folder: S/a/b\n"
"a/b/one.md — One — First lead.\n"
"a/b/two.md — Two"
)
def test_user_line_omits_summary_field_when_absent_or_blank() -> None:
docs = [
("S", "a/x.md", "X", None),
("S", "a/y.md", "Y", " \n\t "),
]
_, user = build_folder_summary_prompt("S", "a", docs)
assert user == "Folder: S/a\na/x.md — X\na/y.md — Y"
assert " — " in user # the path — title join only
assert not any(line.endswith(" — ") for line in user.splitlines())
def test_user_line_uses_only_first_summary_line() -> None:
docs = [
("S", "a/x.md", "X", "First line.\nSecond line.\nSource: S/a/x.md"),
]
_, user = build_folder_summary_prompt("S", "a", docs)
assert user == "Folder: S/a\na/x.md — X — First line."
assert "Second line" not in user
assert "Source:" not in user
def test_user_prompt_truncated_with_marker_when_over_custom_cap() -> None:
docs = [("S", f"a/f{i}.md", f"T{i}", None) for i in range(10)]
_, full = build_folder_summary_prompt("S", "a", docs, max_chars=10_000)
cap = 30
_, user = build_folder_summary_prompt("S", "a", docs, max_chars=cap)
assert user == full[:cap] + "\n" + TRUNCATION_MARKER
assert user.endswith(TRUNCATION_MARKER)
assert len(user) > cap # the marker makes the cut visible past the cap
def test_user_prompt_at_exact_cap_not_truncated() -> None:
docs = [("S", "a/x.md", "X", None)] # "Folder: S/a\na/x.md — X" = 22 chars
_, user = build_folder_summary_prompt("S", "a", docs, max_chars=22)
assert user == "Folder: S/a\na/x.md — X"
assert TRUNCATION_MARKER not in user
def test_user_prompt_truncated_at_default_cap() -> None:
"""No explicit cap → ``BOR_FOLDER_SUMMARY_INPUT_MAX_CHARS`` (read
from the live settings, so the test holds for any configured
value)."""
cap = get_settings().folder_summary_input_max_chars
docs = [("S", f"a/f{i}.md", "T", None) for i in range(3_000)]
_, user = build_folder_summary_prompt("S", "a", docs)
assert user.endswith(TRUNCATION_MARKER)
body = user.removesuffix("\n" + TRUNCATION_MARKER)
assert len(body) == cap # cut exactly at the cap, marker on its own line
assert "f2999.md" not in body # the overflow never reaches the model
# ---------- summarize_folder ----------
# ---------- the E2E mock's FOLDER_SUMMARY_MODE branch ----------
def _mock_body(system: str, user: str) -> dict[str, Any]:
"""A minimal chat-completion body for the mock's ``compose_answer``."""
return {"messages": [
{"role": "system", "content": system},
{"role": "user", "content": user},
]}
def test_mock_returns_canned_folder_summary_naming_the_folder() -> None:
"""The deterministic E2E mock keys on the ``FOLDER_SUMMARY_MODE``
marker in the system prompt and returns the canned one-liner naming
the folder from the ``Folder: …`` header — driven through the
GENERATOR's real prompt, so the two can never drift (the drill-down
E2E asserts on this exact template)."""
system, user = build_folder_summary_prompt(
"Homelab", "deployments/ansible",
[("Homelab", "deployments/ansible/lab-inventory.md", "Lab Inventory", None)],
)
assert compose_answer(_mock_body(system, user)) == (
"Fixture folder summary for Homelab/deployments/ansible."
)
# The source-root row names the source itself.
system, user = build_folder_summary_prompt("Homelab", "",
[("Homelab", "top.md", "Top", None)])
assert compose_answer(_mock_body(system, user)) == (
"Fixture folder summary for Homelab."
)
def test_mock_folder_marker_is_not_shadowed_by_the_summary_branch() -> None:
"""``FOLDER_SUMMARY_MODE`` contains ``SUMMARY_MODE`` as a substring —
the mock must check the folder branch FIRST, or every folder call
would land in the document-summary digest (regression pin)."""
system, user = build_folder_summary_prompt(
"S", "a", [("S", "a/x.md", "X", None)]
)
assert "SUMMARY_MODE" in system # the shadowing hazard is real
answer = compose_answer(_mock_body(system, user))
assert answer == "Fixture folder summary for S/a."
assert not answer.startswith("This document covers")
def test_summarize_folder_happy_path_returns_trimmed_text() -> None:
docs = [("S", "a/x.md", "X", None)]
llm = _FakeLLM(reply=f" {REPLY} \n")
out = asyncio.run(summarize_folder("S", "a", docs, llm))
assert out == REPLY # the model's text, trimmed
assert llm.calls == 1
def test_summarize_folder_calls_the_configured_summary_model_with_marker() -> None:
docs = [("S", "a/x.md", "X", "X lead.")]
llm = _FakeLLM()
asyncio.run(summarize_folder("S", "a", docs, llm))
assert llm.model == llm.settings.llm_summary_model # the ``lite`` default
assert llm.model == "lite"
system, user = llm.requests[0]
assert FOLDER_SUMMARY_MODE in system
assert user.startswith(FOLDER_HEADER_PREFIX + "S/a")
assert "a/x.md — X — X lead." in user
def test_summarize_folder_empty_reply_raises_llm_error() -> None:
docs = [("S", "a/x.md", "X", None)]
for reply in ("", " \n\t "):
llm = _FakeLLM(reply=reply)
with pytest.raises(LLMError, match="empty content for S/a"):
asyncio.run(summarize_folder("S", "a", docs, llm))
def test_summarize_folder_error_propagates() -> None:
docs = [("S", "a/x.md", "X", None)]
llm = _FakeLLM(fail=LLMError("simulated transport failure"))
with pytest.raises(LLMError, match="simulated transport failure"):
asyncio.run(summarize_folder("S", "a", docs, llm))
# ---------- generate_folder_summaries (real Postgres) ----------
def _add_doc(
db: Session, source: str, path: str, title: str, summary: str | None = None
) -> Document:
doc = Document(
id=uuid.uuid4(),
source=source,
path=path,
full_path=f"/tmp/{path}",
title=title,
content="body",
content_hash="0" * 64,
summary=summary,
)
db.add(doc)
db.commit()
return doc
def _truncate(db: Session) -> None:
db.execute(text("TRUNCATE chunks, documents"))
db.execute(text("DELETE FROM folder_summaries"))
db.commit()
def _rows(db: Session) -> dict[tuple[str, str], str]:
"""The stored folder summaries: ``{(source, folder_path): summary}``."""
result = db.execute(
text("SELECT source, folder_path, summary FROM folder_summaries")
).all()
return {(source, folder_path): summary for source, folder_path, summary in result}
def _seed_catalogue(db: Session) -> None:
"""The shared catalogue: FSU has four docs in three candidate
folders (root 4, a 3, a/b 2 — all ≥ the minimum); FSU-solo has one
doc (its root folder is below the minimum — no row, no call)."""
_add_doc(db, "FSU", "a/b/one.md", "One", "One lead.\nSource: FSU/a/b/one.md")
_add_doc(db, "FSU", "a/b/two.md", "Two")
_add_doc(db, "FSU", "a/three.md", "Three")
_add_doc(db, "FSU", "root.md", "Root")
_add_doc(db, "FSU-solo", "solo.md", "Solo")
@pytest.fixture()
def clean_tables(db: Session):
_truncate(db)
yield
_truncate(db)
def test_generate_happy_path_upserts_every_candidate_folder(
db: Session, clean_tables, caplog: pytest.LogCaptureFixture
) -> None:
"""Every folder with ≥ 2 recursive docs gets a row (the source root
row included — ``folder_path = ''``); single-doc folders get none;
rows are stamped fresh; the stats dict and the log line are right;
folders are processed in deterministic (source, folder_path) order."""
_seed_catalogue(db)
llm = _FakeLLM()
with caplog.at_level(logging.INFO, logger="app.rag.folder_summaries"):
stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats == {"generated": 3, "failed": 0, "pruned": 0}
assert llm.calls == 3, "one lite call per candidate folder (the solo folder: none)"
stored = _rows(db)
assert set(stored) == {("FSU", ""), ("FSU", "a"), ("FSU", "a/b")}
assert all(summary == REPLY for summary in stored.values())
assert ("FSU-solo", "") not in stored, (
"a single-doc folder is fully described by its one file line — no row"
)
row = db.get(FolderSummary, ("FSU", "a/b"))
assert row is not None
assert row.summary == REPLY
assert row.updated_at is not None
age = datetime.now(UTC) - row.updated_at
assert age.total_seconds() < 300, "updated_at must be a fresh UTC timestamp"
# Deterministic (source, folder_path) order — root before the
# nested folders, one header per call.
assert [user.splitlines()[0] for _s, user in llm.requests] == [
"Folder: FSU",
"Folder: FSU/a",
"Folder: FSU/a/b",
]
# The recursive-subtree input: the a/ prompt carries a/b's docs too.
a_prompt = llm.requests[1][1]
assert "a/b/one.md — One — One lead." in a_prompt
assert "a/three.md — Three" in a_prompt
assert "root.md — Root" not in a_prompt
assert (
"folder_summaries: generated=3 failed=0 pruned=0" in caplog.text
), "the stats line must be greppable (PLAN §9 ample logging)"
def test_generate_per_folder_fail_soft_keeps_previous_and_lands_others(
db: Session, clean_tables, caplog: pytest.LogCaptureFixture
) -> None:
"""One folder's lite failure is logged and counted, its PREVIOUS
row is kept (an old summary is better than none), and the remaining
folders still land — a lite outage never fails the sync."""
_seed_catalogue(db)
db.add(FolderSummary(source="FSU", folder_path="a/b", summary="old summary"))
db.commit()
llm = _FakeLLM(fail_folders=("FSU/a/b",))
with caplog.at_level(logging.ERROR, logger="app.rag.folder_summaries"):
stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats == {"generated": 2, "failed": 1, "pruned": 0}
assert llm.calls == 3 # the failing folder was attempted too
stored = _rows(db)
assert stored[("FSU", "a/b")] == "old summary", (
"the previous row survives the per-folder failure"
)
assert stored[("FSU", "")] == REPLY and stored[("FSU", "a")] == REPLY, (
"the other folders still land"
)
assert "folder summary failed for FSU/a/b" in caplog.text
assert "simulated lite-model failure for FSU/a/b" in caplog.text
def test_generate_per_folder_fail_soft_without_previous_row_creates_nothing(
db: Session, clean_tables
) -> None:
_seed_catalogue(db)
llm = _FakeLLM(fail_folders=("FSU/a/b",))
stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats["failed"] == 1
stored = _rows(db)
assert ("FSU", "a/b") not in stored, "no row must be invented for a failed folder"
assert ("FSU", "") in stored and ("FSU", "a") in stored
def test_generate_prunes_stale_rows_and_keeps_live_ones(db: Session, clean_tables) -> None:
"""Rows for folders that dropped below 2 recursive docs are deleted
(pruned/renamed — the summary would go stale); rows for folders
that still qualify persist (an unchanged folder's summary is still
true — regenerated in place)."""
_seed_catalogue(db)
# A stale row for a folder no longer in the catalogue (3→1 docs /
# renamed away) + a live row with old content.
db.add(FolderSummary(source="FSU", folder_path="gone/old", summary="stale"))
db.add(FolderSummary(source="FSU", folder_path="a", summary="old a summary"))
db.add(FolderSummary(source="FSU-solo", folder_path="", summary="solo stale"))
db.commit()
stats = asyncio.run(generate_folder_summaries(db, _FakeLLM()))
assert stats["pruned"] == 2 # gone/old + the FSU-solo root (1 doc)
stored = _rows(db)
assert ("FSU", "gone/old") not in stored, "the stale folder row must be pruned"
assert ("FSU-solo", "") not in stored, (
"a folder that dropped below 2 docs loses its row"
)
assert ("FSU", "a") in stored, "the still-qualifying folder keeps its row"
assert stored[("FSU", "a")] == REPLY # regenerated, not stale
assert stored[("FSU", "")] == REPLY and stored[("FSU", "a/b")] == REPLY
def test_generate_skip_is_a_full_noop(db: Session, clean_tables) -> None:
"""``skip=True`` (the ``--limit`` debug run): the LLM is never
called, no rows are touched, zero stats."""
_seed_catalogue(db)
db.add(FolderSummary(source="FSU", folder_path="", summary="existing"))
db.commit()
llm = _FakeLLM()
stats = asyncio.run(generate_folder_summaries(db, llm, skip=True))
assert stats == {"generated": 0, "failed": 0, "pruned": 0}
assert llm.calls == 0
assert _rows(db) == {("FSU", ""): "existing"}
def test_generate_empty_kb_prunes_every_row(db: Session, clean_tables) -> None:
"""No documents → no candidate folders → every stored row is
pruned, with zero wasted lite calls."""
db.add(FolderSummary(source="FSU", folder_path="", summary="old"))
db.add(FolderSummary(source="FSU", folder_path="a/b", summary="old"))
db.commit()
llm = _FakeLLM()
stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats == {"generated": 0, "failed": 0, "pruned": 2}
assert llm.calls == 0
assert _rows(db) == {}
def test_generate_summarizes_folder_whose_prefix_is_also_a_doc_path(
db: Session, clean_tables
) -> None:
"""The ``path == folder`` arm end to end: a file sharing its name
with a directory counts toward the folder's recursive count (2
docs → the folder is summarized, and BOTH docs are in its prompt).
"""
_add_doc(db, "FSU", "a/b", "B") # a file named "b" (its path is a prefix)
_add_doc(db, "FSU", "a/b/c.md", "C") # and a real folder "a/b/"
llm = _FakeLLM()
stats = asyncio.run(generate_folder_summaries(db, llm))
assert stats["generated"] == 3 # root (2), a (2), a/b (2) — all ≥ the minimum
stored = _rows(db)
assert set(stored) == {("FSU", ""), ("FSU", "a"), ("FSU", "a/b")}
a_b_prompt = [
user for _system, user in llm.requests if user.startswith("Folder: FSU/a/b\n")
][0]
assert "a/b — B" in a_b_prompt
assert "a/b/c.md — C" in a_b_prompt
def test_generate_only_flushes_caller_commits(db: Session, clean_tables) -> None:
"""The generator only flushes — the sync path owns the transaction
(the phase-53 ``bump_sources_version`` convention): the catalogue
is committed (the real sync path commits the import before the
summary hooks run), but a second session sees the generator's rows
as NOTHING until the CALLER commits — and sees them after."""
_add_doc(db, "FSU", "x/y/one.md", "One")
_add_doc(db, "FSU", "x/y/two.md", "Two")
stats = asyncio.run(generate_folder_summaries(db, _FakeLLM()))
assert stats["generated"] == 3 # root + x + x/y — all 2 recursive docs
with SessionLocal() as other:
n = other.scalar(
text("SELECT count(*) FROM folder_summaries WHERE source = 'FSU'")
)
assert n == 0, "unflushed-by-caller rows must not be visible yet"
db.commit()
with SessionLocal() as other:
n = other.scalar(
text("SELECT count(*) FROM folder_summaries WHERE source = 'FSU'")
)
assert n == 3, "the caller's commit makes the flushed rows durable"
assert MIN_DOCS_PER_FOLDER == 2 # the ≥ 2 scope rule, pinned by name
def test_folder_summary_table_empty_gate(db: Session, clean_tables) -> None:
"""The sync-path gate probe (phase 94, task 02): empty → True
(the first full sync after migration 0017 must still generate),
one row → False (a populated table waits for a KB change)."""
assert folder_summary_table_empty(db) is True # the truncated table
_add_doc(db, "FSU", "a/one.md", "One")
_add_doc(db, "FSU", "a/two.md", "Two")
asyncio.run(generate_folder_summaries(db, _FakeLLM()))
db.commit()
assert folder_summary_table_empty(db) is False # rows landed
db.execute(text("DELETE FROM folder_summaries"))
db.commit()
assert folder_summary_table_empty(db) is True # emptied again
tests/unit/test_mock_tool_flow.py
+124 -44
View File
@@ -4,12 +4,18 @@ The mock (``tests/e2e/mock_llm.py``) classifies marker requests
statelessly into one step of the agent tool flow. This file pins the
classification at unit speed — no Playwright, no LLM process:
* the phase-37 SINGLE-READ flow (``TOOLS_TRIGGER`` only) stays
byte-identical: list → read (first catalog line, ``call_1``) → answer;
* the phase-45 MULTI-READ flow (``TOOLS_TRIGGER`` + ``MULTI_READ_TRIGGER``)
classifies by the count of ``tool``-role read results: list → read #1
(``call_1``) → read #2 (second catalog line, ``call_2``) → the
byte-stable ``multi_answer`` naming both read paths.
* the phase-37/94 SINGLE-READ flow (``TOOLS_TRIGGER`` only): list (the
top-level source listing) → drill (``ls`` scoped to the first source
— phase 94: the top level carries sources only, so the flow drills
one level for the file lines) → read (first file line, the combined
``source/path``) → answer;
* the phase-45 MULTI-READ flow (``TOOLS_TRIGGER`` +
``MULTI_READ_TRIGGER``) classifies by the count of ``tool``-role read
results: list → drill → read #1 (first file line) → read #2 (second
file line) → the byte-stable ``multi_answer`` naming both read paths;
* the degenerate empty-KB case: every listed source already drilled
with no file lines → the flow falls back to the re-list loop (the
round cap settles it — the phase-70 empty-catalog behavior).
"""
from __future__ import annotations
@@ -36,28 +42,50 @@ SYSTEM_LOW = "<relevance>LOW</relevance>\n"
#: the phase-70 harness-aligned names).
TOOLS = [{"type": "function", "function": {"name": "ls"}}]
#: The agent's ``ls`` output for a two-document KB
#: (``app/rag/agent.py`` ``_execute_tool``): one
#: ``source: X | path: Y | title: Z`` line per document (phase 63: labeled,
#: unambiguous fields), ``(source, path)`` order.
CATALOG_2 = (
"2 documents:\n"
"source: Deployments | path: example-record-file.json | title: Example Record File\n"
"source: Homelab | path: aws-route53.md | title: AWS Route 53 Notes"
#: The agent's drill-down ``ls`` output for a two-source KB
#: (``app/rag/agent.py`` ``render_ls_top`` / ``render_folder_listing``,
#: phase 94): the top level lists the registered sources (registry
#: order, recursive counts — no file lines); the folder level carries
#: the file lines (``source: X | path: Y | title: Z`` — the phase-63
#: labeled fields, unchanged), ``path`` order.
TOP_LEVEL_2 = (
"2 sources:\n"
"\n"
"Deployments — 1 documents\n"
"Homelab — 1 documents"
)
CATALOG_1 = (
"1 documents:\n"
#: The first source's root folder: one file line (the single-read
#: flow's read target).
FOLDER_DEPLOYMENTS = (
"Deployments — 1 documents, 0 folders:\n"
"\n"
"source: Deployments | path: example-record-file.json | title: Example Record File"
)
CATALOG_3 = (
"3 documents:\n"
#: Two file lines in the first source (the multi-read flow's reads).
FOLDER_DEPLOYMENTS_2 = (
"Deployments — 2 documents, 0 folders:\n"
"\n"
"source: Deployments | path: aaa.md | title: AAA\n"
"source: Deployments | path: bbb.md | title: BBB"
)
#: Three file lines in the first source (the listing-order pin: read
#: #2 is the SECOND line, not the last).
FOLDER_DEPLOYMENTS_3 = (
"Deployments — 3 documents, 0 folders:\n"
"\n"
"source: Deployments | path: aaa.md | title: AAA\n"
"source: Deployments | path: bbb.md | title: BBB\n"
"source: Homelab | path: ccc.md | title: CCC"
"source: Deployments | path: ccc.md | title: CCC"
)
#: Empty folder levels (a registered source with no documents — the
#: header line alone; the drill's degenerate arm).
FOLDER_DEPLOYMENTS_EMPTY = "Deployments — 0 documents, 0 folders:"
FOLDER_HOMELAB_EMPTY = "Homelab — 0 documents, 0 folders:"
DOC1_SP = "Deployments/example-record-file.json"
DOC1_CONTENT = (
"The record file keeps every hosted zone record — first line is longer "
@@ -122,10 +150,39 @@ def test_single_flow_list_step() -> None:
assert _tool_flow(_body(SINGLE_USER)) == ("list", "", "")
def test_single_flow_read_step_first_catalog_line() -> None:
flow = _tool_flow(_body(SINGLE_USER, (CATALOG_3,)))
# The FIRST listing line (Deployments/aaa.md), labeled fields.
assert flow == ("read", "Deployments", "aaa.md", "call_1")
def test_single_flow_drill_step_after_top_level() -> None:
# Phase 94: the top level lists SOURCES only — the flow drills one
# level into the FIRST source (listing order = registry order).
flow = _tool_flow(_body(SINGLE_USER, (TOP_LEVEL_2,)))
assert flow == ("drill", "Deployments", "call_1")
def test_single_flow_drill_skips_already_drilled_source() -> None:
# The first source's folder level is already in the messages (an
# empty listing — header only, no file lines): the drill proceeds
# to the NEXT un-drilled source.
flow = _tool_flow(_body(SINGLE_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS_EMPTY)))
assert flow == ("drill", "Homelab", "call_1")
def test_single_flow_all_sources_drilled_empty_falls_back_to_list() -> None:
# Degenerate: every listed source already drilled, no file lines
# anywhere — the flow falls back to the re-list loop (settled at the
# round cap, the phase-70 empty-catalog behavior).
flow = _tool_flow(
_body(
SINGLE_USER,
(TOP_LEVEL_2, FOLDER_DEPLOYMENTS_EMPTY, FOLDER_HOMELAB_EMPTY),
)
)
assert flow == ("list", "", "")
def test_single_flow_read_step_first_file_line() -> None:
# The folder level reached: the FIRST file line (Deployments/aaa.md),
# labeled fields, the combined ``source/path`` join.
flow = _tool_flow(_body(SINGLE_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS_2)))
assert flow == ("read", "Deployments", "aaa.md", "call_2")
def test_read_step_nested_path_stays_intact() -> None:
@@ -133,19 +190,20 @@ def test_read_step_nested_path_stays_intact() -> None:
# ``source/path — title`` + ``rpartition("/")`` parse misread the
# split (``source=brain-of-reese-main/homelab``). The labeled fields
# recover the nested path intact, however deep.
catalog = (
"1 documents:\n"
listing = (
"brain-of-reese-main — 1 documents, 0 folders:\n"
"\n"
"source: brain-of-reese-main | path: homelab/aws-route53.md | title: aws-route53"
)
flow = _tool_flow(_body(SINGLE_USER, (catalog,)))
assert flow == ("read", "brain-of-reese-main", "homelab/aws-route53.md", "call_1")
flow = _tool_flow(_body(SINGLE_USER, (TOP_LEVEL_2, listing)))
assert flow == ("read", "brain-of-reese-main", "homelab/aws-route53.md", "call_2")
def test_single_flow_answer_step_with_tools_offered() -> None:
# Phase 45: the round cap keeps the tools offered until it is hit —
# the answer step fires regardless of the ``tools`` parameter.
flow = _tool_flow(
_body(SINGLE_USER, (CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT)))
_body(SINGLE_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS, _read_result(DOC1_SP, DOC1_CONTENT)))
)
assert flow == ("answer", DOC1_SP, DOC1_CONTENT)
@@ -154,7 +212,7 @@ def test_single_flow_answer_step_without_tools() -> None:
flow = _tool_flow(
_body(
SINGLE_USER,
(CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT)),
(TOP_LEVEL_2, FOLDER_DEPLOYMENTS, _read_result(DOC1_SP, DOC1_CONTENT)),
tools=None,
)
)
@@ -184,22 +242,39 @@ def test_multi_flow_list_step() -> None:
assert _tool_flow(_body(MULTI_USER)) == ("list", "", "")
def test_multi_flow_drill_step_after_top_level() -> None:
# Phase 94: the top level lists SOURCES only — the multi flow drills
# too, before its first read.
flow = _tool_flow(_body(MULTI_USER, (TOP_LEVEL_2,)))
assert flow == ("drill", "Deployments", "call_1")
def test_multi_flow_read_first_step() -> None:
flow = _tool_flow(_body(MULTI_USER, (CATALOG_2,)))
assert flow == ("read", DOC1_SP.split("/", 1)[0], DOC1_SP.rsplit("/", 1)[1], "call_1")
flow = _tool_flow(_body(MULTI_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS)))
assert flow == ("read", DOC1_SP.split("/", 1)[0], DOC1_SP.rsplit("/", 1)[1], "call_2")
def test_multi_flow_read_second_step_skips_already_read() -> None:
flow = _tool_flow(_body(MULTI_USER, (CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT))))
# The second catalog line — the first line differing from DOC1.
assert flow == ("read", "Homelab", "aws-route53.md", "call_2")
flow = _tool_flow(
_body(
MULTI_USER,
(TOP_LEVEL_2, FOLDER_DEPLOYMENTS_2, _read_result("Deployments/aaa.md", DOC1_CONTENT)),
)
)
# The second file line — the first line differing from the read doc.
assert flow == ("read", "Deployments", "bbb.md", "call_3")
def test_multi_flow_read_second_is_listing_order_not_last() -> None:
# Three-doc catalog, first doc read: read #2 is the SECOND line
# Three-file listing, first file read: read #2 is the SECOND line
# (Deployments/bbb.md), not the last one.
flow = _tool_flow(_body(MULTI_USER, (CATALOG_3, _read_result("Deployments/aaa.md", "x"))))
assert flow == ("read", "Deployments", "bbb.md", "call_2")
flow = _tool_flow(
_body(
MULTI_USER,
(TOP_LEVEL_2, FOLDER_DEPLOYMENTS_3, _read_result("Deployments/aaa.md", "x")),
)
)
assert flow == ("read", "Deployments", "bbb.md", "call_3")
def test_multi_flow_answer_step_names_both_paths() -> None:
@@ -207,7 +282,8 @@ def test_multi_flow_answer_step_names_both_paths() -> None:
_body(
MULTI_USER,
(
CATALOG_2,
TOP_LEVEL_2,
FOLDER_DEPLOYMENTS_2,
_read_result(DOC1_SP, DOC1_CONTENT),
_read_result(DOC2_SP, DOC2_CONTENT),
),
@@ -227,7 +303,8 @@ def test_multi_flow_answer_step_without_tools_offered() -> None:
_body(
MULTI_USER,
(
CATALOG_2,
TOP_LEVEL_2,
FOLDER_DEPLOYMENTS_2,
_read_result(DOC1_SP, DOC1_CONTENT),
_read_result(DOC2_SP, DOC2_CONTENT),
),
@@ -238,11 +315,11 @@ def test_multi_flow_answer_step_without_tools_offered() -> None:
assert flow[0] == "multi_answer"
def test_multi_flow_one_document_catalog_degenerates_to_single_answer() -> None:
def test_multi_flow_one_file_listing_degenerates_to_single_answer() -> None:
# Nothing second to read — the single-read answer shape, quoting the
# only read result.
flow = _tool_flow(
_body(MULTI_USER, (CATALOG_1, _read_result(DOC1_SP, DOC1_CONTENT)))
_body(MULTI_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS, _read_result(DOC1_SP, DOC1_CONTENT)))
)
assert flow == ("answer", DOC1_SP, DOC1_CONTENT)
@@ -320,9 +397,12 @@ def test_search_flow_found_step_without_tools_offered() -> None:
def test_search_flow_ignores_catalog_and_read_results() -> None:
# A catalog (labeled lines) and a read result ("Document …" prefix)
# are NOT search results — the flow stays at the search step.
flow = _search_flow(_body(SEARCH_USER, (CATALOG_2, _read_result(DOC1_SP, DOC1_CONTENT))))
# Listings (top-level + folder level, labeled file lines) and a read
# result ("Document …" prefix) are NOT search results — the flow
# stays at the search step.
flow = _search_flow(
_body(SEARCH_USER, (TOP_LEVEL_2, FOLDER_DEPLOYMENTS, _read_result(DOC1_SP, DOC1_CONTENT)))
)
assert flow == ("search",)
tests/unit/test_prompts.py
+30 -22
View File
@@ -11,11 +11,15 @@ And the phase-71 deflection plain-text line (owner-permitted
line; the ``DEFLECT_MODE`` marker-keying contract is unchanged and
the line never leaks into the HIGH prompt.
And the phase-72 ``<tools>`` copy: the document-identity contract is
stated up front (the ``ls`` source-name scope, the combined
``source/path`` identity for ``read``/``grep``) — the same contract
the teaching refusals in :mod:`app.rag.agent` re-state; the
``<tools>`` marker keying (HIGH only) is unchanged.
And the ``<tools>`` copy (phase 72: the document-identity contract
stated up front — the combined ``source/path`` identity for
``read``/``grep``; phase 94, task 03: the ``ls`` clause rewritten to
the drill-down tree contract — one level per call, sources at the
top, folders + files below, ``grep`` as the without-listing locator —
while the ``read``/``grep`` clauses and the discipline rules are
byte-identical): the teaching refusals in :mod:`app.rag.agent`
re-state the same contract; the ``<tools>`` marker keying (HIGH
only) is unchanged.
"""
from __future__ import annotations
@@ -194,20 +198,22 @@ pinned byte-for-byte in
def test_tools_section_phase72_contract_clauses() -> None:
"""Phase 72: the two contract clauses the teaching refusals
"""Phase 72 + phase 94: the contract clauses the teaching refusals
re-state after the fact, pinned byte-for-byte in the constant —
the ``ls`` source-name clause (its optional ``path`` is a source
name, not a directory or file path; omit it to list every
document) and the ``read``/``grep`` combined-identity clause
(the combined ``source/path`` string exactly as shown in the
``ls`` output, *including the source name*; a bare document path
will not resolve)."""
# The ls source-name clause.
assert (
"a source name (e.g. 'homelab'), not a directory or file "
"path — omit it to list every document"
) in TOOLS_SECTION
# The read combined-identity clause.
the ``ls`` clause (phase 94: the drill-down tree contract — one
level per call, sources at the top, folders + files below, never
the whole KB in one call, ``grep`` as the without-listing locator)
and the ``read``/``grep`` combined-identity clause (the combined
``source/path`` string exactly as shown in the ``ls`` output,
*including the source name*; a bare document path will not
resolve)."""
# The ls drill-down clauses (phase 94, task 03).
assert "one level at a time" in TOOLS_SECTION
assert "lists every synced source with its document count" in TOOLS_SECTION
assert "that source's top-level folders and files" in TOOLS_SECTION
assert "never the whole knowledge base in one call" in TOOLS_SECTION
assert "to find one specific document without listing, use `grep`" in TOOLS_SECTION
# The read combined-identity clause (byte-identical across phases).
assert (
"combined `source/path` string, exactly as shown in the `ls` "
"output — including the source name"
@@ -217,23 +223,25 @@ def test_tools_section_phase72_contract_clauses() -> None:
"a bare document path (without the source name) will not resolve"
) == 2
# The pre-phase-70 scope wording is gone — replaced by the
# explicit source-name contract.
# explicit source-name contract (and the phase-72 source-name-only
# clause by the phase-94 drill-down contract).
assert "pass a source name as `path`" not in TOOLS_SECTION
assert "not a directory or file path" not in TOOLS_SECTION
def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
"""Phase 72: the contract clauses ride the HIGH prompt with the
"""Phase 72/94: the contract clauses ride the HIGH prompt with the
rest of the section and never leak into the LOW/deflection prompt
(whose byte-identity is pinned in
:func:`test_zero_note_prompt_is_byte_identical_to_pre_steering`)."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
high = build_high_prompt([doc])
assert "<tools>" in high
assert "not a directory or file path" in high
assert "one level at a time" in high # the phase-94 ls clause
assert "including the source name" in high
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
assert "<tools>" not in low
assert "not a directory or file path" not in low
assert "one level at a time" not in low
assert "including the source name" not in low
tests/unit/test_sync_button.py
+10 -2
View File
@@ -752,8 +752,9 @@ def _patch_sync_seams(
"""The runner's seams, monkeypatched on ``app.api.sync`` (the house
mock-import pattern): fresh settings (no ``.env`` leak), a no-op
model probe, a sentinel LLM client, one git row, the gated clone +
import, a no-op overview, and the DB-free sources-version step
(dummy session + pinned counters)."""
import, a no-op overview, a no-op folder-summary step (phase 94 —
the sentinel LLM client has no ``chat``), and the DB-free
sources-version step (dummy session + pinned counters)."""
monkeypatch.setattr(
sync_api,
"get_settings",
@@ -783,6 +784,13 @@ def _patch_sync_seams(
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
async def fake_folder_summaries(
db: object, llm: object, *, skip: bool = False
) -> dict[str, int]:
return {"generated": 0, "failed": 0, "pruned": 0}
monkeypatch.setattr(sync_api, "generate_folder_summaries", fake_folder_summaries)
class _DummySession:
def close(self) -> None:
pass