phase: 94_ls_tree_drilldown
All green. Verification complete. **Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)** - Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal - Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths - Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met - `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %) - `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings - E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched) - Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed - Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol) **Next pending phase:** `95_read_truncation_cap`
This commit is contained in:
@@ -11,19 +11,26 @@ message contains BOTH ``use your tools`` (``TOOLS_TRIGGER``) and ``read
|
||||
two documents`` (``MULTI_READ_TRIGGER``) **and** the system prompt
|
||||
carries the ``<tools>`` section of the HIGH prompt; phase 70: the flow
|
||||
emits the harness-aligned names — ``ls``, then ``read`` on the JOINED
|
||||
combined ``source/path`` of each catalog line):
|
||||
combined ``source/path`` of each file line; phase 94: the drill-down
|
||||
``ls`` — the top level lists sources only, so the flow drills one
|
||||
level into the first source before the first file line exists):
|
||||
|
||||
1. request 1 (``tools`` offered, no tool results yet) → streams ONLY
|
||||
``tool_calls`` deltas calling ``ls`` (id ``call_0``);
|
||||
2. request 2 (the ``tool``-role catalog result) → ``read`` on the
|
||||
JOINED combined ``source/path`` of the FIRST catalog line
|
||||
(id ``call_1``);
|
||||
3. request 3 (one ``tool``-role read result) → ``read`` on the JOINED
|
||||
combined ``source/path`` of the SECOND catalog line (id ``call_2``)
|
||||
— the pre-phase-45 per-tool budgets would have refused exactly this
|
||||
2. request 2 (the top-level source listing in the messages — no file
|
||||
lines yet) → the drill: ``ls`` scoped to the FIRST source of the
|
||||
listing (id ``call_1``); the seed registers ``Deployments`` first,
|
||||
and both read documents live in it — so the drill's folder listing
|
||||
carries BOTH file lines;
|
||||
3. request 3 (a ``tool``-role folder listing with file lines) →
|
||||
``read`` on the JOINED combined ``source/path`` of the FIRST file
|
||||
line (id ``call_2``);
|
||||
4. request 4 (one ``tool``-role read result) → ``read`` on the JOINED
|
||||
combined ``source/path`` of the SECOND file line (id ``call_3``) —
|
||||
the pre-phase-45 per-tool budgets would have refused exactly this
|
||||
second read (``No reading budget left — answer with what you
|
||||
have.``);
|
||||
4. request 4 (two read results) → the forced answer, byte-stable: the
|
||||
5. request 5 (two read results) → the forced answer, byte-stable: the
|
||||
single-read shape quoting the FIRST read result, plus the line
|
||||
``I read <sp1> and <sp2>.`` naming both read paths in read order.
|
||||
|
||||
@@ -49,10 +56,11 @@ rejection, not the multi-read
|
||||
flow this story proves.
|
||||
|
||||
Test → story mapping (Playwright Mapping Rule):
|
||||
1. ``test_multi_read_turn`` — the turn streams THREE ``tool`` frames /
|
||||
``.tool-call`` lines in order (one list — "is listing documents" —
|
||||
and two reads — "is reading <source/path>" — the #send-status
|
||||
transition recorded deterministically via MutationObserver), then a
|
||||
1. ``test_multi_read_turn`` — the turn streams FOUR ``tool`` frames /
|
||||
``.tool-call`` lines in order (the top-level list, the drill list —
|
||||
phase 94 — "is listing documents", and two reads — "is reading
|
||||
<source/path>" — the #send-status transition recorded
|
||||
deterministically via MutationObserver), then a
|
||||
final non-deflected answer containing the mock's byte-stable
|
||||
``I read <sp1> and <sp2>.`` line; the round cap (default 10) bounds
|
||||
the turn, no budget refusal anywhere.
|
||||
@@ -84,7 +92,7 @@ from sqlalchemy import select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db import SessionLocal
|
||||
from app.models import Chunk, Document, QueryLog
|
||||
from app.models import Chunk, Document, GitSource, QueryLog
|
||||
from e2e.auth_helpers import login
|
||||
from tests.e2e.mock_llm import embed_text
|
||||
|
||||
@@ -209,7 +217,23 @@ def _doc(source: str, path: str, title: str, content: str) -> Document:
|
||||
|
||||
|
||||
def _seed(db: Session) -> None:
|
||||
"""The three-document KB from the module docstring."""
|
||||
"""The three-document KB from the module docstring.
|
||||
|
||||
Phase 94: the drill-down ``ls`` top level reads the registry —
|
||||
register BOTH sources (TRUNCATEd in ``_reset_db``), ``Deployments``
|
||||
FIRST (registry order is ``(added_at, id)``): the mock's drill
|
||||
(first source of the listing) lands on the folder that carries
|
||||
BOTH file lines. A non-empty table also ignores the operator's
|
||||
``BOR_GIT_SOURCES`` fallback — deterministic.
|
||||
"""
|
||||
# COMMIT between the inserts (not flush): ``added_at`` is
|
||||
# ``server_default now()`` — the transaction timestamp — and the
|
||||
# tie-break is the RANDOM uuid ``id``, so two rows in one
|
||||
# transaction order nondeterministically (the integration
|
||||
# ``registry`` fixture's pattern).
|
||||
db.add(GitSource(url=READ1_SOURCE, kind="local"))
|
||||
db.commit()
|
||||
db.add(GitSource(url=SEED_SOURCE, kind="local"))
|
||||
md = _doc(SEED_SOURCE, SEED_PATH, "AWS Route 53 Notes", ROUTE53_CONTENT)
|
||||
db.add(md)
|
||||
db.flush()
|
||||
@@ -240,7 +264,10 @@ def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
|
||||
"""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
if seed is not None:
|
||||
@@ -399,13 +426,16 @@ def test_multi_read_turn(
|
||||
_submit(page, MULTI_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# Wire level: exactly THREE `tool` frames — ls, read #1, read #2
|
||||
# (each read's argument is the JOINED combined source/path), in
|
||||
# order — and all ahead of the first `delta` frame. This third
|
||||
# frame is the one the pre-phase-45 read budget refused.
|
||||
# Wire level: exactly FOUR `tool` frames — the top-level ls, the
|
||||
# drill ls scoped to the first source (phase 94), then read #1 and
|
||||
# read #2 (each read's argument is the JOINED combined
|
||||
# source/path), in order — and all ahead of the first `delta`
|
||||
# frame. The fourth frame is the one the pre-phase-45 read budget
|
||||
# refused.
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "ls", "argument": READ1_SOURCE},
|
||||
{"type": "tool", "name": "read", "argument": READ1_SP},
|
||||
{"type": "tool", "name": "read", "argument": READ2_SP},
|
||||
]
|
||||
@@ -439,14 +469,17 @@ def test_multi_read_turn(
|
||||
), statuses
|
||||
assert i_list < i_read1 < i_read2, statuses
|
||||
|
||||
# Three visible tool lines, in order, above the answer.
|
||||
# Four visible tool lines, in order, above the answer (phase 94:
|
||||
# the drill line is "Listing documents in <source>").
|
||||
lines = page.locator(".msg.brain .tool-call")
|
||||
expect(lines).to_have_count(3)
|
||||
expect(lines).to_have_count(4)
|
||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
||||
expect(lines.nth(1)).to_contain_text(READ1_SP)
|
||||
expect(lines.nth(1)).to_contain_text("Listing documents in")
|
||||
expect(lines.nth(1)).to_contain_text(READ1_SOURCE)
|
||||
expect(lines.nth(2)).to_contain_text("Reading ")
|
||||
expect(lines.nth(2)).to_contain_text(READ2_SP)
|
||||
expect(lines.nth(2)).to_contain_text(READ1_SP)
|
||||
expect(lines.nth(3)).to_contain_text("Reading ")
|
||||
expect(lines.nth(3)).to_contain_text(READ2_SP)
|
||||
|
||||
# The final answer is non-deflected, quotes the FIRST read result,
|
||||
# and names BOTH read paths (the mock's byte-stable line).
|
||||
@@ -563,19 +596,23 @@ def test_single_tool_flow_regression(
|
||||
_submit(page, SINGLE_QUESTION)
|
||||
_wait_settled(page)
|
||||
|
||||
# Exactly TWO tool frames — ls then ONE read of the first catalog
|
||||
# line (the JOINED combined source/path) — no second read (the
|
||||
# marker carries no multi-read trigger).
|
||||
# Exactly THREE tool frames — the top-level ls, the drill ls
|
||||
# (phase 94), then ONE read of the first file line (the JOINED
|
||||
# combined source/path) — no second read (the marker carries no
|
||||
# multi-read trigger).
|
||||
frames = _frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "ls", "argument": READ1_SOURCE},
|
||||
{"type": "tool", "name": "read", "argument": READ1_SP},
|
||||
]
|
||||
lines = page.locator(".msg.brain .tool-call")
|
||||
expect(lines).to_have_count(2)
|
||||
expect(lines).to_have_count(3)
|
||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
||||
expect(lines.nth(1)).to_contain_text(READ1_SP)
|
||||
expect(lines.nth(1)).to_contain_text("Listing documents in")
|
||||
expect(lines.nth(1)).to_contain_text(READ1_SOURCE)
|
||||
expect(lines.nth(2)).to_contain_text("Reading ")
|
||||
expect(lines.nth(2)).to_contain_text(READ1_SP)
|
||||
|
||||
# The single-read answer shape: quotes the read document; it does
|
||||
# NOT carry the multi-read both-named line (READ2 was never read).
|
||||
|
||||
Reference in New Issue
Block a user