phase: 94_ls_tree_drilldown
All green. Verification complete. **Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)** - Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal - Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths - Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met - `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %) - `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings - E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched) - Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed - Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol) **Next pending phase:** `95_read_truncation_cap`
This commit is contained in:
@@ -13,11 +13,14 @@ Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the gate is the
|
||||
deterministic LS-TEACH flow in ``tests/e2e/mock_llm.py``
|
||||
(``LS_TEACH_TRIGGER`` — "list the files in this directory" — + the
|
||||
HIGH prompt's ``<tools>`` section): the incident's misuse (``ls`` with
|
||||
``{"path": "."}``, id ``call_0``) → the agent's teaching refusal
|
||||
(``No source named '.' — check the ls output. (…)``) → the corrected
|
||||
no-arg ``ls()`` (id ``call_1``) → the deterministic
|
||||
``These are the indexed documents: <first catalog line>`` answer.
|
||||
HIGH prompt's ``<tools>`` section; phase 94: the drill-down ``ls`` —
|
||||
the corrected no-arg listing carries sources only, so the flow drills
|
||||
one level before the first file line exists): the incident's misuse
|
||||
(``ls`` with ``{"path": "."}``, id ``call_0``) → the agent's teaching
|
||||
refusal (``No source named '.' — check the ls output. (…)``) → the
|
||||
corrected no-arg ``ls()`` (id ``call_1``) → the drill ``ls`` scoped to
|
||||
the first source of the listing (id ``call_2``) → the deterministic
|
||||
``These are the indexed documents: <first file line>`` answer.
|
||||
|
||||
KB fixture (TRUNCATE-then-seed, house pattern): ONE source with TWO
|
||||
documents of known ``source``/``path``/``title`` (catalog order =
|
||||
@@ -39,21 +42,24 @@ documents of known ``source``/``path``/``title`` (catalog order =
|
||||
Test → phase mapping (Playwright Mapping Rule):
|
||||
1. ``test_ls_misuse_self_corrects_to_noarg_listing`` — the grounded
|
||||
LS-TEACH turn: the turn settles (composer re-enables, ``done``
|
||||
observed), the answer bubble carries the first catalog line — the
|
||||
observed), the answer bubble carries the first file line — the
|
||||
first document's ``source:`` / ``path:`` / title fields (the
|
||||
catalog reached the model and landed in the answer), the UI shows
|
||||
the two tool lines (``🔎 Listing documents in <code>.</code>``
|
||||
then ``🔎 Listing documents``), and no error banner. Wire level:
|
||||
the ``tool`` frames arrive in order — first ``ls`` with
|
||||
``argument: "."``, then ``ls`` with ``argument: null`` — and there
|
||||
is NO third ``tool`` frame (the loop ended in one correction, not
|
||||
folder listing reached the model and landed in the answer), the UI
|
||||
shows the three tool lines (``🔎 Listing documents in
|
||||
<code>.</code>``, ``🔎 Listing documents``, then the drill
|
||||
``🔎 Listing documents in <source>`` — phase 94), and no error
|
||||
banner. Wire level: the ``tool`` frames arrive in order — first
|
||||
``ls`` with ``argument: "."``, then ``ls`` with ``argument: null``,
|
||||
then the drill ``ls`` scoped to the source — and there is NO fourth
|
||||
``tool`` frame (the loop ended in one correction + one drill, not
|
||||
at the round cap).
|
||||
2. ``test_plain_tool_flow_not_swallowed_by_new_trigger`` — in the SAME
|
||||
session, the LS-TEACH turn settles and a follow-up question
|
||||
carrying ``TOOLS_TRIGGER`` (the single-read flow) still settles
|
||||
with the read flow's answer (``ls`` → ``read`` on the first
|
||||
catalog line's combined identity → ``Read <source/path>. <quote>``)
|
||||
— the new flow did not swallow the existing trigger.
|
||||
with the read flow's answer (``ls`` → the drill ``ls`` (phase 94)
|
||||
→ ``read`` on the first file line's combined identity →
|
||||
``Read <source/path>. <quote>``) — the new flow did not swallow
|
||||
the existing trigger.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -69,7 +75,7 @@ from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.db import SessionLocal
|
||||
from app.models import Chunk, Document
|
||||
from app.models import Chunk, Document, GitSource
|
||||
from e2e.auth_helpers import login
|
||||
from tests.e2e.mock_llm import (
|
||||
LS_TEACH_TRIGGER,
|
||||
@@ -211,7 +217,14 @@ def _seed_fixture(db: Session) -> None:
|
||||
cosines well past the E2E 0.30 threshold and FTS-matches too →
|
||||
grounded). DOC2 is the seed context only — the single-read flow
|
||||
reads the catalog-FIRST document (DOC1), which is not in context.
|
||||
|
||||
Phase 94: the drill-down ``ls`` top level reads the registry —
|
||||
register the source (TRUNCATEd in ``_reset_db_fixture``): the
|
||||
corrected no-arg listing names it, and the drill scopes to it. A
|
||||
non-empty table also ignores the operator's ``BOR_GIT_SOURCES``
|
||||
fallback — deterministic.
|
||||
"""
|
||||
db.add(GitSource(url=SEED_SOURCE, kind="local"))
|
||||
db.add(
|
||||
Document(
|
||||
source=SEED_SOURCE,
|
||||
@@ -255,7 +268,10 @@ def _reset_db_fixture() -> None:
|
||||
prompts, byte-stable answers."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text("TRUNCATE chunks, documents, query_log, steering_notes, kb_overview")
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
_seed_fixture(db)
|
||||
@@ -382,24 +398,30 @@ def test_ls_misuse_self_corrects_to_noarg_listing(
|
||||
expect(bubble).to_contain_text(FIRST_CATALOG_LINE)
|
||||
_assert_no_error_banner(page)
|
||||
|
||||
# The UI shows the two tool lines in order: the scoped misuse
|
||||
# (🔎 Listing documents in <code>.</code>) then the corrected
|
||||
# unscoped listing (🔎 Listing documents — no <code>).
|
||||
# The UI shows the three tool lines in order: the scoped misuse
|
||||
# (🔎 Listing documents in <code>.</code>), the corrected
|
||||
# unscoped listing (🔎 Listing documents — no <code>), then the
|
||||
# drill (🔎 Listing documents in <source> — phase 94, the top
|
||||
# level lists sources only, so the file lines need one more level).
|
||||
lines = page.locator(".msg.brain .tool-call")
|
||||
expect(lines).to_have_count(2)
|
||||
expect(lines).to_have_count(3)
|
||||
expect(lines.nth(0)).to_contain_text("Listing documents in")
|
||||
expect(lines.nth(0).locator("code")).to_have_text(".")
|
||||
expect(lines.nth(1)).to_contain_text("Listing documents")
|
||||
expect(lines.nth(1).locator("code")).to_have_count(0)
|
||||
expect(lines.nth(2)).to_contain_text("Listing documents in")
|
||||
expect(lines.nth(2).locator("code")).to_have_text(SEED_SOURCE)
|
||||
|
||||
# Two rounds on the wire: the tool frames arrive in order — first
|
||||
# ls with argument "." (the incident's misuse), then ls with
|
||||
# argument null (the correction) — and there is NO third tool
|
||||
# frame: the loop ended in one correction, not at the round cap.
|
||||
# Three rounds on the wire: the tool frames arrive in order —
|
||||
# first ls with argument "." (the incident's misuse), then ls with
|
||||
# argument null (the correction), then the drill ls scoped to the
|
||||
# source (phase 94) — and there is NO fourth tool frame: the loop
|
||||
# ended in one correction + one drill, not at the round cap.
|
||||
frames = _drain_frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": "."},
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "ls", "argument": SEED_SOURCE},
|
||||
]
|
||||
first_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert all(
|
||||
@@ -433,6 +455,7 @@ def test_plain_tool_flow_not_swallowed_by_new_trigger(
|
||||
assert _tool_frames(teach_frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": "."},
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "ls", "argument": SEED_SOURCE},
|
||||
]
|
||||
expect(
|
||||
page.locator(".msg.brain .bubble").last
|
||||
@@ -445,14 +468,17 @@ def test_plain_tool_flow_not_swallowed_by_new_trigger(
|
||||
_wait_settled(page)
|
||||
|
||||
second_msg = page.locator(".msg.brain").last
|
||||
# The UI shows the single-read flow's two lines: the unscoped ls
|
||||
# then the read of the first catalog line's COMBINED identity.
|
||||
# The UI shows the single-read flow's three lines: the unscoped ls,
|
||||
# the drill ls (phase 94), then the read of the first file line's
|
||||
# COMBINED identity.
|
||||
lines = second_msg.locator(".tool-call")
|
||||
expect(lines).to_have_count(2)
|
||||
expect(lines).to_have_count(3)
|
||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||
expect(lines.nth(0).locator("code")).to_have_count(0)
|
||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
||||
expect(lines.nth(1).locator("code")).to_have_text(DOC1_SP)
|
||||
expect(lines.nth(1)).to_contain_text("Listing documents in")
|
||||
expect(lines.nth(1).locator("code")).to_have_text(SEED_SOURCE)
|
||||
expect(lines.nth(2)).to_contain_text("Reading ")
|
||||
expect(lines.nth(2).locator("code")).to_have_text(DOC1_SP)
|
||||
|
||||
# The answer quotes the read document (the mock's deterministic
|
||||
# echo: "Read <source/path>. <first 80 chars>").
|
||||
@@ -461,11 +487,13 @@ def test_plain_tool_flow_not_swallowed_by_new_trigger(
|
||||
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
|
||||
_assert_no_error_banner(page)
|
||||
|
||||
# Wire level for the follow-up: ls (null) → read (the combined
|
||||
# identity) — the single-read flow, unchanged.
|
||||
# Wire level for the follow-up: ls (null) → the drill ls (the
|
||||
# source, phase 94) → read (the combined identity) — the
|
||||
# single-read flow, grown by the drill step.
|
||||
frames = _drain_frames(page)
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "ls", "argument": None},
|
||||
{"type": "tool", "name": "ls", "argument": SEED_SOURCE},
|
||||
{"type": "tool", "name": "read", "argument": DOC1_SP},
|
||||
]
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
|
||||
Reference in New Issue
Block a user