phase: 94_ls_tree_drilldown
All green. Verification complete. **Phase 94 — `ls` drill-down tree: final verification pass (all 5 tasks were already complete; verified, nothing to fix)** - Verified `ls` 3-level tree (`app/rag/agent.py`): `ls()` sources + summaries, `ls(source)`/`ls(source/folder)` drill-down, 50-line file cap + grep-pointer note, NOT-A-FOLDER teaching refusal - Verified `folder_summaries` (migration 0017, model, `app/rag/folder_summaries.py` generator: `FOLDER_SUMMARY_MODE` marker, fail-soft per folder, ≥2-doc scope + prune) wired change-gated in both sync paths - Verified 10-turn fixture battery verdict recorded in `TOOL_CALLING_TESTING.md` §9 (2026-09-11): turbo PASS 19/19 contract, 98.7 s (−12.5…−13.2 % vs baseline); lite PASS 18/18, 43.6 s (+7.7 %) — accuracy at/above baseline, gate met - `uv run pytest --cov=app --cov-report=term-missing` → 1939 passed, 0 failed; TOTAL coverage **99 %** (folder_summaries.py 100 %) - `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings - E2E in isolation: `test_ls_tree_drilldown.py` 3 passed; `test_agent_document_tools` 4, `test_agent_unlimited_tools` 4, `test_harness_aligned_tools` 3, `test_search_tool` 3, `test_grep_regex_teaching` 2, `test_response_to_docs` 4 — all passed (read/grep contracts untouched) - Dedicated folder-summary tests (fail-soft, prune, both sync paths, migration): 46 passed - Completion criteria: all 6 met; working tree holds only phase-94 changes (commit left to harness per protocol) **Next pending phase:** `95_read_truncation_cap`
This commit is contained in:
@@ -19,27 +19,33 @@ phase-64 default 0.15 is far below the 5 s tool-line threshold):
|
||||
|
||||
NOTE the prelude: the chat flow's QUESTION EMBEDDING also travels
|
||||
through the proxy (one 6 s sleep) before the agent loop starts, so
|
||||
the wall-clock timeline of one turn is:
|
||||
the wall-clock timeline of one turn is (phase 94: the drill-down
|
||||
``ls`` adds a drill step — the top level lists sources only, so the
|
||||
flow drills one level before the first file line exists):
|
||||
|
||||
1. t≈12 s — the first SSE ``tool`` frame ("🔎 Listing documents"): the
|
||||
6 s embedding sleep + request 1's 6 s sleep (``tools`` offered, no
|
||||
tool results yet — streams ONLY the ``ls`` ``tool_calls`` delta,
|
||||
id ``call_0``) + the mock's ~0.2 s tool-call stream;
|
||||
2. t≈19 s — the second ``tool`` frame ("📄 Reading
|
||||
Deployments/example-record-file.json"): request 2's 6 s sleep (a
|
||||
``tool``-role catalog result → streams a ``read`` ``tool_calls``
|
||||
delta on the JOINED combined ``source/path`` of the FIRST catalog
|
||||
line, id ``call_1``);
|
||||
3. t≈25 s — the content answer ``Read <source/path>. <first 80 chars
|
||||
of the read document's content>``: request 3's 6 s sleep (a
|
||||
2. t≈19 s — the second ``tool`` frame ("🔎 Listing documents in
|
||||
Deployments"): request 2's 6 s sleep (the top-level source listing
|
||||
in the messages — no file lines yet → streams the drill ``ls``
|
||||
scoped to the first source, id ``call_1``);
|
||||
3. t≈25 s — the third ``tool`` frame ("📄 Reading
|
||||
Deployments/example-record-file.json"): request 3's 6 s sleep (a
|
||||
``tool``-role folder listing with file lines → streams a ``read``
|
||||
``tool_calls`` delta on the JOINED combined ``source/path`` of the
|
||||
FIRST file line, id ``call_2``);
|
||||
4. t≈31 s — the content answer ``Read <source/path>. <first 80 chars
|
||||
of the read document's content>``: request 4's 6 s sleep (a
|
||||
``tool``-role read result) → the first answer ``delta``.
|
||||
|
||||
So each post-tool-frame gap (≈6.3 s — the 6 s sleep + the short
|
||||
tool-call stream) is PAST the 5 s tool-line threshold
|
||||
(``TOOL_LINE_ELAPSED_AFTER_MS = 5_000``), and the 10 s pre-token
|
||||
typing hint (the existing ``startThinkingClock`` gate, ticking from
|
||||
t≈0) is visible throughout both post-tool gaps — everything before the
|
||||
answer's first delta (the whole turn runs ≈25 s + overhead, acceptable
|
||||
t≈0) is visible throughout all post-tool gaps — everything before the
|
||||
answer's first delta (the whole turn runs ≈31 s + overhead, acceptable
|
||||
for an isolated story suite). The four tests pin: (1) the latest tool
|
||||
line's ticking "(Ns)" suffix, (2) the typing indicator's visible "Ns"
|
||||
hint (+ the kept aria channel), (3) BOTH settling the instant the
|
||||
@@ -55,15 +61,20 @@ task that writes the visible span) — so the two channels read
|
||||
polls until they align (the deterministic same-moment pin).
|
||||
|
||||
**KB fixture** — byte-identical to the phase-37 suite
|
||||
(``tests/e2e/test_agent_document_tools.py``): ``Homelab/aws-route53.md``
|
||||
(``tests/e2e/test_agent_document_tools.py``), plus the phase-94
|
||||
registry rows: both sources are registered in ``git_sources``
|
||||
(``Deployments`` FIRST — registry order ``(added_at, id)``), so the
|
||||
mock's drill (first source of the top-level listing) lands on the
|
||||
folder that carries the file lines. ``Homelab/aws-route53.md``
|
||||
carries one chunk embedded with the mock's own bag-of-words vector —
|
||||
the marker question (phase 37's exact question, which carries the
|
||||
trigger phrase) cosines ≈0.69 against it, well past the E2E 0.30
|
||||
threshold, so the turn is grounded and the HIGH prompt carries the
|
||||
``<tools>`` section; ``Deployments/example-record-file.json`` is
|
||||
indexed WITHOUT chunks, and its ``(source, path)`` sorts FIRST in the
|
||||
catalog (``Deployments`` < ``Homelab``) — exactly the line the mock's
|
||||
second request reads, so the answer is byte-stable.
|
||||
folder listing's file lines (``Deployments`` < ``Homelab``) — exactly
|
||||
the line the mock's read request targets, so the answer is
|
||||
byte-stable.
|
||||
|
||||
Test → story mapping (Playwright Mapping Rule):
|
||||
1. ``test_tool_line_shows_ticking_elapsed``
|
||||
@@ -92,7 +103,7 @@ from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Chunk, Document
|
||||
from app.models import Chunk, Document, GitSource
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
@@ -211,8 +222,10 @@ SUFFIX_TIMEOUT_MS = 12_000
|
||||
#: the indicator is removed at the first delta (≈25 s): 20 s from the
|
||||
#: check start cannot flake.
|
||||
TYPING_HINT_TIMEOUT_MS = 20_000
|
||||
#: The answer settles at ≈19 s from submit; every settle wait here starts
|
||||
#: ≥12 s in — 30 s keeps ≥2× headroom on the remaining window.
|
||||
#: The answer settles at ≈31 s from submit (phase 94: the drill step
|
||||
#: adds a third tool round); every settle wait here starts ≈17 s in
|
||||
#: (after the first line's suffix) — 30 s keeps ≥2× headroom on the
|
||||
#: remaining window.
|
||||
SETTLE_TIMEOUT_MS = 30_000
|
||||
#: The restore is a synchronous boot re-render — 15 s is ample.
|
||||
RESTORE_TIMEOUT_MS = 15_000
|
||||
@@ -360,7 +373,16 @@ def app_url(app_server: str) -> str:
|
||||
|
||||
def _seed(db: Session) -> None:
|
||||
"""The phase-37 two-document pair, byte-identical (see the module
|
||||
docstring)."""
|
||||
docstring), plus the phase-94 registry rows: both sources
|
||||
registered, ``Deployments`` FIRST — the mock's drill (first
|
||||
source of the top-level listing) lands on the JSON file."""
|
||||
# COMMIT between the inserts (not flush): ``added_at`` is
|
||||
# ``server_default now()`` — the transaction timestamp — and the
|
||||
# tie-break is the random uuid ``id``, so one-transaction rows order
|
||||
# nondeterministically.
|
||||
db.add(GitSource(url=READ_SOURCE, kind="local"))
|
||||
db.commit()
|
||||
db.add(GitSource(url=SEED_SOURCE, kind="local"))
|
||||
md = Document(
|
||||
source=SEED_SOURCE,
|
||||
path=SEED_PATH,
|
||||
@@ -408,7 +430,7 @@ def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, "
|
||||
"steering_notes, kb_overview, saved_chats"
|
||||
"steering_notes, kb_overview, saved_chats, git_sources"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
@@ -514,7 +536,7 @@ def _wait_settled(page: Page) -> None:
|
||||
|
||||
#: The FIRST tool line's suffix, pinned to the line ("first .tool-call
|
||||
#: child of the .tool-calls container" — the "Listing documents" line;
|
||||
#: the read frame's second line arms its OWN clock 5 s later and is
|
||||
#: the drill frame's second line arms its OWN clock 5 s later and is
|
||||
#: never the target here).
|
||||
FIRST_LINE_SUFFIX = "#messages .tool-calls .tool-call:first-child .tool-elapsed"
|
||||
|
||||
@@ -611,8 +633,8 @@ def test_typing_indicator_shows_visible_elapsed(
|
||||
)
|
||||
|
||||
# The clock ticks while the gap holds: ≥1.5 s later the value is
|
||||
# strictly greater (the read frame is ≥6 s away; the first delta
|
||||
# — which removes the whole indicator — is ≈25 s from submit).
|
||||
# strictly greater (the drill frame is ≥6 s away; the first delta
|
||||
# — which removes the whole indicator — is ≈31 s from submit).
|
||||
page.wait_for_timeout(int(SAMPLE_GAP_S * 1000))
|
||||
v2 = _elapsed_value(
|
||||
page.locator("#typing-indicator .typing-elapsed").first.text_content(),
|
||||
@@ -644,7 +666,7 @@ def test_indicators_settle_when_the_answer_arrives(
|
||||
# First the ticking state (the test-1 wait, reused)…
|
||||
expect(page.locator(FIRST_LINE_SUFFIX)).to_be_visible(timeout=SUFFIX_TIMEOUT_MS)
|
||||
|
||||
# …then the answer arrives (≈25 s from submit — see the module
|
||||
# …then the answer arrives (≈31 s from submit — see the module
|
||||
# docstring's timeline) and settles.
|
||||
_wait_settled(page)
|
||||
|
||||
@@ -661,13 +683,16 @@ def test_indicators_settle_when_the_answer_arrives(
|
||||
expect(page.locator("#messages .tool-elapsed")).to_have_count(0)
|
||||
expect(page.locator("#typing-indicator")).to_have_count(0)
|
||||
|
||||
# The tool lines remain — the permanent record, both present with
|
||||
# their pinned text (phase 37's pattern).
|
||||
# The tool lines remain — the permanent record, all three present
|
||||
# with their pinned text (phase 37's pattern + the phase-94 drill
|
||||
# line).
|
||||
lines = page.locator("#messages .tool-call")
|
||||
expect(lines).to_have_count(2)
|
||||
expect(lines).to_have_count(3)
|
||||
expect(lines.nth(0)).to_contain_text("Listing documents")
|
||||
expect(lines.nth(1)).to_contain_text("Reading ")
|
||||
expect(lines.nth(1)).to_contain_text(READ_SP)
|
||||
expect(lines.nth(1)).to_contain_text("Listing documents in")
|
||||
expect(lines.nth(1)).to_contain_text(READ_SOURCE)
|
||||
expect(lines.nth(2)).to_contain_text("Reading ")
|
||||
expect(lines.nth(2)).to_contain_text(READ_SP)
|
||||
|
||||
# The answer bubble is complete (the mock's deterministic quote).
|
||||
bubble = page.locator(".msg.brain .bubble").last
|
||||
@@ -705,12 +730,14 @@ def test_restored_turn_has_no_timer(
|
||||
expect(page.locator("#empty-state")).to_be_hidden(timeout=RESTORE_TIMEOUT_MS)
|
||||
|
||||
# The persisted record re-renders the tool lines, in saved order
|
||||
# (the phase-37 reload pin, mirrored)…
|
||||
# (the phase-37 reload pin, mirrored + the phase-94 drill line)…”
|
||||
restored = page.locator("#messages .tool-call")
|
||||
expect(restored).to_have_count(2, timeout=RESTORE_TIMEOUT_MS)
|
||||
expect(restored).to_have_count(3, timeout=RESTORE_TIMEOUT_MS)
|
||||
expect(restored.nth(0)).to_contain_text("Listing documents")
|
||||
expect(restored.nth(1)).to_contain_text("Reading ")
|
||||
expect(restored.nth(1)).to_contain_text(READ_SP)
|
||||
expect(restored.nth(1)).to_contain_text("Listing documents in")
|
||||
expect(restored.nth(1)).to_contain_text(READ_SOURCE)
|
||||
expect(restored.nth(2)).to_contain_text("Reading ")
|
||||
expect(restored.nth(2)).to_contain_text(READ_SP)
|
||||
|
||||
# …with NO timer (A6 — the restore never arms the clock) and the
|
||||
# page otherwise settled.
|
||||
|
||||
Reference in New Issue
Block a user