phase: 192_verbatim_inline_midturn

Phase 192 — final verification pass complete. - **Verified (all 4 tasks already complete):** additive `verbatim_block` SSE frame (`VerbatimBlockPiece` → `ChatVerbatimBlockEvent` → chat dispatch, after the round's `tool`/`tool_result`, before the next round); `hydrateVerbatimNow` matched-only mid-turn pass + app.js branch; `dropPlacedTwin` per-locator done/restore dedupe; the dedicated E2E suite. - **Root cause of the rejected attempt:** my own concurrent background `pytest` run deadlocked the harness validation run on the shared Postgres (`psycopg.errors.DeadlockDetected`). No code defect — re-ran every gate **exclusively**. - `uv run pytest -q --cov=app --cov-report=term-missing` → **exit 0**, `TOTAL 7181 16 99%` (>90%) - `uv run pytest tests/e2e/test_verbatim_inline_midturn.py --no-cov` → **6 passed** (isolated) - `uv run pytest tests/e2e/test_verbatim_show_text.py tests/e2e/test_chat_history.py --no-cov` → **18 passed** (`test_chat_history.py` byte-unedited; the phase-182 suite's "no eighth type" pin was tightened, not loosened, to seven + `verbatim_block` with frame-order asserts) - `uv run ruff check .` → All checks passed (exit 0) · `uv run pyright` → **0 errors** - **Completion criteria:** single frame per success + zero per refusal ✅ · `BOR_ANSWER_TEXT=0` wire byte-identical (integration spelled baseline + E2E App B) ✅ · fence hydrates in place / unfenced lands at the call moment, before `done` ✅ · append+fence → 1 panel, two ranges → 2 panels ✅ · restore 1 panel/locator, `done.answer_text` + record key + `verbatim_blocks=N` unchanged ✅ · suite green, coverage 99%, lint/types clean ✅ · no-show_text turn byte-identical ✅ · commit → left to the harness (no git index/history commands run). - **Docs:** README reviewed line-by-line (2399 lines) + AGENTS.md in full; mechanically verified all 86 `BOR_*` settings, every `.env.example` var, table defaults, internal anchors (0 broken), ports 8123/8901, tree depth 1–8, pagination 25/1–100, 17 palette colors, `0029`/`0030` migrations, and every cited test file. Fixed 1 inconsistency: the verbatim "Parity" bullet claimed the live turn/reload/shared page share "the same three globals" — now names the settle-path trio and states that `hydrateVerbatimNow` is the ONE chat-only addition (a shared page has no stream). - **Notable:** `run_agent`'s four repeated piece unions were collapsed into the ONE exported `AgentPiece` alias, so a future piece type is spelled in one place; `test_frontend_edit_message.py`'s `runTurn` byte-pin was regenerated, not weakened. - **Next pending phase:** `193_sync_remaining_count`.
2026-10-19 12:19:51 -04:00
parent 4f20c256d7
commit fb17d15fbe
49 changed files with 17313 additions and 102 deletions
@@ -51,11 +51,19 @@ retries (a dead-then-recovered round restarts before its first piece
 with a ``RetryPiece``; a mid-stream drop stays terminal — locked A2;
 the forced final no-tools call retries too; ``llm_retries=0`` is one
 plain attempt; retries are invisible to the round cap; consumer abandon
-mid-retry-sleep leaks nothing).
+mid-retry-sleep leaks nothing), and the phase-192 (task 01) mid-turn
+``VerbatimBlockPiece``: exactly ONE piece per SUCCESSFUL ``show_text`` —
+after the round's ``tool`` / ``tool_result`` pieces and before the next
+model round, the piece's ``block`` being the holder's own
+``shown_texts`` entry — and ZERO pieces on every refusal path (missing
+``path``, the ``answer_text=False`` posture, no document, an ``is_image``
+row, the spent ``show_text_max_blocks`` budget, the already-shown dedupe)
+or on the ``BOR_ANSWER_TEXT=0`` turn where the tool is not even offered.
 """
 from __future__ import annotations

 import asyncio
+import dataclasses
 import json
 import logging
 import uuid
@@ -82,7 +90,9 @@ from app.rag.agent import (
    SHOWN_TEXT_TRUNCATION_NOTICE,
    UNKNOWN_TOOL,
    AgentHolder,
+    AgentPiece,
    MalformedReplyError,
+    VerbatimBlockPiece,
    agent_tools,
    run_agent,
 )
@@ -96,6 +106,7 @@ from app.rag.llm import (
 )
 from app.rag.prompts import TOOLS_SECTION, _base, build_deflect_prompt, build_high_prompt
 from app.rag.retriever import TRUNCATION_MARKER
+from app.rag.verbatim import VerbatimBlock

 if TYPE_CHECKING:
    from app.rag.scaffolding import ScaffoldingFilter
@@ -201,12 +212,14 @@ async def _run(
    seed_docs: list[Document] | None = None,
    history: Sequence[dict[str, Any]] = (),
    db_factory: Callable[[], Session] | None = None,
-) -> list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]:
+) -> list[AgentPiece]:
    """Consume one ``run_agent`` turn; *history* (phase 74) is the
    client's prior turns spliced between system and user (default
    ``()`` — the pre-phase-74 two-message request). Phase 95: the loop
-    may also yield a ``ToolResultPiece`` (a truncated ``read``)."""
-    out: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece] = []
+    may also yield a ``ToolResultPiece`` (a truncated ``read``), and
+    phase 192 a ``VerbatimBlockPiece`` (a successful ``show_text``) —
+    both members of the one :data:`app.rag.agent.AgentPiece` union."""
+    out: list[AgentPiece] = []
    factory = db_factory or (lambda: _mock_session())
    async for piece in run_agent(
        cast("LLMClient", llm),
@@ -5143,8 +5156,8 @@ def test_round_failure_after_first_piece_is_terminal(monkeypatch: pytest.MonkeyP
    )
    sleeps = _record_sleeps(monkeypatch)

-    async def drain() -> list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]:
-        out: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece] = []
+    async def drain() -> list[AgentPiece]:
+        out: list[AgentPiece] = []
        with pytest.raises(LLMError, match="mid-stream drop"):
            async for piece in run_agent(
                cast("LLMClient", llm),
@@ -5205,8 +5218,8 @@ def test_zero_retries_is_one_plain_attempt(monkeypatch: pytest.MonkeyPatch) -> N
    llm = FailingLLM([([], LLMError("connection refused"))])
    sleeps = _record_sleeps(monkeypatch)

-    async def drain() -> list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]:
-        out: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece] = []
+    async def drain() -> list[AgentPiece]:
+        out: list[AgentPiece] = []
        with pytest.raises(LLMError, match="connection refused"):
            async for piece in run_agent(
                cast("LLMClient", llm),
@@ -5255,9 +5268,7 @@ def test_abandon_mid_retry_sleep_leaks_nothing(monkeypatch: pytest.MonkeyPatch)
            holder=holder,
        )

-        async def consumer() -> list[
-            StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece
-        ]:
+        async def consumer() -> list[AgentPiece]:
            return [p async for p in gen]

        task = asyncio.ensure_future(consumer())
@@ -6149,3 +6160,233 @@ def test_show_text_scripted_turn_returns_the_slice_to_the_model(
    assert len(holder.shown_texts) == 1
    assert holder.shown_texts[0].locator == f"{_SHOW_IDENTITY}#3-5"
    assert holder.tool_calls == 1
+
+
+# ---------------------------------------------------------------------------
+# Phase 192 (task 01, A1) — the mid-turn ``VerbatimBlockPiece``
+#
+# The block a successful ``show_text`` appended to ``holder.shown_texts``
+# used to reach the client only on the ``done`` frame (phase 182 task 05),
+# so every verbatim panel popped in AFTER the model finished talking about
+# the bytes. The loop now also yields ONE ``VerbatimBlockPiece`` the moment
+# the block exists — after the round's ``tool`` / ``tool_result`` pieces and
+# before the next model round — and the API layer turns it into the additive
+# ``verbatim_block`` SSE frame (pinned in ``tests/integration/test_chat_api.py``
+# and on the client in task 02). Success-only: every refusal path returns
+# before the ``shown_texts.append``, so it yields NOTHING.
+# ---------------------------------------------------------------------------
+
+
+def _patch_two_docs(
+    monkeypatch: pytest.MonkeyPatch, *docs: Document
+) -> None:
+    """Resolve several fixture rows by ``(source, path)`` (the multi-document
+    variant of :func:`_patch_show_doc` — a truncated ``read`` doc AND a
+    showable doc in one turn)."""
+
+    def _find(_db: Session, source: str, path: str) -> Document | None:
+        for doc in docs:
+            if (source, path) == (doc.source, doc.path):
+                return doc
+        return None
+
+    monkeypatch.setattr(agent, "find_document", _find)
+
+
+_DEFAULT_SHOW_DOC = object()  # "the resolvable fixture row" sentinel
+
+
+def _show_turn(
+    monkeypatch: pytest.MonkeyPatch,
+    arguments: dict[str, Any],
+    settings: Settings | None = None,
+    doc: Document | None | object = _DEFAULT_SHOW_DOC,
+) -> tuple[AgentHolder, ScriptedLLM, list[AgentPiece]]:
+    """One scripted turn whose model calls ``show_text`` with *arguments*
+    (the default resolves the fixture row; pass ``None`` for the
+    no-document refusal, or an ``is_image`` row for the A6 refusal)."""
+    _patch_show_doc(
+        monkeypatch, _show_doc() if doc is _DEFAULT_SHOW_DOC else cast("Document | None", doc)
+    )
+    holder = AgentHolder()
+    llm = ScriptedLLM(
+        [ToolCallPiece(id="c1", name="show_text", arguments=arguments)],
+        [StreamPiece("content", "ans")],
+    )
+    return holder, llm, asyncio.run(_run(llm, holder, settings or _settings()))
+
+
+def test_verbatim_block_piece_is_one_frozen_block() -> None:
+    """The piece's shape (the :class:`app.rag.llm.ToolResultPiece` pattern):
+    a FROZEN dataclass with exactly one field, ``block``, holding the
+    server's :class:`app.rag.verbatim.VerbatimBlock` — an immutable wire
+    value, never a re-derivation."""
+    block = VerbatimBlock(
+        source=_SHOW_SRC,
+        path=_SHOW_PATH,
+        title="Gufo quadlet",
+        start_line=2,
+        end_line=3,
+        total_lines=5,
+        text="L2-two\nL3-three",
+    )
+    piece = VerbatimBlockPiece(block=block)
+    assert [f.name for f in dataclasses.fields(piece)] == ["block"]
+    assert piece.block is block
+    with pytest.raises(dataclasses.FrozenInstanceError):
+        piece.block = None  # pyright: ignore[reportAttributeAccessIssue]
+
+
+def test_show_text_success_yields_one_verbatim_block_piece(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """The success pin: a scripted ``show_text`` yields EXACTLY ONE
+    ``VerbatimBlockPiece``, AFTER the round's ``ToolCallPiece`` (the ``tool``
+    frame) and BEFORE the next model round's first piece (the answer's
+    ``StreamPiece``). Its ``block`` IS the ``holder.shown_texts`` entry — the
+    same object, so the mid-turn frame and the done frame can never
+    disagree."""
+    _patch_show_doc(monkeypatch, _show_doc())
+    holder = AgentHolder()
+    llm = ScriptedLLM(
+        [
+            ToolCallPiece(
+                id="c1", name="show_text", arguments={"path": _SHOW_IDENTITY, "offset": 2}
+            )
+        ],
+        [StreamPiece("content", "here"), StreamPiece("content", " it is")],
+    )
+    out = asyncio.run(_run(llm, holder, _settings()))
+
+    pieces = [p for p in out if isinstance(p, VerbatimBlockPiece)]
+    assert len(pieces) == 1
+    piece = pieces[0]
+    # Order: the tool call, the block, THEN the next round's answer pieces.
+    assert isinstance(out[0], ToolCallPiece) and out[0].name == "show_text"
+    assert out[1] is piece
+    assert all(isinstance(p, StreamPiece) for p in out[2:])
+    # The block is the holder's own entry (identity, not a copy).
+    assert piece.block is holder.shown_texts[0]
+    assert piece.block.locator == f"{_SHOW_IDENTITY}#2-5"
+    assert piece.block.text == "L2-two\nL3-three\nL4-four\nL5-five"
+    # The done-frame bookkeeping is untouched by the piece (task 05's path).
+    assert holder.tool_calls == 1
+    assert len(holder.shown_texts) == 1
+    assert len(llm.requests) == 2  # the tool round + the answer round
+
+
+def test_verbatim_block_piece_lands_after_a_truncation_piece(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """The emission-order contract with BOTH derived pieces in one turn: a
+    truncated ``read`` (round 1 → ``ToolResultPiece``) then a
+    ``show_text`` (round 2 → ``VerbatimBlockPiece``) stream
+    ``tool, tool_result, tool, verbatim_block`` and only THEN the answer —
+    every derived piece stays inside its own round."""
+    big = _doc("S", "big.md", "Big", "\n".join(f"line {i}" for i in range(1, 30)))
+    _patch_two_docs(monkeypatch, big, _show_doc())
+    holder = AgentHolder()
+    llm = ScriptedLLM(
+        [ToolCallPiece(id="c1", name="read", arguments={"path": "S/big.md"})],
+        [ToolCallPiece(id="c2", name="show_text", arguments={"path": _SHOW_IDENTITY})],
+        [StreamPiece("content", "ans")],
+    )
+    out = asyncio.run(_run(llm, holder, _settings(read_max_chars=20)))
+
+    kinds = [
+        (
+            "tool:" + p.name
+            if isinstance(p, ToolCallPiece)
+            else "trunc"
+            if isinstance(p, ToolResultPiece)
+            else "block"
+            if isinstance(p, VerbatimBlockPiece)
+            else "retry"
+            if isinstance(p, RetryPiece)
+            else p.kind
+        )
+        for p in out
+    ]
+    assert kinds == [
+        "tool:read",
+        "trunc",
+        "tool:show_text",
+        "block",
+        "content",
+    ]
+    assert len([p for p in out if isinstance(p, VerbatimBlockPiece)]) == 1
+
+
+@pytest.mark.parametrize(
+    "case",
+    ["missing_args", "flag_off", "no_document", "image_doc", "block_cap"],
+)
+def test_show_text_refusals_yield_no_verbatim_block_piece(
+    monkeypatch: pytest.MonkeyPatch, case: str
+) -> None:
+    """Every refusal path yields ZERO pieces — the single success gate is
+    the ``shown_texts.append`` (the mid-turn frame is success-only by
+    construction, and the refusal itself is byte-identical to phase 182):
+    the missing-``path`` refusal, the ``answer_text=False`` UNKNOWN_TOOL
+    posture, the no-document teaching, the ``is_image`` refusal (LOCKED A6)
+    and the spent ``show_text_max_blocks`` budget."""
+    if case == "missing_args":
+        arguments: dict[str, Any] = {}
+        settings, doc = None, _DEFAULT_SHOW_DOC
+    elif case == "flag_off":
+        arguments = {"path": _SHOW_IDENTITY}
+        settings, doc = _settings(answer_text=False), _DEFAULT_SHOW_DOC
+    elif case == "no_document":
+        arguments = {"path": "nosuch/file.md"}
+        settings, doc = None, None
+    elif case == "image_doc":
+        arguments = {"path": _SHOW_IDENTITY}
+        settings, doc = None, _show_doc(is_image=True)
+    else:
+        arguments = {"path": _SHOW_IDENTITY}
+        settings, doc = _settings(show_text_max_blocks=0), _DEFAULT_SHOW_DOC
+
+    holder, _llm, out = _show_turn(monkeypatch, arguments, settings, doc)
+    assert [p for p in out if isinstance(p, VerbatimBlockPiece)] == []
+    _assert_refusal(holder)  # counts in nothing, displays nothing
+
+
+def test_show_text_already_shown_dedupe_yields_no_second_piece(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """The canonical-locator dedupe is a REFUSAL (D7), so the second,
+    identical show adds no holder entry and yields no second piece —
+    exactly ONE ``VerbatimBlockPiece`` across both rounds (the client can
+    therefore never be told to paint the same block twice)."""
+    _patch_show_doc(monkeypatch, _show_doc())
+    holder = AgentHolder()
+    call = ToolCallPiece(
+        id="c1", name="show_text", arguments={"path": _SHOW_IDENTITY, "offset": 2, "limit": 2}
+    )
+    llm = ScriptedLLM(
+        [call],
+        [ToolCallPiece(id="c2", name="show_text", arguments=call.arguments)],
+        [StreamPiece("content", "ans")],
+    )
+    out = asyncio.run(_run(llm, holder, _settings()))
+
+    pieces = [p for p in out if isinstance(p, VerbatimBlockPiece)]
+    assert len(pieces) == 1
+    assert pieces[0].block.locator == f"{_SHOW_IDENTITY}#2-3"
+    assert len(holder.shown_texts) == 1  # the dedupe refused the second show
+
+
+def test_kill_switch_turn_yields_no_verbatim_block_piece(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    """``BOR_ANSWER_TEXT=0`` (the phase-182 D8 posture): the tool is NOT
+    offered — so no call can succeed — and a scripted call anyway yields
+    zero pieces. The turn's piece sequence is byte-identical to pre-phase
+    (the wire pin lives in ``tests/integration/test_chat_api.py``)."""
+    holder, llm, out = _show_turn(
+        monkeypatch, {"path": _SHOW_IDENTITY}, _settings(answer_text=False)
+    )
+    offered = [t["function"]["name"] for t in (llm.requests[0][1] or [])]
+    assert "show_text" not in offered
+    assert [p for p in out if isinstance(p, VerbatimBlockPiece)] == []
+    _assert_refusal(holder)