phase: 131_read_line_ranges
All 44 progress lines are pure dots, final line at 100%, exit 0. Everything is verified. Final report: **Phase 131 — final verification pass (all 4 tasks already complete in working tree)** - **Verified implemented:** `read(offset, limit)` schema + teaching descriptions; rewritten truncation notice; full 6-row dedupe matrix + range validation + `lines: a-b of N` header + slice cap in `_execute_tool`; wire-level `offset`/`limit` on `ChatToolEvent` (absent keys dropped → non-range frames byte-identical); mock-LLM `read the lines` flow; handbook repro (wording + line numbers derived from the real PDF at test time) - **Tests:** `uv run pytest --no-cov` → 3097 passed, exit 0; `uv run pytest --cov=app` → **99% on app/** (>90%); `uv run pytest tests/e2e/test_read_line_ranges.py -v --no-cov` → 2/2 in isolation (47.6 s); `tests/integration/test_read_line_ranges_handbook.py` → 2/2 (3.5 s); 10-file targeted E2E batch pinning pre-phase surfaces (grep/ls/plain-read/ALREADY_IN_CONTEXT/truncation) → 31/31; `ruff check .` clean; `pyright` 0 errors - **Completion criteria:** all 5 green — (1) exact range lines + `lines:` header, plain reads byte-identical (existing pins pass unchanged) ✅ (2) truncated-then-range succeeds end-to-end on the real handbook; other re-reads get teaching refusals ✅ (3) find-traps answered from real PDF text, no hardcoded wording ✅ (4) coverage + isolated E2E ✅ (5) no behavior change in completed phases ✅ - **Docs:** README + AGENTS reviewed in full. Fixed 3 inconsistencies: README missing `BOR_HISTORY_MAX_TURNS`/`BOR_HISTORY_MAX_CHARS` rows (pre-existing phase-74 gap — env-tunable, unit-pinned, undocumented); `.env.example` missing the same two vars (added); `.env.example` stale `BOR_READ_MAX_CHARS` comment (still said "grep-pointer notice (phase 95)" — updated to the phase-131 workflow notice). AGENTS.md verified consistent — no changes needed. All 67 `.env.example` vars documented, all defaults match `app/config.py`, pages/auth tables match live routes - **Note:** PLAN.md §7.1 still states the retired 46rem chat column (superseded by phase 100, which AGENTS.md rule 5 records correctly) — PLAN.md is off-limits for this pass, flagged only - **Next pending phase:** `132_image_upload_feedback`
This commit is contained in:
+239
-2
@@ -539,6 +539,50 @@ Implements just enough of the aipi surface:
|
||||
question always gets the echo whatever the honesty gate says.
|
||||
Verified 2026-09-27: no existing E2E question or fixture file
|
||||
contains the phrase, so every other suite is unaffected.
|
||||
- user message containing ``read the lines``
|
||||
(``READ_LINES_TRIGGER``, phase 131 task 04 — the read
|
||||
line-ranges' dedicated story suite
|
||||
``tests/e2e/test_read_line_ranges.py``) **and** the system
|
||||
prompt carries the ``<tools>`` section -> the deterministic
|
||||
grep -> read(offset/limit) DANCE (the owner's stuck scenario,
|
||||
TODO.md L7, end to end): classified statelessly by the tool
|
||||
results in the messages (the house count pattern — streaming
|
||||
only):
|
||||
* request 1 (``tools`` offered, no tool results yet): stream
|
||||
ONLY ``tool_calls`` deltas — ``grep`` with
|
||||
``{"pattern": READ_LINES_PATTERN}`` (id ``call_0``), no
|
||||
content, no reasoning;
|
||||
* a ``tool``-role grep result with a ``source/path:line: text``
|
||||
match line is in the messages (the FIRST match line's line
|
||||
number is parsed): the RANGE READ — a ``tool_calls`` delta —
|
||||
``read`` on the match line's combined ``source/path`` with
|
||||
``offset`` = the parsed line number (an int) and ``limit`` =
|
||||
``READ_LINES_LIMIT`` (30) (id ``call_1``) — the owner's exact
|
||||
dance: the range the model names is the part it does NOT
|
||||
have;
|
||||
* a ``tool``-role read result (content starting with the
|
||||
agent's ``"Document "`` prefix) is in the messages: the
|
||||
forced answer, byte-stable: ``Found it at line N. <first 80
|
||||
chars of the read result's content>`` — N = the offset the
|
||||
model passed (parsed from the read call's arguments in the
|
||||
assistant tool-call message — the agent loop echoes every
|
||||
executed call back into the messages), the quote = the read
|
||||
slice with the header lines (the phase-106 D5 ``date:`` line
|
||||
+ the phase-131 ``lines: a-b of N`` extent) skipped, so the
|
||||
quote is the pure range content (the phase-94 read-echo
|
||||
convention — the RANGE reached the model's context);
|
||||
* a tool result is in the messages that is neither a read
|
||||
result nor a grep match (the grep found nothing — the
|
||||
premise broke): the deterministic terminal answer ``No
|
||||
matches — the knowledge base has no such text.`` (the
|
||||
grep-teach ``nomatch`` convention — the suite fails loudly
|
||||
on the missing marker).
|
||||
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint
|
||||
trigger phrases — the phase-71/72/94/118 ordering convention;
|
||||
the trigger needs the ``<tools>`` section, so deflected turns
|
||||
never hit it); verified 2026-09-28: no existing E2E question or
|
||||
fixture file contains the phrase, so every other suite is
|
||||
unaffected.
|
||||
|
||||
Failure injection (phase 67, LLM retry, TODO.md L3) — deterministic
|
||||
dead-endpoint behavior for the retry E2E suite (``tests/e2e/
|
||||
@@ -638,6 +682,7 @@ test_llm_retry.py``). The mock is single-conversation per e2e server, so
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
@@ -2020,6 +2065,144 @@ def _summary_seed_read_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
return ("echo", results[-1])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 131 (task 04, the read line-ranges' dedicated story suite):
|
||||
# the deterministic grep -> read(offset/limit) dance — see the module
|
||||
# docstring
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: A user message containing this substring (case-insensitive) —
|
||||
#: combined with the ``<tools>`` section in the system prompt — drives
|
||||
#: the deterministic RANGE-READ flow (the read line-ranges' story
|
||||
#: suite, ``tests/e2e/test_read_line_ranges.py``): the owner's stuck
|
||||
#: scenario (TODO.md L7) — the huge document: ``grep`` finds the
|
||||
#: line, the ``read`` lands exactly that part with ``offset`` /
|
||||
#: ``limit``, and the answer echoes the read slice. Checked BEFORE
|
||||
#: the plain ``TOOLS_TRIGGER`` flow (disjoint trigger phrases — the
|
||||
#: phase-71/72/94/118 ordering convention); verified 2026-09-28: no
|
||||
#: existing E2E question or fixture file contains the phrase, so
|
||||
#: every other suite is unaffected.
|
||||
READ_LINES_TRIGGER = "read the lines"
|
||||
|
||||
#: The canned pattern the flow greps for (the suite's fixture document
|
||||
#: carries exactly one line containing it — the flow's read target
|
||||
#: line; the sentinel convention of ``SEARCH_PATTERN``).
|
||||
READ_LINES_PATTERN = "LINE_MARKER"
|
||||
|
||||
#: The line count the flow's range read requests (a PART of the
|
||||
#: document, not the whole — the owner's dance, TODO.md L7).
|
||||
READ_LINES_LIMIT = 30
|
||||
|
||||
|
||||
def _read_call_offset(body: dict[str, Any]) -> int:
|
||||
"""The offset the model's ``read`` call passed (phase 131, task 04).
|
||||
|
||||
The agent loop echoes every executed call back into the messages
|
||||
(the assistant ``tool_calls`` message, the arguments as a JSON
|
||||
string — ``app/rag/agent.py``), so the range-read request
|
||||
(request 3 of the flow) carries the call it was made from: this
|
||||
parses the FIRST assistant ``read`` call's ``offset``. ``0`` when
|
||||
absent or unparseable (the premise broke — the suite fails
|
||||
loudly on ``line 0``).
|
||||
"""
|
||||
for m in _messages(body):
|
||||
if m.get("role") != "assistant":
|
||||
continue
|
||||
# The agent loop's echoed call (the ``tool_calls`` LIST on the
|
||||
# assistant message — ``app/rag/agent.py``; :func:`_messages`
|
||||
# is typed for string content, so the list comes back as
|
||||
# ``Any`` here — the house mock convention over the wire
|
||||
# shape).
|
||||
raw_calls: Any = m.get("tool_calls")
|
||||
for call in (raw_calls if isinstance(raw_calls, list) else []):
|
||||
if not isinstance(call, dict):
|
||||
continue
|
||||
fn = call.get("function") or {}
|
||||
if not isinstance(fn, dict) or fn.get("name") != "read":
|
||||
continue
|
||||
try:
|
||||
args = json.loads(fn.get("arguments") or "{}")
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
offset = args.get("offset") if isinstance(args, dict) else None
|
||||
if isinstance(offset, int) and not isinstance(offset, bool):
|
||||
return offset
|
||||
return 0
|
||||
|
||||
|
||||
def _read_lines_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
"""Classify a phase-131 range-read request (see the module
|
||||
docstring). Classified statelessly by the tool results in the
|
||||
messages (the house count pattern), like the other marker flows:
|
||||
|
||||
* ``("answer", offset, quote)`` — a ``tool``-role read result
|
||||
(content starting with the agent's ``"Document "`` prefix) is in
|
||||
the messages: the forced answer, byte-stable — ``Found it at
|
||||
line N. <quote>`` with N the offset the model passed (parsed
|
||||
from the read call's arguments in the assistant tool-call
|
||||
message — a string on this return for the homogeneous
|
||||
``tuple[str, ...]`` shape; the dispatch formats it into the
|
||||
answer, the display surface) and quote the read result's
|
||||
content with the header lines (the phase-106 D5 ``date:`` line
|
||||
+ the phase-131 ``lines: a-b of N`` extent) skipped — the pure
|
||||
read slice, first 80 chars (the phase-94 read-echo convention:
|
||||
the RANGE reached the model's context).
|
||||
* ``("read", combined, line, "call_1")`` — a ``tool``-role grep
|
||||
result carrying ``source/path:line: text`` match lines is in the
|
||||
messages: the RANGE READ on the FIRST match line's combined
|
||||
``source/path`` — ``offset`` = the parsed line number (the
|
||||
grep's own string here — the dispatch parses the int for the
|
||||
wire) and ``limit`` = :data:`READ_LINES_LIMIT`.
|
||||
* ``("nomatch",)`` — a tool result is in the messages but it is
|
||||
neither a read result nor a grep match (the grep found nothing
|
||||
— the premise broke): the deterministic terminal answer (the
|
||||
grep-teach ``nomatch`` line — the suite fails loudly on the
|
||||
missing marker).
|
||||
* ``("grep",)`` — ``tools`` are offered and no tool results are in
|
||||
the messages yet: the first step — ``grep`` with
|
||||
``{"pattern": READ_LINES_PATTERN}`` (id ``call_0``).
|
||||
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
|
||||
section is missing (deflected turns never carry it), or
|
||||
``tools`` are not offered and no tool results are in the
|
||||
messages yet (e.g. ``agent_max_rounds=0``).
|
||||
"""
|
||||
if READ_LINES_TRIGGER not in _user(body).lower():
|
||||
return None
|
||||
if "<tools>" not in _system(body):
|
||||
return None
|
||||
reads = _read_results(body)
|
||||
if reads:
|
||||
_sp, content = reads[-1]
|
||||
# The read result's header lines sit between the ``Document
|
||||
# …:`` header and the document text — the phase-106 D5
|
||||
# ``date:`` line always, the phase-131 ``lines: a-b of N``
|
||||
# extent on a range read (a plain read carries no extent
|
||||
# line) — skipped (the drill-flow convention) so the quote is
|
||||
# the pure read content (the suite's marker-line assertion).
|
||||
if content.startswith("date: "):
|
||||
content = content.partition("\n")[2]
|
||||
if content.startswith("lines: "):
|
||||
content = content.partition("\n")[2]
|
||||
return ("answer", str(_read_call_offset(body)), content[:80])
|
||||
for result in _tool_results(body):
|
||||
if result.startswith(_READ_RESULT_PREFIX):
|
||||
continue
|
||||
for line in result.splitlines():
|
||||
match = _SEARCH_LINE_RE.match(line)
|
||||
if match:
|
||||
return (
|
||||
"read",
|
||||
match.group("sp"),
|
||||
match.group("line"),
|
||||
"call_1",
|
||||
)
|
||||
if _tool_results(body):
|
||||
return ("nomatch",)
|
||||
if not body.get("tools"):
|
||||
return None
|
||||
return ("grep",)
|
||||
|
||||
|
||||
def long_answer() -> str:
|
||||
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
||||
a unique final line that must survive the stream untruncated."""
|
||||
@@ -2844,8 +3027,6 @@ def _sse_stream(
|
||||
|
||||
|
||||
def json_dumps(obj: dict[str, Any]) -> str:
|
||||
import json
|
||||
|
||||
return json.dumps(obj)
|
||||
|
||||
|
||||
@@ -3186,6 +3367,62 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 131 (task 04): the deterministic grep ->
|
||||
# read(offset/limit) DANCE (the read line-ranges' story suite,
|
||||
# tests/e2e/test_read_line_ranges.py — the owner's stuck
|
||||
# scenario, TODO.md L7): grep finds the line, the range read
|
||||
# lands the part the model does NOT have, the answer echoes
|
||||
# the read slice. Checked BEFORE the plain TOOLS_TRIGGER flow
|
||||
# (disjoint trigger phrases — the phase-71/72/94/118 ordering
|
||||
# convention; the trigger needs the ``<tools>`` section, so
|
||||
# deflected turns never hit it).
|
||||
read_lines = _read_lines_flow(body)
|
||||
if read_lines is not None:
|
||||
if read_lines[0] == "grep":
|
||||
stream = _tool_call_stream(
|
||||
"grep", {"pattern": READ_LINES_PATTERN}, "call_0"
|
||||
)
|
||||
elif read_lines[0] == "read":
|
||||
# The owner's dance, deterministic: ``offset`` = the
|
||||
# grep-reported line number (parsed from the FIRST
|
||||
# match line — the classifier keeps it as the grep's
|
||||
# own string for the homogeneous return; the int
|
||||
# here is the wire's), ``limit`` = READ_LINES_LIMIT
|
||||
# (a part of the document, not the whole).
|
||||
stream = _tool_call_stream(
|
||||
"read",
|
||||
{
|
||||
"path": read_lines[1],
|
||||
"offset": int(read_lines[2]),
|
||||
"limit": READ_LINES_LIMIT,
|
||||
},
|
||||
read_lines[3],
|
||||
)
|
||||
elif read_lines[0] == "nomatch":
|
||||
# The premise broke (the grep found nothing) — the
|
||||
# deterministic terminal answer (the grep-teach
|
||||
# ``nomatch`` convention — the suite fails loudly on
|
||||
# the missing marker).
|
||||
stream = _sse_stream(
|
||||
_apply_max_tokens(
|
||||
"No matches — the knowledge base has no such text.",
|
||||
body.get("max_tokens"),
|
||||
),
|
||||
0.0,
|
||||
)
|
||||
else: # "answer" — the forced read-echo (phase-94 shape)
|
||||
stream = _sse_stream(
|
||||
_apply_max_tokens(
|
||||
f"Found it at line {read_lines[1]}. {read_lines[2]}",
|
||||
body.get("max_tokens"),
|
||||
),
|
||||
0.0,
|
||||
)
|
||||
return StreamingResponse(
|
||||
stream,
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 124 (task 07, image chips and answers, TODO L3): the
|
||||
# scripted SHOW_IMAGE flow — the mock shows the seeded image
|
||||
# document (the ``<documents>`` block with the image-
|
||||
|
||||
@@ -0,0 +1,655 @@
|
||||
"""Phase 131 task 04 E2E (Playwright, mock-only): the read line-ranges'
|
||||
dedicated story suite — the owner's stuck scenario, end to end.
|
||||
|
||||
The whole TODO.md L7 item (``131_read_line_ranges`` task 04 is the
|
||||
story's gate): with the owner's ``BOR_READ_MAX_CHARS=24000`` in force,
|
||||
the agent deterministically performs the grep → read(offset/limit)
|
||||
dance on a ~100 KB document — the grep finds the marker line and
|
||||
reports its line number, the range read lands exactly that part (the
|
||||
part the model did NOT have), the SSE ``tool`` frames carry the REAL
|
||||
``offset``/``limit`` ints, and the final answer is grounded in the
|
||||
document's tail (the marker line). The second turn pins the other half
|
||||
of the fix: a plain read of the same large document truncates at
|
||||
24000, and the surfaced truncation carries the REWRITTEN notice that
|
||||
teaches the grep → read(offset/limit) workflow (the pre-phase dead-end
|
||||
wording is gone).
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_read_line_ranges.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real
|
||||
``turbo`` does whatever it does with the tools, while this story's
|
||||
gate is the deterministic SCRIPTED flows in
|
||||
``tests/e2e/mock_llm.py`` (the mock is the only E2E lens on the
|
||||
LLM's context):
|
||||
|
||||
* the range turn — ``READ_LINES_TRIGGER``: a user message containing
|
||||
``read the lines`` **and** the system prompt carrying the HIGH
|
||||
prompt's ``<tools>`` section. Request 1 greps for
|
||||
``READ_LINES_PATTERN``; request 2 (a ``tool``-role grep result with
|
||||
a ``source/path:line:`` match line) reads the match line's document
|
||||
with ``offset`` = the parsed line number and ``limit`` =
|
||||
``READ_LINES_LIMIT`` (30); request 3 (the read result present)
|
||||
answers ``Found it at line N. <first 80 chars of the read slice>``
|
||||
— N the offset the model passed (the phase-94 read-echo
|
||||
convention: the RANGE reached the context).
|
||||
* the plain-read turn — the mock's EXISTING ``READ_CAP_TRIGGER`` flow
|
||||
(``read the capped document: read source/path``, the
|
||||
``test_read_truncation_cap.py`` convention) reused on THIS suite's
|
||||
large document: the mock ECHOES the ENTIRE read tool result into
|
||||
its final answer (the house scripted-turn way of asserting on tool
|
||||
results), so the truncated shape — first cap chars +
|
||||
``[…truncated…]`` + the REWRITTEN notice — lands on the rendered
|
||||
answer and the suite asserts the notice's presence AND the old
|
||||
dead-end's absence through it.
|
||||
|
||||
The app under test boots with the OWNER'S cap (the
|
||||
env-override-for-the-app-under-test pattern — this suite's module app,
|
||||
as in ``test_read_truncation_cap.py``): ``BOR_READ_MAX_CHARS=24000``
|
||||
(the owner's own setting, TODO.md L7) — small enough that the ~99 KB
|
||||
fixture document's plain read truncates deterministically at 24000,
|
||||
big enough that the 30-line range slice (~1 300 chars) never does.
|
||||
|
||||
KB fixture — ONE large deterministic markdown document
|
||||
(``linedoc/big.md``, ~99 KB): this module generates it — 2 000
|
||||
numbered filler lines (``Line <i>: …`` — the line numbers are exact)
|
||||
under 200 ``##`` headings (one heading per 10 lines), with the marker
|
||||
line ``LINE_MARKER the trap components are documented here`` at the
|
||||
1 500th numbered line (= line 1 652 of the document — a known line
|
||||
near ~1500, FAR beyond the 24 000-char cap: the document's first
|
||||
24 000 chars hold NO marker). The marker's true line is computed from
|
||||
the very string written to the fixture (``BIG_DOC.splitlines().index(
|
||||
…) + 1``) — the assertions use that value, never a literal, so the
|
||||
suite stays true if the fixture is regenerated.
|
||||
|
||||
The app under test is mock-calibrated (conftest pattern):
|
||||
``BOR_RELEVANCE_THRESHOLD=0.30`` + ``BOR_LEXICAL_SUPPORT_FLOOR=0.15``
|
||||
+ ``BOR_SOURCE_USEFULNESS_FLOOR=0.15`` — both questions' best-chunk
|
||||
cosine (~0.23 on the mock's token-overlap distribution, the marker
|
||||
chunk wins) clears the floor with FTS hits (``trap`` / ``components``
|
||||
in the marker chunk), so every scripted turn stays GROUNDED (the
|
||||
``<tools>`` section is present), and the read document clears the
|
||||
usefulness bar into ``done.sources``.
|
||||
|
||||
Test → observable mapping (Playwright Mapping Rule; the story is the
|
||||
owner's TODO item — one Playwright file per story, A16):
|
||||
1. ``test_range_read_flow_over_sse`` — a question containing
|
||||
``read the lines`` → the SSE ``tool`` frames, IN ORDER: the
|
||||
``grep`` call (arg ``pattern="LINE_MARKER"``), then the ``read``
|
||||
call (asserted on the FRAME's args, not a substring: ``path`` =
|
||||
the doc's combined ``source/path``, ``offset`` an int == the
|
||||
marker's true line, ``limit`` == 30 — the wire's assertion
|
||||
surface for the owner's dance); the ``done`` frame is
|
||||
non-deflected with the read document cited (phase 119 A1); the
|
||||
final answer is the GROUNDED echo — ``Found it at line <N>``
|
||||
(N == the marker's true line) + the marker text itself (the range
|
||||
reached the model's context); NO truncation-notice dead-end
|
||||
(neither the ``[…truncated…]`` marker nor the notice is in the
|
||||
answer — the range read did not truncate, so no ``tool_result``
|
||||
frame either).
|
||||
2. ``test_plain_read_truncation_notice_teaches_the_workflow`` — a
|
||||
question carrying the mock's existing plain-read trigger (``read
|
||||
the capped document: read linedoc/big.md`` — the
|
||||
``READ_CAP_TRIGGER`` flow on this suite's large document) → the
|
||||
plain read truncates at 24 000: the SSE ``tool_result`` frame
|
||||
(``truncated: true``, ``chars_shown: 24000``, ``chars_total:`` the
|
||||
document's true length) lands after the ``tool`` frame and before
|
||||
the answer's first delta (the pinned order); the surfaced
|
||||
truncation — the mock's echo of the read result, the LLM-side
|
||||
surface the notice rides on — carries the REWRITTEN notice
|
||||
naming ``offset`` and ``limit`` (the pinned substrings — the
|
||||
pre-phase-131 dead-end wording "to locate what you need — grep
|
||||
searches the whole document" is gone).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Locator, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document
|
||||
from app.rag.agent import READ_TRUNCATION_NOTICE
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
from e2e.mock_llm import READ_LINES_LIMIT, READ_LINES_PATTERN
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns
|
||||
# its port in a combined run — this module app binds its own port
|
||||
# instead (a same-port second uvicorn dies on bind and would drive
|
||||
# the wrong server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_READLINES", "8148"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
#: The owner's read cap (TODO.md L7 — the env under which the owner
|
||||
#: hit the stuck scenario): the app under test boots with it.
|
||||
READ_CAP = 24000
|
||||
|
||||
SOURCE = "linedoc" # the local directory's basename = the source name
|
||||
BIG_REL = "big.md"
|
||||
BIG_SP = f"{SOURCE}/{BIG_REL}" # the combined read identity
|
||||
|
||||
#: The marker line the flow greps for: the mock's canned pattern
|
||||
#: (``READ_LINES_PATTERN`` — imported so the mock can never drift
|
||||
#: from the fixture) + the owner's question wording. The fixture
|
||||
#: document carries exactly one line containing it — the range
|
||||
#: read's target; the echo assertion asserts on this very string.
|
||||
MARKER_LINE_TEXT = f"{READ_LINES_PATTERN} the trap components are documented here"
|
||||
|
||||
# The fixture's layout (deterministic — see the module docstring):
|
||||
# 2 000 numbered lines under 200 headings (10 lines each), the
|
||||
# marker replacing the 1 500th numbered line.
|
||||
NUMBERED_LINES = 2000
|
||||
LINES_PER_BLOCK = 10
|
||||
MARKER_NUMBERED = 1500
|
||||
|
||||
_FILLER_WORDS = [
|
||||
"quiet", "mirror", "vault", "raid", "pool", "chain", "slot", "cycle",
|
||||
"guard", "probe", "audit", "manifest", "retention", "rotation",
|
||||
"label", "window", "image", "snapshot", "archive", "copy", "verify",
|
||||
"steady", "order", "exact", "clean", "whole", "night", "weekly",
|
||||
"plan", "track", "mark", "note", "file", "page", "entry", "index",
|
||||
"column", "row", "cell",
|
||||
]
|
||||
|
||||
|
||||
def _big_doc() -> str:
|
||||
"""The ~99 KB fixture document (deterministic — module docstring).
|
||||
|
||||
The H1, then 200 ``##`` headings (one per 10 numbered lines), each
|
||||
followed by its 10 numbered lines (``Line <i>: …``) — the marker
|
||||
line REPLACES the ``MARKER_NUMBERED``-th numbered line. The line
|
||||
numbers are exact: the marker's true line is computed from the
|
||||
returned string (``splitlines().index(…) + 1``), never a literal.
|
||||
"""
|
||||
lines: list[str] = ["# Lined document", ""]
|
||||
numbered = 0
|
||||
wi = 0
|
||||
for block in range(1, NUMBERED_LINES // LINES_PER_BLOCK + 1):
|
||||
lines.append(f"## Lines of the big lined read {block:04d}")
|
||||
for _ in range(LINES_PER_BLOCK):
|
||||
numbered += 1
|
||||
if numbered == MARKER_NUMBERED:
|
||||
lines.append(MARKER_LINE_TEXT)
|
||||
else:
|
||||
words = _FILLER_WORDS
|
||||
lines.append(
|
||||
f"Line {numbered}: "
|
||||
+ " ".join(words[wi % len(words) : wi % len(words) + 6])
|
||||
)
|
||||
wi += 6
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
BIG_DOC = _big_doc()
|
||||
BIG_DOC_TOTAL = len(BIG_DOC)
|
||||
#: The marker's true (1-based) line — computed from the very string
|
||||
#: written to the fixture (the determinism pin: the assertions use
|
||||
#: this, never a literal).
|
||||
MARKER_LINE_NO = BIG_DOC.splitlines().index(MARKER_LINE_TEXT) + 1
|
||||
|
||||
# The suite's length contracts (the task's pinned-count assumptions —
|
||||
# the assertions use ``len(BIG_DOC)`` of the very string written to
|
||||
# the fixture, and ``synced_kb`` pins the stored content
|
||||
# byte-identical to it): the document is OVER the owner's cap (the
|
||||
# plain read truncates), the marker is FAR past the cap (the first
|
||||
# cap chars hold NO marker — the stuck premise), the marker is unique
|
||||
# (the grep has exactly one match line), and the range slice is UNDER
|
||||
# the cap (the range read does not truncate — no ``tool_result``
|
||||
# frame on the range turn).
|
||||
assert BIG_DOC_TOTAL > READ_CAP, BIG_DOC_TOTAL
|
||||
assert MARKER_LINE_TEXT not in BIG_DOC[:READ_CAP], (
|
||||
f"the first {READ_CAP} chars must hold no marker"
|
||||
)
|
||||
assert BIG_DOC.count(MARKER_LINE_TEXT) == 1
|
||||
_slice = "\n".join(
|
||||
BIG_DOC.splitlines()[MARKER_LINE_NO - 1 : MARKER_LINE_NO - 1 + READ_LINES_LIMIT]
|
||||
)
|
||||
assert len(_slice) < READ_CAP, len(_slice)
|
||||
|
||||
# The REWRITTEN notice for this fixture's counts (the app.rag.agent
|
||||
# constant). The bubble renders the raw LLM-side text through
|
||||
# renderMarkdown (escape-first, markdown.js) — the notice's inline-
|
||||
# code backticks (`` `offset` `` / `` `limit` ``) become <code>
|
||||
# elements, so the DOM text_content() carries the notice with the
|
||||
# backticks stripped (the test_read_truncation_cap convention).
|
||||
NOTICE = READ_TRUNCATION_NOTICE.format(shown=READ_CAP, total=BIG_DOC_TOTAL)
|
||||
NOTICE_RENDERED = NOTICE.replace("`", "")
|
||||
# The pinned substrings (task 04): the notice names BOTH range
|
||||
# arguments — the workflow teaching.
|
||||
assert "offset" in NOTICE and "limit" in NOTICE
|
||||
# The pre-phase-131 dead-end wording (must be GONE from the
|
||||
# surfaced truncation).
|
||||
OLD_DEAD_END_1 = "to locate what you need"
|
||||
OLD_DEAD_END_2 = "grep searches the whole document"
|
||||
|
||||
# The scripted turns. Q_RANGE carries the mock's ``READ_LINES_TRIGGER``
|
||||
# (the range dance); Q_PLAIN carries the mock's existing
|
||||
# ``READ_CAP_TRIGGER`` (``read the capped document: read
|
||||
# source/path`` — the plain-read flow's scripted call, the
|
||||
# test_read_truncation_cap convention) on this suite's large
|
||||
# document. The shared ``trap components`` words keep both turns
|
||||
# GROUNDED on the single-document KB (the mock-calibrated gate — see
|
||||
# the module docstring).
|
||||
Q_RANGE = f"Read the lines of {BIG_SP} — what are the trap components?"
|
||||
Q_PLAIN = f"Read the capped document: read {BIG_SP} — the trap components, plain read check"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def linedoc_dirs(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The story's local-directory source: one host temp dir (the app
|
||||
server runs on the same host, so the path is visible to it)
|
||||
holding the ONE generated large document. The directory's
|
||||
basename is the source name (``kind=local``, phase 38)."""
|
||||
root = tmp_path_factory.mktemp("bor_read_lines") / SOURCE
|
||||
root.mkdir()
|
||||
(root / BIG_REL).write_text(BIG_DOC, encoding="utf-8")
|
||||
assert (root / BIG_REL).read_text(encoding="utf-8") == BIG_DOC
|
||||
assert not (root / ".git").exists() # the local-kind story: NOT git
|
||||
return root
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, linedoc_dirs: Path) -> Iterator[str]:
|
||||
"""The real app under test — per-module env (the conftest
|
||||
pattern, cf. ``test_read_truncation_cap.py``): NO
|
||||
``BOR_GIT_SOURCES`` (the env fallback is git-only — the source
|
||||
here is a DB-registered local directory), the mock LLM, the
|
||||
mock-calibrated gate (threshold 0.30 + lexical floor 0.15 — the
|
||||
conftest pair; the code-default floor 0.35 would refuse boot
|
||||
against the 0.30 threshold, phase 112) — and the SUBJECT: the
|
||||
owner's cap ``BOR_READ_MAX_CHARS=24000`` (TODO.md L7). The
|
||||
session app is never started in this isolated run, so no port
|
||||
clash."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (conftest pattern): both scripted turns
|
||||
# stay GROUNDED (the questions' best-chunk cosine ~0.23 clears
|
||||
# the 0.15 floor with FTS hits — the marker chunk wins).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
# The read document must clear the citation bar into
|
||||
# done.sources (phase 119 A1 — the range turn asserts it).
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
# Phase 67: instant retry waits + the code-default budget (the
|
||||
# conftest leak-guard pattern).
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's settings (the app reads
|
||||
# it from cwd) — override the source list with an EMPTY value (the
|
||||
# env var beats the .env file): the registry must hold EXACTLY the
|
||||
# local directory this suite registers (a leftover env git list
|
||||
# would pollute the KB the scripted turns run against).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||
# .env cannot leak corpus-specific settings into the app under
|
||||
# test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
# THE SUBJECT: the owner's cap (TODO.md L7 — the env under which
|
||||
# the owner hit the stuck scenario; the production default of
|
||||
# 128 000 stays pinned in tests/unit/test_config.py).
|
||||
env["BOR_READ_MAX_CHARS"] = str(READ_CAP)
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the E2E
|
||||
suites share one Postgres, so a leftover git_sources row or
|
||||
document would pollute the retrieval the scripted turns run
|
||||
against (the single-document KB's gate margins are pinned
|
||||
against EXACTLY this one document)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
|
||||
"""Poll the (cookie-authenticated) status endpoint until the run
|
||||
reaches a terminal state (the test_ls_tree_drilldown pattern, over
|
||||
plain httpx — this fixture has no browser page yet)."""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def synced_kb(app_server: str, linedoc_dirs: Path) -> None:
|
||||
"""The story's precondition: the one-source KB synced under the
|
||||
deterministic mock. Registers the temp directory through the
|
||||
authenticated API (the ``test_local_directory_sources.py``
|
||||
pattern), runs the REAL in-process sync (``POST /api/sync`` —
|
||||
walk → chunk → embed → overview → version bump), pins the stored
|
||||
content byte-identical to the fixture string (the ``chars_total``
|
||||
every assertion below uses is ``len(BIG_DOC)`` of the very string
|
||||
written to the directory)."""
|
||||
_truncate_all()
|
||||
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": str(linedoc_dirs)}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
body = _wait_sync_done_http(client)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == 1, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
# The import stored the fixture string BYTE-IDENTICALLY — the
|
||||
# ``chars_total`` every assertion below uses is ``len()`` of the
|
||||
# very string written to the directory.
|
||||
with SessionLocal() as db:
|
||||
stored = db.scalar(
|
||||
select(Document.content).where(
|
||||
Document.source == SOURCE, Document.path == BIG_REL
|
||||
)
|
||||
)
|
||||
assert stored == BIG_DOC, "stored content drifted from the fixture"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
"""Per-test query_log isolation (the KB itself is module-scoped —
|
||||
the scripted turns never change it, so the registry persists
|
||||
across the tests of this module)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the test_read_truncation_cap / test_ls_tree_drilldown
|
||||
# house patterns)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions
|
||||
#: for the ``tool`` / ``tool_result`` frames, independent of the UI
|
||||
#: rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_page_hooks(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _frames(page: Page, deadline_s: float = 120.0) -> list[dict]:
|
||||
"""The SSE frames captured since the last submit (``_submit``
|
||||
clears the buffer), once the hook's background read settles. The
|
||||
default deadline is generous: the plain-read turn's echo streams
|
||||
the whole 24 000-char truncated result (the mock's 12-char /
|
||||
0.02 s pacing ≈ 40 s)."""
|
||||
deadline = time.monotonic() + deadline_s
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _result_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool_result"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.evaluate("window.__sseFrames = []")
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page, timeout_ms: int = 120_000) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button
|
||||
recovered (the phase-48 settle wait; the timeout is a parameter —
|
||||
the plain-read turn's 24 000-char echo streams for ~40 s)."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text(
|
||||
"", timeout=timeout_ms
|
||||
)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=timeout_ms)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=timeout_ms)
|
||||
|
||||
|
||||
def _last_brain(page: Page) -> Locator:
|
||||
return page.locator(".msg.brain").last
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. The range turn: the owner's grep → read(offset/limit) dance on
|
||||
# the wire, grounded in the large doc's tail
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_range_read_flow_over_sse(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(60_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, Q_RANGE)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
# The owner's dance on the wire — IN ORDER, asserted on the
|
||||
# frames' ARGS (not a substring): the grep (the canned pattern),
|
||||
# then the range read — path = the doc's combined identity,
|
||||
# offset = the marker's TRUE line (the mock parsed it from the
|
||||
# grep result's first match line — the mock does the owner's
|
||||
# exact dance), limit = READ_LINES_LIMIT. The range fields ride
|
||||
# the SAME frame (the phase-131 wire surface — absent keys on
|
||||
# every non-range call).
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "grep", "argument": READ_LINES_PATTERN},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "read",
|
||||
"argument": BIG_SP,
|
||||
"offset": MARKER_LINE_NO,
|
||||
"limit": READ_LINES_LIMIT,
|
||||
},
|
||||
], _tool_frames(frames)
|
||||
# The range read did NOT truncate (30 lines ≈ 1 300 chars << the
|
||||
# 24 000 cap) — it streams NO tool_result frame (one frame = one
|
||||
# noteworthy event, the phase-95 additive contract).
|
||||
assert _result_frames(frames) == [], _result_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# The read document is the turn's cited source (phase 119 A1 —
|
||||
# done.sources is the READ docs).
|
||||
assert any(
|
||||
s["path"] == BIG_REL and s["source"] == SOURCE for s in done["sources"]
|
||||
), done["sources"]
|
||||
|
||||
# The GROUNDED echo — the range reached the model's context and
|
||||
# landed in the answer: ``Found it at line <N>`` (N == the
|
||||
# marker's true line) + the marker text itself (the answer is
|
||||
# grounded in the large doc's TAIL — far past the 24 000-char
|
||||
# cap a plain read could reach)…
|
||||
text = _last_brain(page).locator(".bubble").text_content() or ""
|
||||
assert f"Found it at line {MARKER_LINE_NO}." in text, text
|
||||
assert MARKER_LINE_TEXT in text, text
|
||||
# …and NO truncation-notice dead-end (the pre-phase stuck shape —
|
||||
# "the rest is NOT shown, use grep" — is not the answer: the
|
||||
# range read landed).
|
||||
assert TRUNCATION_MARKER not in text, text
|
||||
assert "TRUNCATED —" not in text, text
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. The plain-read turn: the truncation still fires at the owner's
|
||||
# cap, and the surfaced notice teaches the workflow (the dead-end
|
||||
# is gone)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_plain_read_truncation_notice_teaches_the_workflow(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(150_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, Q_PLAIN)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
# The plain read executed — the tool frame carries NO range keys
|
||||
# (a plain read: the phase-131 additive fields are ABSENT, never
|
||||
# null — the pre-phase frame shape, byte-identical).
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "read", "argument": BIG_SP}
|
||||
], _tool_frames(frames)
|
||||
# EXACTLY ONE tool_result frame — the pinned counts (the
|
||||
# owner's cap: showing the first READ_CAP of the fixture's true
|
||||
# length).
|
||||
assert _result_frames(frames) == [
|
||||
{
|
||||
"type": "tool_result",
|
||||
"name": "read",
|
||||
"argument": BIG_SP,
|
||||
"truncated": True,
|
||||
"chars_shown": READ_CAP,
|
||||
"chars_total": BIG_DOC_TOTAL,
|
||||
}
|
||||
], _result_frames(frames)
|
||||
# Frame order: the marker lands AFTER the call is shown and
|
||||
# BEFORE the answer's first delta (the phase-37/48 tool-line
|
||||
# timing is untouched — the beat between the two is the
|
||||
# truncation).
|
||||
i_tool = next(i for i, f in enumerate(frames) if f.get("type") == "tool")
|
||||
i_result = next(
|
||||
i for i, f in enumerate(frames) if f.get("type") == "tool_result"
|
||||
)
|
||||
i_delta = next(i for i, f in enumerate(frames) if f.get("type") == "delta")
|
||||
assert i_tool < i_result < i_delta, (i_tool, i_result, i_delta)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
|
||||
# The surfaced truncation — the mock's echo of the read result
|
||||
# (the LLM-side surface the notice rides on — the house
|
||||
# scripted-turn lens): the REWRITTEN notice landed in the model's
|
||||
# context and in the rendered answer, naming BOTH range
|
||||
# arguments (the workflow teaching: grep reports line numbers,
|
||||
# then read with `offset` + `limit`)…
|
||||
text = _last_brain(page).locator(".bubble").text_content() or ""
|
||||
assert TRUNCATION_MARKER in text, text
|
||||
assert NOTICE_RENDERED in text, text
|
||||
# …and the pre-phase-131 dead-end wording is GONE.
|
||||
assert OLD_DEAD_END_1 not in text, text
|
||||
assert OLD_DEAD_END_2 not in text, text
|
||||
@@ -222,6 +222,12 @@ assert SHORT_DOC.count(SHORT_SENTINEL) == 1
|
||||
# app.rag.agent constant, formatted for this fixture's counts).
|
||||
MARKER = f" (truncated — showing {READ_CAP} of {CAP_DOC_TOTAL} chars)"
|
||||
NOTICE = READ_TRUNCATION_NOTICE.format(shown=READ_CAP, total=CAP_DOC_TOTAL)
|
||||
# The bubble renders the raw LLM-side text through renderMarkdown
|
||||
# (escape-first, markdown.js) — the phase-131 notice's inline-code
|
||||
# backticks (`` `offset` `` / `` `limit` ``) become <code> elements, so
|
||||
# the DOM text_content() carries the notice with the backticks
|
||||
# stripped.
|
||||
NOTICE_RENDERED = NOTICE.replace("`", "")
|
||||
|
||||
# The scripted turns (the mock's ``READ_CAP_TRIGGER`` questions — each
|
||||
# carries its own tool call after the colon; the turn-specific flavor
|
||||
@@ -679,7 +685,7 @@ def test_truncated_read_frame_order_live_marker_and_llm_notice(
|
||||
bubble = _last_brain(page).locator(".bubble")
|
||||
text = bubble.text_content() or ""
|
||||
assert TRUNCATION_MARKER in text, text
|
||||
assert NOTICE in text, text
|
||||
assert NOTICE_RENDERED in text, text
|
||||
# The first cap chars reached the model…
|
||||
assert HEAD_SENTINEL in text, text
|
||||
# …and the past-the-cap tail did NOT (the cut is real, not cosmetic).
|
||||
@@ -761,7 +767,7 @@ def test_truncated_read_save_and_shared_fidelity(
|
||||
# told about (same stored text, same renderer).
|
||||
anon_text = anon.locator(".msg.brain .bubble").first.text_content() or ""
|
||||
assert TRUNCATION_MARKER in anon_text, anon_text
|
||||
assert NOTICE in anon_text, anon_text
|
||||
assert NOTICE_RENDERED in anon_text, anon_text
|
||||
assert HEAD_SENTINEL in anon_text, anon_text
|
||||
assert TAIL_SENTINEL not in anon_text, anon_text
|
||||
anon_ctx.close()
|
||||
|
||||
@@ -197,8 +197,10 @@ def test_agent_tools_offers_the_harness_aligned_surface() -> None:
|
||||
assert set(ls["properties"]) == {"path"}
|
||||
read = by_name["read"]["function"]["parameters"]
|
||||
assert read["type"] == "object"
|
||||
assert read["required"] == ["path"]
|
||||
assert set(read["properties"]) == {"path"}
|
||||
assert read["required"] == ["path"] # offset/limit optional (131)
|
||||
assert set(read["properties"]) == {"path", "offset", "limit"}
|
||||
assert read["properties"]["offset"]["type"] == "integer"
|
||||
assert read["properties"]["limit"]["type"] == "integer"
|
||||
grep = by_name["grep"]["function"]["parameters"]
|
||||
assert grep["type"] == "object"
|
||||
assert grep["required"] == ["pattern"]
|
||||
|
||||
@@ -0,0 +1,495 @@
|
||||
"""Integration: phase 131 (task 03) — the owner's handbook repro.
|
||||
|
||||
The exact incident from ``TODO.md`` L7: the agent was asked "What are
|
||||
the components of the find traps spell?" over the real D&D player's
|
||||
handbook (``test_sources/players_handbook.pdf`` — the untracked fixture
|
||||
from ``pdf_tests.tar``), it ``grep``'d the text and FOUND it "including
|
||||
line number", and then dead-ended: "However, read takes a path, not a
|
||||
line number." This suite proves the fixed loop closes the dance on the
|
||||
REAL document: the handbook is indexed through the REAL importer
|
||||
(:func:`app.rag.importer.import_sources` — the deterministic in-process
|
||||
:class:`~tests.fakes.FakeEmbedder` stands in for the embed/summary
|
||||
calls, so the ~1200-chunk index build stays fast and offline), and a
|
||||
scripted fake LLM (the ``tests/unit/test_agent.py`` round pattern)
|
||||
drives the turn with the owner's ``BOR_READ_MAX_CHARS=24000`` in force:
|
||||
|
||||
* the **grep-first** repro — round 1 a ``grep`` (``pattern="find
|
||||
traps"``, lowercase — grep is case-insensitive), round 2 a ``read``
|
||||
with ``offset`` = the line the REAL grep result reported (parsed,
|
||||
never hardcoded) + ``limit`` = 40, round 3 the answer: the read
|
||||
result's content, verbatim (the phase-94 "the read document reached
|
||||
the context" echo convention — the loop's reply IS the slice).
|
||||
Exactly 3 rounds, ``tool_calls == 2`` (the owner's stuck loop made 2
|
||||
and dead-ended — the fixed loop closes in 3).
|
||||
* the **plain-read-first** variant — round 1 a plain ``read`` (the
|
||||
handbook truncates at 24000 with the REWRITTEN phase-131 notice,
|
||||
which names ``offset``/``limit`` — the dead-end is gone), round 2
|
||||
the grep, round 3 the range read of the ALREADY-TRUNCATED document
|
||||
(the phase-131 dedupe matrix's truncated-then-range row, end-to-end
|
||||
on the real document — it PROCEEDS; the ``REREAD_TRUNCATED``
|
||||
teaching is never needed because the model went straight to the
|
||||
range), round 4 the echo answer.
|
||||
|
||||
The expected find-traps text is DERIVED from the fixture at test time
|
||||
(extract the PDF's text with the phase-126
|
||||
:func:`app.rag.pdf.extract_pdf_text`, grep it the same way the model
|
||||
does — :func:`app.rag.agent.grep_document`'s case-insensitive fixed
|
||||
substring — and take the passage's first 60 chars): no hardcoded
|
||||
handbook wording, so the test stays true if the fixture is
|
||||
re-extracted.
|
||||
|
||||
House KB pattern (``test_agent_tools.py`` / ``test_importer_e2e.py``):
|
||||
the ``documents`` table is truncated around each test, so the whole-KB
|
||||
``grep`` sees EXACTLY the handbook — the grep's line numbers are then
|
||||
the extracted text's line numbers, one-to-one. The untracked fixture
|
||||
is only ever COPIED into a temp source dir, never mutated.
|
||||
|
||||
Requires: podman compose up -d db
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import re
|
||||
import shutil
|
||||
from collections.abc import AsyncIterator, Iterator
|
||||
from copy import deepcopy
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, cast
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import delete, select, text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document, GitSource
|
||||
from app.rag import agent
|
||||
from app.rag.agent import (
|
||||
AGENT_TOOLS,
|
||||
READ_TRUNCATION_NOTICE,
|
||||
REREAD_TRUNCATED,
|
||||
AgentHolder,
|
||||
run_agent,
|
||||
)
|
||||
from app.rag.importer import import_sources
|
||||
from app.rag.llm import (
|
||||
LLMClient,
|
||||
RetryPiece,
|
||||
StreamPiece,
|
||||
ToolCallPiece,
|
||||
ToolResultPiece,
|
||||
)
|
||||
from app.rag.pdf import extract_pdf_text
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.rag.scaffolding import ScaffoldingFilter
|
||||
|
||||
#: The repo root (this file lives in ``tests/integration/``).
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
#: The owner's real fixture (untracked — unpacked from
|
||||
#: ``test_sources/pdf_tests.tar``): the D&D player's handbook.
|
||||
HANDBOOK = REPO_ROOT / "test_sources" / "players_handbook.pdf"
|
||||
|
||||
#: The imported identity: the source dir is named ``test_sources``
|
||||
#: (the importer's ``source = root.name``) and the PDF sits directly
|
||||
#: inside it.
|
||||
SOURCE = "test_sources"
|
||||
PDF_NAME = "players_handbook.pdf"
|
||||
IDENTITY = f"{SOURCE}/{PDF_NAME}"
|
||||
|
||||
#: The owner's exact question (``TODO.md`` L7) — the turn's user
|
||||
#: message; the scripted loop answers it through the tools.
|
||||
QUESTION = "What are the components of the find traps spell?"
|
||||
|
||||
#: The owner's env: ``BOR_READ_MAX_CHARS=24000`` (the handbook is far
|
||||
#: longer, so a plain read of it truncates).
|
||||
READ_CAP = 24_000
|
||||
|
||||
#: The owner's pattern — lowercase (grep is case-insensitive).
|
||||
PATTERN = "find traps"
|
||||
|
||||
#: The scripted range read's line count (both variants).
|
||||
RANGE_LIMIT = 40
|
||||
|
||||
pytestmark = pytest.mark.skipif(
|
||||
not HANDBOOK.is_file(),
|
||||
reason=(
|
||||
"test_sources/players_handbook.pdf is missing — unpack "
|
||||
"test_sources/pdf_tests.tar (the untracked owner fixture)"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _settings() -> Settings:
|
||||
"""The turn's settings: the owner's read cap in force and the code
|
||||
default import extensions plus ``pdf`` (the owner's
|
||||
``BOR_IMPORT_EXTENSIONS`` names it — the handbook is an indexed
|
||||
source). ``_env_file=None`` keeps an operator's local ``.env`` out
|
||||
of the pins."""
|
||||
defaults = Settings.model_fields["import_extensions"].default
|
||||
kwargs: dict[str, Any] = {
|
||||
"_env_file": None,
|
||||
"import_extensions": f"{defaults},pdf",
|
||||
"read_max_chars": READ_CAP,
|
||||
}
|
||||
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _expected_passage() -> str:
|
||||
"""The first 60 chars of the handbook's find-traps passage, DERIVED
|
||||
from the fixture at test time (the task's no-hardcoded-wording
|
||||
rule): the PDF's extracted text (the same text the importer stored)
|
||||
is grepped the same way the model does —
|
||||
:func:`app.rag.agent.grep_document`, the tool's case-insensitive
|
||||
fixed-substring semantics — and the passage is the
|
||||
:data:`RANGE_LIMIT` lines from the FIRST grep match (the exact
|
||||
window the scripted loop reads). If the fixture is re-extracted the
|
||||
line numbers and wording are re-derived, so the assertion stays
|
||||
true."""
|
||||
content = extract_pdf_text(HANDBOOK)
|
||||
first_line, _ = agent.grep_document(content, PATTERN)[0]
|
||||
lines = content.splitlines()
|
||||
return "\n".join(lines[first_line - 1 : first_line - 1 + RANGE_LIMIT])[:60]
|
||||
|
||||
|
||||
def _first_grep_line(messages: list[dict[str, Any]], identity: str) -> int:
|
||||
"""The first line number the loop's grep reported for *identity* —
|
||||
parsed from the REAL grep result the loop produced (the task's
|
||||
no-hardcoded-line rule). The last ``tool`` message carries the
|
||||
grep result, one ``source/path:line: text`` line per match; the
|
||||
first match line for the handbook names the line the scripted
|
||||
range read takes."""
|
||||
last_tool = next(m for m in reversed(messages) if m.get("role") == "tool")
|
||||
content = last_tool["content"]
|
||||
assert isinstance(content, str)
|
||||
for line in content.splitlines():
|
||||
if line.startswith(f"{identity}:"):
|
||||
match = re.match(rf"^{re.escape(identity)}:(\d+):", line)
|
||||
assert match is not None, f"unparseable grep line: {line!r}"
|
||||
return int(match.group(1))
|
||||
raise AssertionError(f"no grep result line for {identity!r}")
|
||||
|
||||
|
||||
class HandbookAgentLLM:
|
||||
"""The scripted owner-scenario loop (the ``test_agent.py`` round
|
||||
pattern) — deterministic, and it parses the REAL line number from
|
||||
the grep result the loop produced (never a hardcoded line):
|
||||
|
||||
``plain_first=False`` (the grep-first repro — test 1):
|
||||
|
||||
* round 1 → ``grep`` (``pattern="find traps"``);
|
||||
* round 2 → ``read`` (``path`` = the handbook identity, ``offset``
|
||||
= the first grep-reported line, ``limit`` = 40);
|
||||
* round 3 → the answer: the read result's content, verbatim (the
|
||||
phase-94 echo convention — the loop's reply IS the slice).
|
||||
|
||||
``plain_first=True`` (the plain-read-first variant — test 2):
|
||||
|
||||
* round 1 → plain ``read`` (``path``);
|
||||
* round 2 → ``grep`` (``pattern="find traps"``);
|
||||
* round 3 → ``read`` (``path``, ``offset`` = the first grep-
|
||||
reported line, ``limit`` = 40) — the dedupe matrix's
|
||||
truncated-then-range row on the real document;
|
||||
* round 4 → the answer: the read result's content, verbatim.
|
||||
|
||||
Every ``chat_stream`` request's messages and tools are recorded
|
||||
(``requests``); the scripted :class:`ToolCallPiece` s, in call
|
||||
order, in ``calls``. The canned answer pieces are fed through the
|
||||
round's :class:`ScaffoldingFilter` exactly like
|
||||
``LLMClient.chat_stream`` (the ``test_agent.py`` ScriptedLLM
|
||||
pattern)."""
|
||||
|
||||
def __init__(self, identity: str, *, plain_first: bool) -> None:
|
||||
self.identity = identity
|
||||
self.plain_first = plain_first
|
||||
self.requests: list[
|
||||
tuple[list[dict[str, Any]], list[dict[str, Any]] | None]
|
||||
] = []
|
||||
self.calls: list[ToolCallPiece] = []
|
||||
|
||||
def _next_call(self, index: int, messages: list[dict[str, Any]]) -> ToolCallPiece | None:
|
||||
"""The scripted call for request *index* — ``None`` on the
|
||||
final (answer) round. The range read's ``offset`` is parsed
|
||||
from the grep result just delivered."""
|
||||
if self.plain_first:
|
||||
if index == 0:
|
||||
return ToolCallPiece(id="call_1", name="read", arguments={"path": self.identity})
|
||||
if index == 1:
|
||||
return ToolCallPiece(id="call_2", name="grep", arguments={"pattern": PATTERN})
|
||||
if index == 2:
|
||||
return ToolCallPiece(
|
||||
id="call_3",
|
||||
name="read",
|
||||
arguments={
|
||||
"path": self.identity,
|
||||
"offset": _first_grep_line(messages, self.identity),
|
||||
"limit": RANGE_LIMIT,
|
||||
},
|
||||
)
|
||||
else:
|
||||
if index == 0:
|
||||
return ToolCallPiece(id="call_1", name="grep", arguments={"pattern": PATTERN})
|
||||
if index == 1:
|
||||
return ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read",
|
||||
arguments={
|
||||
"path": self.identity,
|
||||
"offset": _first_grep_line(messages, self.identity),
|
||||
"limit": RANGE_LIMIT,
|
||||
},
|
||||
)
|
||||
return None
|
||||
|
||||
async def chat_stream(
|
||||
self,
|
||||
messages: list[dict[str, str]],
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
scaffolding: ScaffoldingFilter | None = None,
|
||||
) -> AsyncIterator[StreamPiece | ToolCallPiece]:
|
||||
self.requests.append((deepcopy(messages), deepcopy(tools)))
|
||||
call = self._next_call(len(self.requests) - 1, messages)
|
||||
if call is not None:
|
||||
self.calls.append(call)
|
||||
yield call
|
||||
return
|
||||
# The answer round: the last tool result (the range read's
|
||||
# result) — the loop's reply IS the slice, verbatim.
|
||||
last_tool = next(m for m in reversed(messages) if m.get("role") == "tool")
|
||||
content = last_tool["content"]
|
||||
assert isinstance(content, str)
|
||||
if scaffolding is None:
|
||||
yield StreamPiece("content", content)
|
||||
return
|
||||
cleaned = scaffolding.feed(content)
|
||||
if cleaned:
|
||||
yield StreamPiece("content", cleaned)
|
||||
tail = scaffolding.flush()
|
||||
if tail:
|
||||
yield StreamPiece("content", tail)
|
||||
|
||||
|
||||
async def _consume(
|
||||
llm: HandbookAgentLLM,
|
||||
holder: AgentHolder,
|
||||
settings: Settings,
|
||||
) -> list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]:
|
||||
"""Consume one scripted ``run_agent`` turn (the owner's question,
|
||||
no seed docs). SEC-14-04: a short-lived session per tool call —
|
||||
the same pattern as production and the phase-70 integration suite."""
|
||||
out: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece] = []
|
||||
async for piece in run_agent(
|
||||
cast("LLMClient", llm),
|
||||
lambda: SessionLocal(),
|
||||
system_prompt="SYSTEM_PROMPT",
|
||||
user_message=QUESTION,
|
||||
seed_docs=[],
|
||||
settings=settings,
|
||||
holder=holder,
|
||||
):
|
||||
out.append(piece)
|
||||
return out
|
||||
|
||||
|
||||
def _answer(pieces: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]) -> str:
|
||||
"""The turn's final answer text (the content pieces)."""
|
||||
return "".join(
|
||||
piece.text for piece in pieces if isinstance(piece, StreamPiece) and piece.kind == "content"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def handbook_kb(db: Session, tmp_path: Path) -> Iterator[Document]:
|
||||
"""The real handbook, indexed once per test through the REAL
|
||||
importer: a temp source dir containing ``test_sources/
|
||||
players_handbook.pdf`` (a COPY — the untracked fixture is never
|
||||
mutated), registered as a local source, imported with the
|
||||
deterministic in-process :class:`~tests.fakes.FakeEmbedder` (the
|
||||
~1200-chunk embed is instant — the whole test stays well under the
|
||||
task's ~60 s bar). Yields the indexed
|
||||
:class:`~app.models.Document` row.
|
||||
|
||||
The house KB pattern: ``chunks``/``documents``/``folder_summaries``
|
||||
are truncated around the test, so the whole-KB ``grep`` sees
|
||||
EXACTLY the handbook (its line numbers are the extracted text's
|
||||
line numbers, one-to-one)."""
|
||||
db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
|
||||
db.commit()
|
||||
root = tmp_path / SOURCE
|
||||
root.mkdir()
|
||||
shutil.copy2(HANDBOOK, root / PDF_NAME)
|
||||
row = GitSource(url=str(root), kind="local", path=str(root))
|
||||
db.add(row)
|
||||
db.commit()
|
||||
fake = FakeEmbedder(dim=768)
|
||||
fake.settings = _settings() # the owner's cap + the pdf token
|
||||
asyncio.run(import_sources([root], fake, session=db))
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == SOURCE, Document.path == PDF_NAME)
|
||||
)
|
||||
assert doc is not None, "the handbook did not import"
|
||||
assert len(doc.content) > READ_CAP, "the handbook must outgrow the owner's read cap"
|
||||
yield doc
|
||||
db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
|
||||
db.execute(delete(GitSource).where(GitSource.id == row.id))
|
||||
db.commit()
|
||||
db.expire_all()
|
||||
|
||||
|
||||
def test_grep_then_range_read_answers_the_find_traps_question(
|
||||
handbook_kb: Document,
|
||||
) -> None:
|
||||
"""The owner's stuck loop, fixed — grep → range read → answer:
|
||||
exactly 3 rounds with ``tool_calls == 2`` (the owner's stuck loop
|
||||
made 2 and dead-ended), the read call carries the grep-reported
|
||||
``offset`` + ``limit`` (parsed, not literal), and the final answer
|
||||
— the read result, verbatim — contains the handbook's REAL
|
||||
find-traps text (derived from the fixture at test time)."""
|
||||
doc = handbook_kb
|
||||
expected = _expected_passage()
|
||||
|
||||
llm = HandbookAgentLLM(IDENTITY, plain_first=False)
|
||||
holder = AgentHolder()
|
||||
pieces = asyncio.run(_consume(llm, holder, _settings()))
|
||||
|
||||
# Exactly 3 rounds: grep → range read → answer (the fixed loop
|
||||
# closes where the owner's stuck loop dead-ended).
|
||||
assert len(llm.requests) == 3
|
||||
assert [call.name for call in llm.calls] == ["grep", "read"]
|
||||
# The tools stay offered on every request (phase 45 — no budgets).
|
||||
assert llm.requests[0][1] == AGENT_TOOLS
|
||||
assert llm.requests[2][1] == AGENT_TOOLS
|
||||
|
||||
# Round 1: the lowercase pattern (grep is case-insensitive).
|
||||
assert llm.calls[0].arguments == {"pattern": PATTERN}
|
||||
# The grep result the loop produced: the handbook's REAL match
|
||||
# lines (``source/path:line: text``), the first naming the line
|
||||
# the range read must take.
|
||||
grep_result = llm.requests[1][0][-1]
|
||||
assert grep_result["role"] == "tool"
|
||||
assert isinstance(grep_result["content"], str)
|
||||
assert grep_result["content"].startswith(f"{IDENTITY}:")
|
||||
grep_line = _first_grep_line(llm.requests[1][0], IDENTITY)
|
||||
|
||||
# Round 2: the read call carried offset/limit — offset is the
|
||||
# grep-reported line (parsed from the real grep result, never a
|
||||
# literal), limit 40.
|
||||
read_call = llm.calls[1]
|
||||
assert read_call.arguments == {
|
||||
"path": IDENTITY,
|
||||
"offset": grep_line,
|
||||
"limit": RANGE_LIMIT,
|
||||
}
|
||||
|
||||
# The range read's result: the byte-identical header lines, the
|
||||
# true extent on the third line, the slice below.
|
||||
read_result = llm.requests[2][0][-1]
|
||||
assert read_result["role"] == "tool"
|
||||
assert isinstance(read_result["content"], str)
|
||||
total_lines = len(doc.content.splitlines())
|
||||
assert read_result["content"].startswith(f"Document {IDENTITY}:\n")
|
||||
assert (
|
||||
f"lines: {grep_line}-{grep_line + RANGE_LIMIT - 1} of {total_lines}\n"
|
||||
in read_result["content"]
|
||||
)
|
||||
|
||||
# Round 3: the answer IS the read result (the phase-94 echo
|
||||
# convention) and it carries the REAL find-traps text.
|
||||
answer = _answer(pieces)
|
||||
assert answer == read_result["content"]
|
||||
assert expected in answer
|
||||
|
||||
# Executed-call bookkeeping: both calls counted; the doc read into
|
||||
# the context exactly once.
|
||||
assert holder.tool_calls == 2
|
||||
assert [d.id for d in holder.read_docs] == [doc.id]
|
||||
assert holder.read_truncations == [] # the range slice fits the cap
|
||||
|
||||
|
||||
def test_plain_read_truncates_then_range_read_answers_the_find_traps_question(
|
||||
handbook_kb: Document,
|
||||
) -> None:
|
||||
"""The plain-read-first variant on the REAL document: round 1's
|
||||
plain read truncates at the owner's 24000 cap with the REWRITTEN
|
||||
phase-131 notice (it names ``offset``/``limit`` — the dead-end is
|
||||
gone) and surfaces a truncation signal; round 2 greps; round 3's
|
||||
range read of the ALREADY-TRUNCATED document PROCEEDS (the dedupe
|
||||
matrix's truncated-then-range row, end-to-end — the
|
||||
``REREAD_TRUNCATED`` teaching is never needed because the model
|
||||
went straight to the range); round 4 answers with the slice."""
|
||||
doc = handbook_kb
|
||||
expected = _expected_passage()
|
||||
|
||||
llm = HandbookAgentLLM(IDENTITY, plain_first=True)
|
||||
holder = AgentHolder()
|
||||
pieces = asyncio.run(_consume(llm, holder, _settings()))
|
||||
|
||||
# Four rounds: plain read → grep → range read → answer.
|
||||
assert len(llm.requests) == 4
|
||||
assert [call.name for call in llm.calls] == ["read", "grep", "read"]
|
||||
# Round 1: the PLAIN read (no range arguments at all).
|
||||
assert llm.calls[0].arguments == {"path": IDENTITY}
|
||||
|
||||
# Round 1's result: cut at the owner's cap, the shared marker +
|
||||
# the REWRITTEN notice (``{total}`` = the handbook's true length,
|
||||
# ``{shown}`` = the cap) — the notice names the offset/limit
|
||||
# workflow the loop then follows.
|
||||
plain_result = llm.requests[1][0][-1]
|
||||
assert plain_result["role"] == "tool"
|
||||
assert isinstance(plain_result["content"], str)
|
||||
notice = READ_TRUNCATION_NOTICE.format(shown=READ_CAP, total=len(doc.content))
|
||||
assert plain_result["content"].startswith(f"Document {IDENTITY}:\n")
|
||||
assert plain_result["content"].endswith(f"\n{TRUNCATION_MARKER}\n{notice}")
|
||||
# The truncation surfaced to the API layer as a ToolResultPiece
|
||||
# (the SSE ``tool_result`` frame's wire value).
|
||||
truncations = [p for p in pieces if isinstance(p, ToolResultPiece)]
|
||||
assert [(p.argument, p.chars_shown, p.chars_total) for p in truncations] == [
|
||||
(IDENTITY, READ_CAP, len(doc.content))
|
||||
]
|
||||
|
||||
# Round 3: the range read of the truncated doc — offset is the
|
||||
# grep-reported line (parsed), limit 40 — and the dedupe matrix
|
||||
# lets it PROCEED (no ``REREAD_TRUNCATED`` teaching anywhere in
|
||||
# the loop's tool results).
|
||||
assert llm.calls[1].arguments == {"pattern": PATTERN}
|
||||
grep_line = _first_grep_line(llm.requests[2][0], IDENTITY)
|
||||
read_call = llm.calls[2]
|
||||
assert read_call.arguments == {
|
||||
"path": IDENTITY,
|
||||
"offset": grep_line,
|
||||
"limit": RANGE_LIMIT,
|
||||
}
|
||||
reread = REREAD_TRUNCATED.format(shown=READ_CAP, source=SOURCE, path=PDF_NAME)
|
||||
tool_results = [
|
||||
m["content"]
|
||||
for messages, _tools in llm.requests
|
||||
for m in messages
|
||||
if m.get("role") == "tool" and isinstance(m["content"], str)
|
||||
]
|
||||
assert reread not in tool_results
|
||||
# The range read's result: the true extent on the third line.
|
||||
read_result = llm.requests[3][0][-1]
|
||||
assert read_result["role"] == "tool"
|
||||
assert isinstance(read_result["content"], str)
|
||||
total_lines = len(doc.content.splitlines())
|
||||
assert read_result["content"].startswith(f"Document {IDENTITY}:\n")
|
||||
assert (
|
||||
f"lines: {grep_line}-{grep_line + RANGE_LIMIT - 1} of {total_lines}\n"
|
||||
in read_result["content"]
|
||||
)
|
||||
|
||||
# Round 4: the answer IS the range read's result and it carries
|
||||
# the REAL find-traps text.
|
||||
answer = _answer(pieces)
|
||||
assert answer == read_result["content"]
|
||||
assert expected in answer
|
||||
|
||||
# All three calls executed (the truncated plain read, the grep,
|
||||
# the range read); the doc is committed to the context exactly
|
||||
# once; the holder's truncation record is the plain read's tuple.
|
||||
assert holder.tool_calls == 3
|
||||
assert [d.id for d in holder.read_docs] == [doc.id]
|
||||
assert holder.read_truncations == [(IDENTITY, READ_CAP, len(doc.content))]
|
||||
+755
-28
@@ -17,7 +17,18 @@ unknown-source refusal that counts nothing, the NOT-A-FOLDER teaching
|
||||
with the parent's subfolders, the 50-file cap + grep-pointer note),
|
||||
``read`` on the canonical combined form (split at the
|
||||
FIRST slash, full content, the bare-source-name refusal, the
|
||||
already-in-context dedupe, missing-args refusals), the phase-68 ``grep``
|
||||
already-in-context dedupe, missing-args refusals), the phase-131
|
||||
(task 02) range-read execution (the optional ``offset``/``limit``:
|
||||
:func:`agent._read_range_args` validation — whole ints ≥ 1, bools
|
||||
excluded — with the pinned :data:`agent.BAD_RANGE_ARGS` echo, the
|
||||
six-row dedupe matrix (a plain re-read of a TRUNCATED doc gets the
|
||||
pinned :data:`agent.REREAD_TRUNCATED` teaching; the truncated-then-
|
||||
range row PROCEEDS — the stuck-scenario fix —; every other re-read of
|
||||
a fully-read doc stays ``ALREADY_IN_CONTEXT``),
|
||||
:data:`agent.OFFSET_PAST_END` (the true line count named), the slice
|
||||
math + the ``lines: a-b of N`` header shape (plain reads byte-
|
||||
identical to pre-phase-131), the cap on the slice with the slice-
|
||||
length tuple ``chars_total``), the phase-68 ``grep``
|
||||
contract under its new name (the locked A5 pins: fixed substring,
|
||||
case-insensitive, 20-cap in catalog order, 200-char lines, locator-only
|
||||
— ``read_docs`` untouched, no-match lines counted), the round cap
|
||||
@@ -227,9 +238,12 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
# the prompt yet); read adds one of them (or any other document)
|
||||
# to the context; only an already-read document is refused. The
|
||||
# combined-identity handoff to the `ls` output / the <documents>
|
||||
# blocks stays; the truncation-notice paragraph (phase 95) and
|
||||
# the one-call-at-a-time discipline clause (task 05) survive the
|
||||
# rewrite byte-identical.
|
||||
# blocks stays; the one-call-at-a-time discipline clause (task
|
||||
# 05) survives byte-identical. Phase 131 (task 01): the old
|
||||
# truncation dead-end paragraph ("Follow it and use `grep`
|
||||
# (pattern) to locate the rest") is replaced by the working
|
||||
# workflow — grep reports line numbers, then read the part with
|
||||
# `offset` (1-based first line) + `limit` (number of lines).
|
||||
assert read["description"] == (
|
||||
"The <documents> section shows the SUMMARIES of the "
|
||||
"top-ranked documents — their full texts are NOT in "
|
||||
@@ -240,17 +254,20 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
"document you have already read — its full text is "
|
||||
"already in your prompt. Very large documents are "
|
||||
"truncated: you receive the first part plus a TRUNCATED "
|
||||
"notice naming how many more characters exist — the "
|
||||
"notice is authoritative, the document did NOT end "
|
||||
"where it stopped. Follow it and use `grep` (pattern) "
|
||||
"to locate the rest — it searches the whole document. "
|
||||
"notice naming how many more characters exist. To read "
|
||||
"the rest: use `grep` (pattern) to find where your text "
|
||||
"is — it reports line numbers — then call `read` again "
|
||||
"on the same document with `offset` (1-based first line) "
|
||||
"and `limit` (number of lines) to read exactly that "
|
||||
"part. Do not re-read a document you have already read "
|
||||
"in full — its full text is already in your prompt. "
|
||||
"Call one tool at a time — wait for this result before "
|
||||
"your next call."
|
||||
)
|
||||
# The byte-preserved contracts inside the rewrite, pinned as
|
||||
# substrings: the combined source/path identity, the
|
||||
# truncation-notice paragraph, the one-call-at-a-time
|
||||
# discipline sentence.
|
||||
# phase-131 grep → read(offset, limit) workflow paragraph, the
|
||||
# one-call-at-a-time discipline sentence.
|
||||
assert (
|
||||
"by its combined `source/path` string, exactly as shown in "
|
||||
"the `ls` output or the <documents> blocks"
|
||||
@@ -258,10 +275,11 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
assert (
|
||||
"Very large documents are truncated: you receive the first "
|
||||
"part plus a TRUNCATED notice naming how many more "
|
||||
"characters exist — the notice is authoritative, the "
|
||||
"document did NOT end where it stopped. Follow it and use "
|
||||
"`grep` (pattern) to locate the rest — it searches the "
|
||||
"whole document."
|
||||
"characters exist. To read the rest: use `grep` (pattern) "
|
||||
"to find where your text is — it reports line numbers — "
|
||||
"then call `read` again on the same document with "
|
||||
"`offset` (1-based first line) and `limit` (number of "
|
||||
"lines) to read exactly that part."
|
||||
) in read["description"]
|
||||
assert (
|
||||
"Call one tool at a time — wait for this result before "
|
||||
@@ -269,9 +287,30 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
) in read["description"]
|
||||
read_params = read["parameters"]
|
||||
assert read_params["type"] == "object"
|
||||
# Phase 131 (task 01, LOCKED A2): offset/limit are OPTIONAL —
|
||||
# ``required`` stays ``["path"]``.
|
||||
assert read_params["required"] == ["path"]
|
||||
assert set(read_params["properties"]) == {"path"}
|
||||
assert set(read_params["properties"]) == {"path", "offset", "limit"}
|
||||
assert read_params["properties"]["path"]["type"] == "string"
|
||||
# The line-range parameters (phase 131, task 01, LOCKED A2):
|
||||
# BOTH integer, 1-based (``offset`` = first line, ``limit`` =
|
||||
# line count) — the pi-harness ``read`` shape the model was
|
||||
# trained on. The descriptions carry the grep → read(offset,
|
||||
# limit) teaching (pinned copy).
|
||||
assert read_params["properties"]["offset"]["type"] == "integer"
|
||||
assert read_params["properties"]["limit"]["type"] == "integer"
|
||||
assert read_params["properties"]["offset"]["description"] == (
|
||||
"The first line to read, 1-based. Omit it to read from the "
|
||||
"beginning of the document. Use it to read a specific part "
|
||||
"of a large document — grep reports the line numbers of "
|
||||
"what you found."
|
||||
)
|
||||
assert read_params["properties"]["limit"]["description"] == (
|
||||
"The maximum number of lines to read, starting at offset. "
|
||||
"Omit it to read as many lines as fit under the character "
|
||||
"cap. Keep it small — you are reading a part, not the "
|
||||
"document."
|
||||
)
|
||||
# The combined source/path string is the canonical document identity
|
||||
# (phase 70) — the description pins it with a worked example; the
|
||||
# bare-path contract is stated up front (phase 72, task 02). Phase
|
||||
@@ -303,13 +342,14 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
"Search the indexed documents for an exact string "
|
||||
"(case-insensitive) and return up to 20 matching lines "
|
||||
"as `source/path:line: text` — a locator, not a "
|
||||
"context-adder: read the winner with `read`. The "
|
||||
"pattern is a plain substring, NEVER a regex — if a "
|
||||
"pattern with regex syntax (like '.*' or '\\.') comes "
|
||||
"back with no matches, retry with the plain text you "
|
||||
"expect to see. For a normal search pass ONLY `pattern` "
|
||||
"— it searches every document and that is how you "
|
||||
"search the knowledge base; never pass a source name "
|
||||
"context-adder: read the winner with `read` — for a "
|
||||
"large document, use the reported line number as "
|
||||
"`offset`. The pattern is a plain substring, NEVER a "
|
||||
"regex — if a pattern with regex syntax (like '.*' or "
|
||||
"'\\.') comes back with no matches, retry with the plain "
|
||||
"text you expect to see. For a normal search pass ONLY "
|
||||
"`pattern` — it searches every document and that is how "
|
||||
"you search the knowledge base; never pass a source name "
|
||||
"as `path` (a source name is not a document). Call one "
|
||||
"tool at a time — wait for this result before your "
|
||||
"next call."
|
||||
@@ -1956,16 +1996,27 @@ def test_read_at_cap_plus_one_truncates_with_marker_and_notice(
|
||||
|
||||
|
||||
def test_read_truncation_notice_is_pinned() -> None:
|
||||
"""Phase 95: the notice copy is pinned — it names the true length
|
||||
(``{total}``), the cap kept (``{shown}``), states the rest is NOT
|
||||
shown (the document did not end where it stopped), and points at
|
||||
``grep`` (which searches the whole document)."""
|
||||
"""Phase 95 (REWRITTEN phase 131, task 01): the notice copy is
|
||||
pinned — it names the true length (``{total}``), the cap kept
|
||||
(``{shown}``), states the rest is NOT shown (the document did
|
||||
not end where it stopped), and teaches the working workflow —
|
||||
the old dead-end is gone: ``grep`` reports line numbers, then
|
||||
read that part with ``offset`` / ``limit`` (BOTH parameters
|
||||
named in the notice itself)."""
|
||||
assert READ_TRUNCATION_NOTICE.format(shown=100, total=250) == (
|
||||
"TRUNCATED — this document is 250 characters; only the first "
|
||||
"100 are in your context. The rest is NOT shown. Use grep "
|
||||
"(pattern) to locate what you need — grep searches the whole "
|
||||
"document."
|
||||
"(pattern) to find what you need — it reports line numbers — "
|
||||
"then read that part with `offset` (1-based first line) and "
|
||||
"`limit` (number of lines)."
|
||||
)
|
||||
# Both line-range parameters are named in the notice — the model
|
||||
# stuck at the truncation sees the exact follow-up call.
|
||||
assert "`offset`" in READ_TRUNCATION_NOTICE
|
||||
assert "`limit`" in READ_TRUNCATION_NOTICE
|
||||
# The old dead-end sentence (no way to act on the line number)
|
||||
# is gone.
|
||||
assert "to locate what you need" not in READ_TRUNCATION_NOTICE
|
||||
|
||||
|
||||
def test_run_agent_yields_tool_result_after_tool_frame_before_next_round(
|
||||
@@ -2081,6 +2132,682 @@ def test_read_truncation_paths_with_suggested_seed(
|
||||
assert holder2.tool_calls == 0 and holder2.read_docs == []
|
||||
|
||||
|
||||
# ---------- phase 131 (task 02): the range-read execution (offset/limit) ----------
|
||||
|
||||
|
||||
def _lines_doc(source: str, path: str, n_lines: int, prefix: str = "line ") -> Document:
|
||||
"""A fixture document of EXACTLY *n_lines* lines (``prefix{i}``)."""
|
||||
content = "\n".join(f"{prefix}{i}" for i in range(1, n_lines + 1))
|
||||
return _doc(source, path, path, content)
|
||||
|
||||
|
||||
def test_bad_range_args_is_pinned() -> None:
|
||||
"""Phase 131 (task 02): the range-argument validation refusal is
|
||||
pinned — the contract (whole line numbers ≥ 1) with the bad values
|
||||
echoed (``!r`` — an absent value prints as ``None``)."""
|
||||
assert agent.BAD_RANGE_ARGS.format(offset=2.5, limit=None) == (
|
||||
"`offset` and `limit` are whole line numbers — pass integers ≥ 1 "
|
||||
"(you passed offset=2.5, limit=None)."
|
||||
)
|
||||
assert agent.BAD_RANGE_ARGS.format(offset=None, limit="10") == (
|
||||
"`offset` and `limit` are whole line numbers — pass integers ≥ 1 "
|
||||
"(you passed offset=None, limit='10')."
|
||||
)
|
||||
assert agent.BAD_RANGE_ARGS.format(offset=True, limit=None) == (
|
||||
"`offset` and `limit` are whole line numbers — pass integers ≥ 1 "
|
||||
"(you passed offset=True, limit=None)."
|
||||
)
|
||||
|
||||
|
||||
def test_reread_truncated_is_pinned() -> None:
|
||||
"""Phase 131 (task 02): the plain re-read teaching for a TRUNCATED
|
||||
document is pinned — the shown count, the resolved identity, and
|
||||
the working workflow (grep reports line numbers → read that part
|
||||
with ``offset`` + ``limit``)."""
|
||||
assert agent.REREAD_TRUNCATED.format(shown=100, source="S", path="big.md") == (
|
||||
"You already have the first 100 characters of S/big.md in your "
|
||||
"context. To read the rest, find where your text is with grep "
|
||||
"(it reports line numbers), then call read with `offset` "
|
||||
"(1-based first line) and `limit` (lines)."
|
||||
)
|
||||
|
||||
|
||||
def test_offset_past_end_is_pinned() -> None:
|
||||
"""Phase 131 (task 02): the offset-past-end teaching is pinned —
|
||||
the RESOLVED identity, the true line count, the grep pointer."""
|
||||
assert agent.OFFSET_PAST_END.format(
|
||||
source="S", path="a.md", total_lines=10, offset=11
|
||||
) == (
|
||||
"S/a.md is 10 lines long — `offset` 11 is past the end. Use a "
|
||||
"smaller offset (grep reports the exact line numbers)."
|
||||
)
|
||||
|
||||
|
||||
def test_read_range_args_plain_and_valid() -> None:
|
||||
"""Phase 131 (task 02): the pure validator — both absent is a plain
|
||||
read; valid ints pass through (either one-sided is ok: ``offset``
|
||||
without ``limit`` reads to the end, ``limit`` without ``offset``
|
||||
counts from the start — the caller normalizes)."""
|
||||
assert agent._read_range_args(None, None) == (True, "", None, None)
|
||||
assert agent._read_range_args(1, None) == (True, "", 1, None)
|
||||
assert agent._read_range_args(None, 7) == (True, "", None, 7)
|
||||
assert agent._read_range_args(3, 42) == (True, "", 3, 42)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("offset", "limit"),
|
||||
[
|
||||
(2.5, None), # a float
|
||||
("3", None), # a string
|
||||
(True, None), # a bool (an int subclass — the house guard excludes it)
|
||||
(0, None), # zero
|
||||
(-1, None), # a negative
|
||||
(None, 2.5), # a float limit
|
||||
(None, "10"), # a string limit
|
||||
(None, False), # a bool limit
|
||||
(None, 0), # a zero limit
|
||||
(None, -3), # a negative limit
|
||||
],
|
||||
)
|
||||
def test_read_range_args_bad_values_are_refused(
|
||||
offset: Any, limit: Any
|
||||
) -> None:
|
||||
"""Phase 131 (task 02): a PRESENT bad value (not a whole int ≥ 1 —
|
||||
the house ``isinstance(x, int) and not isinstance(x, bool)`` guard,
|
||||
or ``< 1``) refuses with the values echoed; the range comes back
|
||||
unusable (``None``/``None``)."""
|
||||
ok, line, got_offset, got_limit = agent._read_range_args(offset, limit)
|
||||
assert not ok
|
||||
assert got_offset is None and got_limit is None
|
||||
assert line == agent.BAD_RANGE_ARGS.format(offset=offset, limit=limit)
|
||||
|
||||
|
||||
def test_bad_range_args_refused_before_any_lookup(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 131 (task 02): the validation refusal fires in the flow —
|
||||
BEFORE the dedupe and the DB resolve (a float ``offset`` with an
|
||||
unknown path still gets the range refusal, not the no-document
|
||||
line): counts in nothing, consumes a round, the tools stay
|
||||
offered."""
|
||||
|
||||
def _boom(*_a: Any, **_k: Any) -> None:
|
||||
raise AssertionError("no DB lookup for a bad range argument")
|
||||
|
||||
monkeypatch.setattr(agent, "find_document", _boom)
|
||||
monkeypatch.setattr(agent, "all_documents", _boom)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c1", name="read", arguments={"path": "S/ghost.md", "offset": 2.5}
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
assert llm.requests[1][0][3]["content"] == agent.BAD_RANGE_ARGS.format(
|
||||
offset=2.5, limit=None
|
||||
)
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_bad_range_args_beats_the_dedupe(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 131 (task 02): a bad range argument on a document ALREADY
|
||||
IN CONTEXT is the range refusal, not ``ALREADY_IN_CONTEXT`` — the
|
||||
validation precedes the dedupe (a refusal either way: no count,
|
||||
no re-append)."""
|
||||
doc = _doc("S", "a.md", "A", "A-CONTENT")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="c1", name="read", arguments={"path": "S/a.md"})],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c2", name="read", arguments={"path": "S/a.md", "offset": 0}
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [doc] and holder.tool_calls == 1
|
||||
assert llm.requests[2][0][5]["content"] == agent.BAD_RANGE_ARGS.format(
|
||||
offset=0, limit=None
|
||||
)
|
||||
|
||||
|
||||
def test_offset_past_end_refused_naming_line_count(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 131 (task 02): ``offset`` = total+1 on a 10-line doc →
|
||||
the pinned teaching (the true line count named, the grep pointer)
|
||||
— a refusal: nothing counted, the doc NOT added to the context (a
|
||||
corrected retry still proceeds)."""
|
||||
doc = _lines_doc("S", "ten.md", 10)
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c1", name="read", arguments={"path": "S/ten.md", "offset": 11}
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
assert llm.requests[1][0][3]["content"] == agent.OFFSET_PAST_END.format(
|
||||
source="S", path="ten.md", total_lines=10, offset=11
|
||||
)
|
||||
|
||||
|
||||
def test_offset_past_end_on_one_line_doc(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 131 (task 02): a 1-line doc — ``offset=2`` is past the
|
||||
end (the true count is 1); ``offset=1`` with an overshooting
|
||||
``limit`` clamps to the single line."""
|
||||
doc = _doc("S", "one.md", "One", "only")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
# (a) offset past the end of the 1-line doc.
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c1", name="read", arguments={"path": "S/one.md", "offset": 2}
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[1][0][3]["content"] == agent.OFFSET_PAST_END.format(
|
||||
source="S", path="one.md", total_lines=1, offset=2
|
||||
)
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
# (b) the same doc read from line 1, limit overshoots — clamped.
|
||||
holder2 = AgentHolder()
|
||||
llm2 = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c1",
|
||||
name="read",
|
||||
arguments={"path": "S/one.md", "offset": 1, "limit": 5},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm2, holder2, _settings()))
|
||||
assert llm2.requests[1][0][3]["content"] == (
|
||||
"Document S/one.md:\ndate: 2024-06-15\nlines: 1-1 of 1\nonly"
|
||||
)
|
||||
assert holder2.read_docs == [doc] and holder2.tool_calls == 1
|
||||
|
||||
|
||||
# --- the six-row dedupe matrix (one test per row) ---
|
||||
|
||||
|
||||
def test_matrix_row1_plain_read_unread_doc_reads_full(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Matrix row 1: plain ``read(path)``, doc NOT in ``read_docs`` →
|
||||
the full read (truncate at the cap as today — the pre-phase-131
|
||||
path, no ``lines:`` header line)."""
|
||||
doc = _doc("S", "m1.md", "M1", "one\ntwo\nthree")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="c1", name="read", arguments={"path": "S/m1.md"})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert holder.read_docs == [doc] and holder.tool_calls == 1
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document S/m1.md:\ndate: 2024-06-15\none\ntwo\nthree"
|
||||
)
|
||||
|
||||
|
||||
def test_matrix_row2_plain_read_truncated_doc_gets_reread_teaching(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Matrix row 2 (the stuck scenario's first hop): a plain re-read
|
||||
of a document whose first read TRUNCATED → the pinned
|
||||
``REREAD_TRUNCATED`` teaching (the shown count from the matching
|
||||
truncation tuple — the cap) — a refusal: no re-emit, no count,
|
||||
no second DB lookup (the raw-arg fast path fires pre-resolve)."""
|
||||
cap = 100
|
||||
doc = _lines_doc("S", "m2.md", 20) # 150 chars > the cap
|
||||
lookups: list[tuple[str, str]] = []
|
||||
|
||||
def _find(db: Any, source: str, path: str) -> Document | None:
|
||||
lookups.append((source, path))
|
||||
return doc if (source, path) == ("S", "m2.md") else None
|
||||
|
||||
monkeypatch.setattr(agent, "find_document", _find)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="c1", name="read", arguments={"path": "S/m2.md"})],
|
||||
[ToolCallPiece(id="c2", name="read", arguments={"path": "S/m2.md"})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(read_max_chars=cap)))
|
||||
# Round 1: the truncated plain read (the stuck trigger) — the
|
||||
# pre-phase-131 truncation bytes, the holder tuple recorded.
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document S/m2.md:\ndate: 2024-06-15\n"
|
||||
+ doc.content[:cap]
|
||||
+ "\n"
|
||||
+ TRUNCATION_MARKER
|
||||
+ "\n"
|
||||
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=len(doc.content))
|
||||
)
|
||||
assert holder.read_truncations == [("S/m2.md", cap, len(doc.content))]
|
||||
# Round 2: the plain re-read → the teaching (the shown count = the
|
||||
# cap from the matching tuple).
|
||||
assert llm.requests[2][0][5]["content"] == agent.REREAD_TRUNCATED.format(
|
||||
shown=cap, source="S", path="m2.md"
|
||||
)
|
||||
# A refusal: the counters stay at the first read's, no re-append,
|
||||
# no second lookup.
|
||||
assert holder.read_docs == [doc]
|
||||
assert holder.tool_calls == 1
|
||||
assert lookups == [("S", "m2.md")]
|
||||
|
||||
|
||||
def test_matrix_row3_plain_read_full_doc_stays_already_in_context(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Matrix row 3: a plain re-read of a document read IN FULL (never
|
||||
truncated) → ``ALREADY_IN_CONTEXT`` — byte-identical to
|
||||
pre-phase-131 (the raw-arg fast path, no DB lookup)."""
|
||||
doc = _doc("S", "m3.md", "M3", "FULL-TEXT") # fits under any cap
|
||||
lookups: list[tuple[str, str]] = []
|
||||
|
||||
def _find(db: Any, source: str, path: str) -> Document | None:
|
||||
lookups.append((source, path))
|
||||
return doc if (source, path) == ("S", "m3.md") else None
|
||||
|
||||
monkeypatch.setattr(agent, "find_document", _find)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="c1", name="read", arguments={"path": "S/m3.md"})],
|
||||
[ToolCallPiece(id="c2", name="read", arguments={"path": "S/m3.md"})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
assert holder.read_docs == [doc] and holder.tool_calls == 1
|
||||
assert lookups == [("S", "m3.md")] # the re-read deduped pre-lookup
|
||||
|
||||
|
||||
def test_matrix_row4_range_read_truncated_doc_proceeds(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Matrix row 4 (the stuck scenario's FIX): after a TRUNCATED plain
|
||||
read, a range read of the SAME document PROCEEDS — the ``lines:``
|
||||
header + exactly the named lines. The doc was appended ONCE (the
|
||||
known set prevents the re-append); the range read counts as a
|
||||
successful call; the slice fits, so no new truncation tuple."""
|
||||
cap = 100
|
||||
doc = _lines_doc("S", "m4.md", 20) # 150 chars > the cap
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="c1", name="read", arguments={"path": "S/m4.md"})],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c2",
|
||||
name="read",
|
||||
arguments={"path": "S/m4.md", "offset": 5, "limit": 3},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(read_max_chars=cap)))
|
||||
# Round 1: the truncated plain read.
|
||||
assert llm.requests[1][0][3]["content"].endswith(
|
||||
"\n"
|
||||
+ TRUNCATION_MARKER
|
||||
+ "\n"
|
||||
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=len(doc.content))
|
||||
)
|
||||
# Round 2: the range read PROCEEDED — lines 5-7 of 20, exactly.
|
||||
assert llm.requests[2][0][5]["content"] == (
|
||||
"Document S/m4.md:\ndate: 2024-06-15\nlines: 5-7 of 20\n"
|
||||
"line 5\nline 6\nline 7"
|
||||
)
|
||||
assert holder.read_docs == [doc] # appended exactly once
|
||||
assert holder.tool_calls == 2 # both reads are successful calls
|
||||
assert holder.read_truncations == [("S/m4.md", cap, len(doc.content))]
|
||||
|
||||
|
||||
def test_matrix_row5_range_read_full_doc_stays_already_in_context(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Matrix row 5: a range read of a document read IN FULL (never
|
||||
truncated) → ``ALREADY_IN_CONTEXT`` — the range is a SUBSET of the
|
||||
text already in the prompt (the matrix needs the resolved pair, so
|
||||
the check runs after the resolve)."""
|
||||
doc = _lines_doc("S", "m5.md", 5)
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="c1", name="read", arguments={"path": "S/m5.md"})],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c2",
|
||||
name="read",
|
||||
arguments={"path": "S/m5.md", "offset": 2, "limit": 2},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
assert holder.read_docs == [doc] and holder.tool_calls == 1
|
||||
assert holder.read_truncations == []
|
||||
|
||||
|
||||
def test_matrix_row6_range_read_unread_doc_proceeds(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Matrix row 6: a range read of a NOT-yet-read document PROCEEDS —
|
||||
the ``lines:`` header + the slice, the doc appended, the call
|
||||
counted (a first read, plain or range, behaves like a first read)."""
|
||||
doc = _lines_doc("S", "m6.md", 8)
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c1",
|
||||
name="read",
|
||||
arguments={"path": "S/m6.md", "offset": 3, "limit": 2},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document S/m6.md:\ndate: 2024-06-15\nlines: 3-4 of 8\nline 3\nline 4"
|
||||
)
|
||||
assert holder.read_docs == [doc] and holder.tool_calls == 1
|
||||
|
||||
|
||||
# --- the slice math + the result shape ---
|
||||
|
||||
|
||||
def test_range_read_offset_one_no_limit_reads_whole_doc(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 131 (task 02): ``offset=1`` without ``limit`` reads to the
|
||||
end of the document — the true extent ``1-N of N``, the whole
|
||||
content (the slice, not the plain-read shape)."""
|
||||
doc = _doc("S", "r1.md", "R1", "a\nb\nc")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c1", name="read", arguments={"path": "S/r1.md", "offset": 1}
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document S/r1.md:\ndate: 2024-06-15\nlines: 1-3 of 3\na\nb\nc"
|
||||
)
|
||||
assert holder.read_docs == [doc] and holder.tool_calls == 1
|
||||
|
||||
|
||||
def test_range_read_offset_at_total_reads_last_line(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 131 (task 02): ``offset`` = the last line number →
|
||||
exactly the last line (``N-N of N``)."""
|
||||
doc = _doc("S", "r2.md", "R2", "a\nb\nc")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c1",
|
||||
name="read",
|
||||
arguments={"path": "S/r2.md", "offset": 3, "limit": 5},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document S/r2.md:\ndate: 2024-06-15\nlines: 3-3 of 3\nc"
|
||||
)
|
||||
|
||||
|
||||
def test_range_read_limit_overshoot_clamps_to_total(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 131 (task 02): an overshooting ``limit`` clamps ``end`` to
|
||||
the last line (reads to the end, never past it)."""
|
||||
doc = _doc("S", "r3.md", "R3", "a\nb\nc")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c1",
|
||||
name="read",
|
||||
arguments={"path": "S/r3.md", "offset": 2, "limit": 10},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document S/r3.md:\ndate: 2024-06-15\nlines: 2-3 of 3\nb\nc"
|
||||
)
|
||||
|
||||
|
||||
def test_range_read_limit_without_offset_counts_from_start(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 131 (task 02): ``limit`` WITHOUT ``offset`` is a line count
|
||||
from the start of the document (offset = 1 — the helper's
|
||||
documented one-sided semantics)."""
|
||||
doc = _doc("S", "r4.md", "R4", "a\nb\nc")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c1", name="read", arguments={"path": "S/r4.md", "limit": 2}
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document S/r4.md:\ndate: 2024-06-15\nlines: 1-2 of 3\na\nb"
|
||||
)
|
||||
|
||||
|
||||
def test_range_read_image_doc_carries_marker_after_lines_line(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 131 (task 02): a range read of a standalone-IMAGE document
|
||||
carries the :data:`agent.IMAGE_DOC_MARKER` line exactly where the
|
||||
plain read puts it — after the header lines (here, after
|
||||
``lines:``)."""
|
||||
doc = _doc("S", "img.png", "Img", "IMG-DESC-OF-LINE-1\nline-2-text")
|
||||
doc.is_image = True
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c1",
|
||||
name="read",
|
||||
arguments={"path": "S/img.png", "offset": 2, "limit": 1},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document S/img.png:\ndate: 2024-06-15\nlines: 2-2 of 2\n"
|
||||
f"{agent.IMAGE_DOC_MARKER}\n"
|
||||
"line-2-text"
|
||||
)
|
||||
assert holder.read_docs == [doc] and holder.tool_calls == 1
|
||||
|
||||
|
||||
def test_range_read_slice_over_cap_truncates_with_slice_length_total(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 131 (task 02): the cap applies to the SLICE — an over-cap
|
||||
slice is cut at the cap + the shared marker + the pinned notice
|
||||
(``{total}`` = the SLICE's length), and the holder tuple's
|
||||
``chars_total`` is the slice's length, NOT the document's (the
|
||||
loop still yields one ToolResultPiece for it)."""
|
||||
cap = 20
|
||||
lines = [f"L{i:02d}" + "x" * 30 for i in range(1, 11)] # 10 lines × 33 chars
|
||||
content = "\n".join(lines)
|
||||
doc = _doc("S", "cap.md", "Cap", content)
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c1",
|
||||
name="read",
|
||||
arguments={"path": "S/cap.md", "offset": 5, "limit": 6},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
out = asyncio.run(_run(llm, holder, _settings(read_max_chars=cap)))
|
||||
# The slice: lines 5-10 (6 × 33 chars + 5 newlines = 203 > cap).
|
||||
slice_text = "\n".join(lines[4:10])
|
||||
assert len(slice_text) == 203
|
||||
assert len(content) == 339 # the document is much bigger
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document S/cap.md:\ndate: 2024-06-15\nlines: 5-10 of 10\n"
|
||||
+ slice_text[:cap]
|
||||
+ "\n"
|
||||
+ TRUNCATION_MARKER
|
||||
+ "\n"
|
||||
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=len(slice_text))
|
||||
)
|
||||
# The tuple: the argument, the cap kept, the SLICE's length (203,
|
||||
# not the document's 339).
|
||||
assert holder.read_truncations == [("S/cap.md", cap, 203)]
|
||||
assert holder.read_docs == [doc] and holder.tool_calls == 1
|
||||
# The loop yields the ToolResultPiece with the slice's total.
|
||||
pieces = [p for p in out if isinstance(p, ToolResultPiece)]
|
||||
assert len(pieces) == 1
|
||||
assert pieces[0].argument == "S/cap.md"
|
||||
assert pieces[0].chars_shown == cap
|
||||
assert pieces[0].chars_total == 203
|
||||
|
||||
|
||||
def test_range_read_truncated_then_plain_reread_teaching_then_range_proceeds(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 131 (task 02): a TRUNCATED range read (offset=1, no limit
|
||||
— the whole 339-char doc under a 20-char cap) records the slice
|
||||
tuple; a plain re-read then gets ``REREAD_TRUNCATED`` with the
|
||||
shown count from THAT (range) tuple; a subsequent range read still
|
||||
PROCEEDS (truncated-then-range, regardless of which read
|
||||
truncated)."""
|
||||
cap = 20
|
||||
lines = [f"L{i:02d}" + "x" * 30 for i in range(1, 11)]
|
||||
content = "\n".join(lines)
|
||||
doc = _doc("S", "t.md", "T", content)
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c1", name="read", arguments={"path": "S/t.md", "offset": 1}
|
||||
)
|
||||
],
|
||||
[ToolCallPiece(id="c2", name="read", arguments={"path": "S/t.md"})],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c3",
|
||||
name="read",
|
||||
arguments={"path": "S/t.md", "offset": 5, "limit": 2},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(read_max_chars=cap)))
|
||||
# Round 1: the truncated range read (the whole doc, offset=1) —
|
||||
# the slice is the whole doc here (339 chars), the tuple's total
|
||||
# is the slice's length.
|
||||
assert llm.requests[1][0][3]["content"].startswith(
|
||||
"Document S/t.md:\ndate: 2024-06-15\nlines: 1-10 of 10\n"
|
||||
)
|
||||
assert llm.requests[1][0][3]["content"].endswith(
|
||||
"\n"
|
||||
+ TRUNCATION_MARKER
|
||||
+ "\n"
|
||||
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=339)
|
||||
)
|
||||
# Round 2: the plain re-read → the teaching, shown from the (range)
|
||||
# truncation tuple.
|
||||
assert llm.requests[2][0][5]["content"] == agent.REREAD_TRUNCATED.format(
|
||||
shown=cap, source="S", path="t.md"
|
||||
)
|
||||
# Round 3: the range read of the truncated doc PROCEEDS — lines 5-6
|
||||
# (66 + 1 = 67 chars > the cap → truncated again; the tuple's total
|
||||
# is the slice's length, the document's is irrelevant).
|
||||
slice_text = "\n".join(lines[4:6])
|
||||
assert llm.requests[3][0][7]["content"] == (
|
||||
"Document S/t.md:\ndate: 2024-06-15\nlines: 5-6 of 10\n"
|
||||
+ slice_text[:cap]
|
||||
+ "\n"
|
||||
+ TRUNCATION_MARKER
|
||||
+ "\n"
|
||||
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=len(slice_text))
|
||||
)
|
||||
assert holder.read_truncations == [("S/t.md", cap, 339), ("S/t.md", cap, 67)]
|
||||
assert holder.read_docs == [doc] # appended exactly once, still
|
||||
assert holder.tool_calls == 2 # the refusal counted nothing
|
||||
|
||||
|
||||
def test_plain_read_after_range_read_is_already_in_context(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 131 (task 02): a range read appends the doc to
|
||||
``read_docs`` ONCE — so a later PLAIN read of the range-read
|
||||
(never-truncated) document is ``ALREADY_IN_CONTEXT`` (the model has
|
||||
committed to this doc; the ``lines:`` line told it what it holds)."""
|
||||
doc = _doc("S", "p.md", "P", "a\nb\nc")
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="c1",
|
||||
name="read",
|
||||
arguments={"path": "S/p.md", "offset": 1, "limit": 2},
|
||||
)
|
||||
],
|
||||
[ToolCallPiece(id="c2", name="read", arguments={"path": "S/p.md"})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings()))
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document S/p.md:\ndate: 2024-06-15\nlines: 1-2 of 3\na\nb"
|
||||
)
|
||||
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
assert holder.read_docs == [doc] and holder.tool_calls == 1
|
||||
|
||||
|
||||
def test_unknown_tool_name_refused(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
|
||||
@@ -127,6 +127,46 @@ def test_tool_event_shape_is_type_name_argument_only() -> None:
|
||||
assert dumped["type"] == "tool" # default — call sites never spell it out
|
||||
|
||||
|
||||
def test_tool_frame_range_read_carries_offset_and_limit() -> None:
|
||||
"""Phase 131 (task 04): a range ``read``'s frame carries the line
|
||||
range the model requested — the wire's assertion surface for the
|
||||
owner's grep → read(offset/limit) dance (the story E2E pins the
|
||||
REAL ints the mock parsed from the grep result)."""
|
||||
frame = sse_event(
|
||||
ChatToolEvent(
|
||||
name="read", argument="S/p.md", offset=1500, limit=30
|
||||
).model_dump()
|
||||
)
|
||||
assert frame == (
|
||||
'data: {"type": "tool", "name": "read", "argument": "S/p.md", '
|
||||
'"offset": 1500, "limit": 30}\n\n'
|
||||
)
|
||||
assert _payload(frame) == {
|
||||
"type": "tool",
|
||||
"name": "read",
|
||||
"argument": "S/p.md",
|
||||
"offset": 1500,
|
||||
"limit": 30,
|
||||
}
|
||||
|
||||
|
||||
def test_tool_event_range_fields_absent_not_null_when_omitted() -> None:
|
||||
"""Phase 131 (task 04): the ``offset`` / ``limit`` keys are ABSENT
|
||||
(never ``null``) when omitted — a plain ``read`` (no range) and a
|
||||
``grep`` (no range parameters at all) both serialize to the
|
||||
byte-identical pre-phase shape (the omission pattern of
|
||||
``ChatDoneEvent.answer_image``). A bad range VALUE never reaches
|
||||
the frame at all: ``app.api.chat`` applies the whole-line-number
|
||||
guard before building the event (the E2E suite pins the frame the
|
||||
endpoint emits)."""
|
||||
for dumped in (
|
||||
ChatToolEvent(name="read", argument="S/p.md").model_dump(),
|
||||
ChatToolEvent(name="read", argument="S/p.md", offset=None, limit=None).model_dump(),
|
||||
ChatToolEvent(name="grep", argument="trap").model_dump(),
|
||||
):
|
||||
assert set(dumped.keys()) == {"type", "name", "argument"}, dumped
|
||||
|
||||
|
||||
def test_retry_frame_serializes_exactly() -> None:
|
||||
"""Phase 67: the ``retry`` frame is exactly ``{type: "retry",
|
||||
attempt: int, max_attempts: int}`` — a transient status the client's
|
||||
|
||||
Reference in New Issue
Block a user