phase: 141_read_range_dedupe_fix
All gates green and verified. Final report: **Phase 141 final verification pass — complete** - **Verified (tasks 01–04 already in `complete/`):** `AgentHolder.read_coverage` span recording (all 4 read outcomes, refusals record nothing); coverage dedupe matrix (range read of uncovered lines proceeds, fully-covered re-read → `ALREADY_IN_CONTEXT` with a now-true claim, partial plain re-read → `REREAD_PARTIAL` naming merged spans, `REREAD_TRUNCATED` retired, fast path retired per task 02); `TOOLS_SECTION` re-read sentence re-cut to coverage-true form (prompt lock re-pinned); owner-scenario integration repro (pure + truncated variants) and mock `read two ranges` flow + dedicated E2E — all match the task specs exactly; no code defects found. - **Tests/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → exit 0, TOTAL 99% (5498 stmts, `app/rag/agent.py` 99%) — run twice, both green; `uv run pytest tests/e2e/test_read_range_dedupe.py -v --no-cov` → 2 passed in isolation; `uv run ruff check . && uv run pyright` → All checks passed / 0 errors. - **Completion criteria:** (1) two different ranges both reach the model, re-covered re-read refused truthfully — ✅ (integration + E2E); (2) partial plain re-read gets the `offset`/`limit` teaching, false claim gone both directions — ✅; (3) plain-read + phase-131 result shapes byte-identical, existing pins pass unchanged — ✅; (4) all gate commands green, >90% — ✅; (5) no behavior change in completed phases (grep/ls, truncation notice, `read_docs`/`read_truncations` signals) — ✅ (whole project suite green). - **Documentation:** README.md — fixed pre-existing phase-140 gap: `BOR_PDF_PICTORIAL` / `BOR_PDF_PICTORIAL_MIN_CHARS` were in `.env.example`/Settings but undocumented (README claims every `.env.example` var is covered); added the pictorial-PDF-pass bullet to the PDF section + two config-reference rows. AGENTS.md — verified consistent (rules, quick-ref commands, phase refs all check out) — no changes needed. - **Notable:** overview's "fast path survives for holder-state rows" note is superseded by task 02's authoritative decision (every dedupe row resolves — "full" needs the true line count); implementation follows task 02. No phase files touched; changes left in the working tree for the harness commit. - **Next pending phase:** `142_kb_overview_freshness`.
This commit is contained in:
+276
-5
@@ -644,17 +644,76 @@ Implements just enough of the aipi surface:
|
||||
N`` extent) skipped, first 80 chars (the phase-94 read-echo
|
||||
convention — BOTH ranges reached the model's context). NOTE:
|
||||
for the second read to PROCEED (not be refused
|
||||
``ALREADY_IN_CONTEXT``) the document must already be in the
|
||||
agent's truncated set — the story suite boots the app under
|
||||
test with a ``BOR_READ_MAX_CHARS`` small enough that the
|
||||
FIRST (20-line) range slice truncates, which is exactly the
|
||||
large-document premise of the owner's confusion.
|
||||
``ALREADY_IN_CONTEXT``) the document was required to already
|
||||
be in the agent's truncated set — the story suite boots the
|
||||
app under test with a ``BOR_READ_MAX_CHARS`` small enough
|
||||
that the FIRST (20-line) range slice truncates, which is
|
||||
exactly the large-document premise of the owner's confusion.
|
||||
Phase 141 RETIRED that premise: the coverage-tracked dedupe
|
||||
proceeds with a second range read whenever its lines are not
|
||||
fully covered — the suite's small cap stays as the
|
||||
large-document premise, but is no longer LOAD-BEARING for the
|
||||
second read to proceed.
|
||||
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
|
||||
phrases — the phase-71/72/94/118/131 ordering convention; the
|
||||
trigger needs the ``<tools>`` section, so deflected turns never
|
||||
hit it); verified 2026-09-30: no existing E2E question or
|
||||
fixture file contains the phrase, so every other suite is
|
||||
unaffected.
|
||||
- user message containing ``read two ranges``
|
||||
(``TWO_RANGES_TRIGGER``, phase 141 task 04 — the read
|
||||
range-dedupe fix's dedicated story suite
|
||||
``tests/e2e/test_read_range_dedupe.py``) **and** the system
|
||||
prompt carries the ``<tools>`` section -> the owner's stuck
|
||||
scenario (TODO.md L3), end to end: the model range-reads ONE
|
||||
document in TWO different 100-line ranges — pre-phase the
|
||||
second read was refused ``ALREADY_IN_CONTEXT`` ("the document
|
||||
is already in my context, but I haven't actually seen the full
|
||||
text") while ONLY the first slice was in the prompt; the
|
||||
coverage-tracked server proceeds with it, and the answer
|
||||
echoes the SECOND range's content (the marker line lives in
|
||||
range B, so an echo of it proves the slice reached the model).
|
||||
The question carries its own target after the colon (``read
|
||||
two ranges: read source/path`` — the question-carried-call
|
||||
convention of ``READ_CAP_TRIGGER`` / ``SAME_FILE_TRIGGER``),
|
||||
and the step is then discriminated statelessly by the tool
|
||||
results in the messages (the house count pattern — streaming
|
||||
only):
|
||||
* request 1 (``tools`` offered, no tool results yet): a
|
||||
``tool_calls`` delta — ``read`` on the parsed target with
|
||||
``offset`` = ``TWO_RANGES_FIRST_OFFSET`` (1) and ``limit``
|
||||
= ``TWO_RANGES_FIRST_LIMIT`` (100) (id ``call_0``);
|
||||
* EXACTLY ONE ``tool``-role read result is in the messages
|
||||
(content starting with the agent's ``"Document "`` prefix —
|
||||
``_READ_RESULT_PREFIX``) AND it carries the ``lines: 1-100
|
||||
of …`` extent header (the mock's own keying, asserted):
|
||||
the SECOND range read — a ``tool_calls`` delta — ``read``
|
||||
on the SAME target with ``offset`` =
|
||||
``TWO_RANGES_SECOND_OFFSET`` (101) and ``limit`` =
|
||||
``TWO_RANGES_SECOND_LIMIT`` (100) (id ``call_1``) — the
|
||||
flow only works if the second read PROCEEDS (the pre-phase
|
||||
server would have refused it);
|
||||
* TWO or more read results are in the messages and the
|
||||
SECOND carries the ``lines: 101-…`` header (asserted the
|
||||
same way — the mock's keying stays honest): the forced
|
||||
answer, byte-stable, in the existing read-echo shape —
|
||||
``Both parts read. <first 80 chars of the second read
|
||||
result's content, header lines skipped>`` (the
|
||||
:func:`_same_file_quote` convention — the RANGE reached
|
||||
the model's context);
|
||||
* a tool result is in the messages that is not a well-keyed
|
||||
read result for the step (a refusal — the second range
|
||||
read was refused, the pre-phase bug — or an unkeyed extent
|
||||
header): the deterministic broken-premise echo — the
|
||||
offending result VERBATIM under a ``Premise broke:`` line
|
||||
— so a suite fails loudly on it (the read-cap flow's
|
||||
broken-premise convention).
|
||||
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint
|
||||
trigger phrases — the phase-71/72/94/118/131/139 ordering
|
||||
convention; the trigger needs the ``<tools>`` section, so
|
||||
deflected turns never hit it); verified 2026-10-01: no existing
|
||||
E2E question or fixture file contains the phrase, so every
|
||||
other suite is unaffected.
|
||||
|
||||
Failure injection (phase 67, LLM retry, TODO.md L3) — deterministic
|
||||
dead-endpoint behavior for the retry E2E suite (``tests/e2e/
|
||||
@@ -2465,6 +2524,149 @@ def _same_file_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
return ("read1", target)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 141 (task 04): the owner's stuck scenario — the model
|
||||
# range-reads ONE document in TWO different ranges; the
|
||||
# coverage-tracked server proceeds with the second (the pre-phase
|
||||
# pair-set dedupe refused it ``ALREADY_IN_CONTEXT`` while only the
|
||||
# first slice was in the prompt). See the module docstring.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: A user message containing this substring (case-insensitive) —
|
||||
#: combined with the ``<tools>`` section in the system prompt — drives
|
||||
#: the deterministic TWO-RANGES flow (the read range-dedupe fix's
|
||||
#: story suite, ``tests/e2e/test_read_range_dedupe.py`` — the
|
||||
#: owner's stuck scenario, TODO.md L3). Checked BEFORE the plain
|
||||
#: ``TOOLS_TRIGGER`` flow (disjoint trigger phrases — the
|
||||
#: phase-71/72/94/118/131/139 ordering convention); verified
|
||||
#: 2026-10-01: no existing E2E question or fixture file contains
|
||||
#: the phrase, so every other suite is unaffected.
|
||||
TWO_RANGES_TRIGGER = "read two ranges"
|
||||
|
||||
#: The scripted target in the two-ranges question (case-
|
||||
#: insensitive — the suite's question capitalizes the trigger's
|
||||
#: first letter): the verb (``read``) plus the target — a combined
|
||||
#: ``source/path``, parsed from the RAW user message so the target
|
||||
#: keeps its case (the ``_SAME_FILE_CALL_RE`` convention; the target
|
||||
#: is a ``[a-z0-9_./-]`` run, so a `` — `` flavor separator can
|
||||
#: never bleed into it).
|
||||
_TWO_RANGES_CALL_RE = re.compile(
|
||||
r"read two ranges:\s*read\s+(?P<arg>[a-z0-9_./-]+)",
|
||||
re.I,
|
||||
)
|
||||
|
||||
#: The FIRST range read's requested extent — range A (lines 1–100
|
||||
#: of the suite's 200-line fixture document).
|
||||
TWO_RANGES_FIRST_OFFSET = 1
|
||||
TWO_RANGES_FIRST_LIMIT = 100
|
||||
|
||||
#: The SECOND range read's requested extent — range B (lines 101–
|
||||
#: …): the SAME file again, a different range. Pre-phase the server
|
||||
#: refused this call ``ALREADY_IN_CONTEXT`` (the false "full text is
|
||||
#: in your prompt" claim — the owner's stuck turn); the coverage-
|
||||
#: tracked server proceeds with it (its lines are NOT covered by
|
||||
#: the first read's span).
|
||||
TWO_RANGES_SECOND_OFFSET = 101
|
||||
TWO_RANGES_SECOND_LIMIT = 100
|
||||
|
||||
|
||||
def _two_ranges_extent(content: str) -> str | None:
|
||||
"""The ``lines: a-b of N`` extent line of one read result's
|
||||
content (the ``Document …:`` header already stripped by
|
||||
:func:`_read_results`, the phase-106 D5 ``date:`` line skipped)
|
||||
— ``None`` when the result carries no extent line (a plain
|
||||
read). The flow keys its step on this line (the mock's own
|
||||
keying, asserted — a refusal or a differently-shaped result is
|
||||
the broken premise, never a step input).
|
||||
"""
|
||||
body = content
|
||||
if body.startswith("date: "):
|
||||
body = body.partition("\n")[2]
|
||||
if body.startswith("lines: "):
|
||||
return body.partition("\n")[0]
|
||||
return None
|
||||
|
||||
|
||||
def _two_ranges_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
"""Classify a phase-141 two-ranges request (see the module
|
||||
docstring). The question carries its own target (``read two
|
||||
ranges: read source/path``); the step is then discriminated
|
||||
statelessly by the tool results in the messages (the house count
|
||||
pattern — the flow only ever issues ``read`` calls, so counting
|
||||
read results is reliable):
|
||||
|
||||
* ``("answer", quote)`` — TWO or more ``tool``-role read results
|
||||
are in the messages and the SECOND carries the ``lines: 101-
|
||||
…`` extent header (asserted — the second range read landed as
|
||||
advertised): the forced answer, byte-stable, in the existing
|
||||
read-echo shape — ``Both parts read. <quote>`` with quote the
|
||||
second result's pure content (header lines skipped, first 80
|
||||
chars — the :func:`_same_file_quote` convention; the marker
|
||||
line lives in range B, so an echo of it proves the slice
|
||||
reached the model).
|
||||
* ``("read2", target)`` — EXACTLY ONE read result is in the
|
||||
messages and it carries the ``lines: 1-100 of …`` extent
|
||||
header (asserted — the first range read landed as advertised):
|
||||
the SECOND range read on the SAME target — ``offset`` =
|
||||
:data:`TWO_RANGES_SECOND_OFFSET`, ``limit`` =
|
||||
:data:`TWO_RANGES_SECOND_LIMIT` (id ``call_1``).
|
||||
* ``("read1", target)`` — ``tools`` are offered and no tool
|
||||
result is in the messages yet: the FIRST range read on the
|
||||
parsed target — ``offset`` = :data:`TWO_RANGES_FIRST_OFFSET`,
|
||||
``limit`` = :data:`TWO_RANGES_FIRST_LIMIT` (id ``call_0``).
|
||||
* ``("bad", result)`` — a tool result is in the messages that is
|
||||
not a well-keyed read result for the step (a refusal — the
|
||||
second range read was refused, the pre-phase bug — or an
|
||||
unkeyed extent header): the deterministic broken-premise echo
|
||||
— the offending result verbatim under ``Premise broke:`` (the
|
||||
read-cap flow's broken-premise convention — a suite fails
|
||||
loudly on it).
|
||||
* ``None`` — not the flow: the trigger is absent, the
|
||||
``<tools>`` section is missing (deflected turns never carry
|
||||
it), the target is unparseable, or ``tools`` are not offered
|
||||
and no tool result is in the messages yet (e.g.
|
||||
``agent_max_rounds=0``).
|
||||
"""
|
||||
user = _user(body)
|
||||
if TWO_RANGES_TRIGGER not in user.lower():
|
||||
return None
|
||||
if "<tools>" not in _system(body):
|
||||
return None
|
||||
match = _TWO_RANGES_CALL_RE.search(user)
|
||||
if match is None:
|
||||
return None
|
||||
target = match.group("arg")
|
||||
reads = _read_results(body)
|
||||
results = _tool_results(body)
|
||||
if len(reads) >= 2:
|
||||
extent = _two_ranges_extent(reads[1][1])
|
||||
if extent is None or not extent.startswith(
|
||||
f"lines: {TWO_RANGES_SECOND_OFFSET}-"
|
||||
):
|
||||
return ("bad", reads[1][1])
|
||||
return ("answer", _same_file_quote(reads[1][1]))
|
||||
if len(reads) == 1:
|
||||
if len(results) > 1:
|
||||
# A NON-read tool result after the read result — a
|
||||
# refusal (the pre-phase bug: the second range read was
|
||||
# refused ``ALREADY_IN_CONTEXT``): the premise broke —
|
||||
# echo it so a suite fails loudly.
|
||||
return ("bad", results[-1])
|
||||
extent = _two_ranges_extent(reads[0][1])
|
||||
if extent is None or not extent.startswith(
|
||||
f"lines: {TWO_RANGES_FIRST_OFFSET}-{TWO_RANGES_FIRST_LIMIT} of "
|
||||
):
|
||||
return ("bad", reads[0][1])
|
||||
return ("read2", target)
|
||||
if results:
|
||||
# A tool result that is NOT a read result before any read
|
||||
# landed — the first read was refused: the premise broke.
|
||||
return ("bad", results[-1])
|
||||
if not body.get("tools"):
|
||||
return None
|
||||
return ("read1", target)
|
||||
|
||||
|
||||
def long_answer() -> str:
|
||||
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
||||
a unique final line that must survive the stream untruncated."""
|
||||
@@ -3773,6 +3975,75 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 141 (task 04): the owner's stuck scenario (the read
|
||||
# range-dedupe fix's story suite,
|
||||
# tests/e2e/test_read_range_dedupe.py — TODO.md L3): the
|
||||
# model range-reads ONE document in TWO different ranges —
|
||||
# the second PROCEEDS on the coverage-tracked server (the
|
||||
# pre-phase pair-set dedupe refused it ``ALREADY_IN_
|
||||
# CONTEXT``), and the answer echoes the second range's
|
||||
# content. Checked BEFORE the plain TOOLS_TRIGGER flow
|
||||
# (disjoint trigger phrases — the phase-71/72/94/118/131/
|
||||
# 139 ordering convention; the trigger needs the ``<tools>``
|
||||
# section, so deflected turns never hit it).
|
||||
two_ranges = _two_ranges_flow(body)
|
||||
if two_ranges is not None:
|
||||
if two_ranges[0] == "read1":
|
||||
# The FIRST range read on the parsed target — range
|
||||
# A (lines 1–100 of the suite's document).
|
||||
stream = _tool_call_stream(
|
||||
"read",
|
||||
{
|
||||
"path": two_ranges[1],
|
||||
"offset": TWO_RANGES_FIRST_OFFSET,
|
||||
"limit": TWO_RANGES_FIRST_LIMIT,
|
||||
},
|
||||
"call_0",
|
||||
)
|
||||
elif two_ranges[0] == "read2":
|
||||
# The SECOND range read — the SAME file again, a
|
||||
# different range (range B, lines 101–…). It only
|
||||
# PROCEEDS on the coverage-tracked server: its lines
|
||||
# are NOT covered by the first read's span (the pre-
|
||||
# phase pair-set dedupe refused it — the owner's
|
||||
# stuck turn).
|
||||
stream = _tool_call_stream(
|
||||
"read",
|
||||
{
|
||||
"path": two_ranges[1],
|
||||
"offset": TWO_RANGES_SECOND_OFFSET,
|
||||
"limit": TWO_RANGES_SECOND_LIMIT,
|
||||
},
|
||||
"call_1",
|
||||
)
|
||||
elif two_ranges[0] == "bad":
|
||||
# The premise broke (a refusal — the second range
|
||||
# read was refused — or an unkeyed extent header):
|
||||
# echo the offending result verbatim so a suite
|
||||
# fails loudly on it (the read-cap flow's broken-
|
||||
# premise convention).
|
||||
stream = _sse_stream(
|
||||
_apply_max_tokens(
|
||||
f"Premise broke:\n{two_ranges[1]}",
|
||||
body.get("max_tokens"),
|
||||
),
|
||||
0.0,
|
||||
)
|
||||
else: # "answer" — the forced read-echo of range B
|
||||
# (the marker line lives in range B — the echo of it
|
||||
# proves the second slice reached the model).
|
||||
stream = _sse_stream(
|
||||
_apply_max_tokens(
|
||||
f"Both parts read. {two_ranges[1]}",
|
||||
body.get("max_tokens"),
|
||||
),
|
||||
0.0,
|
||||
)
|
||||
return StreamingResponse(
|
||||
stream,
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 124 (task 07, image chips and answers, TODO L3): the
|
||||
# scripted SHOW_IMAGE flow — the mock shows the seeded image
|
||||
# document (the ``<documents>`` block with the image-
|
||||
|
||||
@@ -0,0 +1,665 @@
|
||||
"""Phase 141 task 04 E2E (Playwright, mock-only): the read
|
||||
range-dedupe fix's dedicated story suite — the owner's stuck
|
||||
scenario, end to end through the SSE stream.
|
||||
|
||||
TODO.md L3: "From a chat I was having, the LLM is trying to read
|
||||
different ranges of a document and keeps saying: 'The system is
|
||||
telling me the document is already in my context, but I haven't
|
||||
actually seen the full text … The read calls keep returning
|
||||
"Already in your context"…' This seems like a critical bug, fix
|
||||
it." Pre-phase, ``AgentHolder``'s pair-set dedupe recorded ANY
|
||||
successful read — including a RANGE read — as if the FULL document
|
||||
were in context, so a second read of a DIFFERENT range of the same
|
||||
document was refused with a claim that was false. Phase 141 tracks
|
||||
the covered line SPANS (``AgentHolder.read_coverage``) and refuses
|
||||
only genuinely-covered re-reads (tasks 01–03, unit/integration
|
||||
pinned) — this suite is the browser-visible proof: the
|
||||
deterministic mock LLM drives a two-range turn, and the assertions
|
||||
show both ``read`` tool frames succeeding and the final answer
|
||||
carrying content from the SECOND range.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_read_range_dedupe.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — this story's
|
||||
gate is the deterministic SCRIPTED flow in
|
||||
``tests/e2e/mock_llm.py`` (the mock is the only E2E lens on the
|
||||
LLM's context):
|
||||
|
||||
* the two-range turn — ``TWO_RANGES_TRIGGER``: a user message
|
||||
containing ``read two ranges`` **and** the system prompt carrying
|
||||
the HIGH prompt's ``<tools>`` section. Request 1 reads
|
||||
``offset=1, limit=100`` (range A) on the question-carried target
|
||||
(``read two ranges: read source/path``); request 2 (one
|
||||
``tool``-role read result in the messages, carrying the
|
||||
``lines: 1-100 of …`` extent header — the mock's own keying,
|
||||
asserted) issues the SECOND read ``offset=101, limit=100`` (range
|
||||
B) on the SAME target — pre-phase the server refused it
|
||||
``ALREADY_IN_CONTEXT`` while only range A was in the prompt; the
|
||||
flow only works if the second read PROCEEDS. Request 3 (two read
|
||||
results in the messages, the second carrying the ``lines: 101-…``
|
||||
header — asserted the same way) answers ``Both parts read.
|
||||
<first 80 chars of the second read result's content, header lines
|
||||
skipped>`` (the existing read-echo shape) — the marker line lives
|
||||
in range B, so an echo of it proves the slice reached the model.
|
||||
A broken premise (a refusal in the messages — the pre-phase bug —
|
||||
or an unkeyed extent header) lands as the deterministic
|
||||
``Premise broke:`` echo, so a suite fails loudly on it.
|
||||
* the single-range turn — the mock's EXISTING ``READ_LINES_TRIGGER``
|
||||
flow (``read the lines``, phase 131 — the grep →
|
||||
read(offset/limit) dance) reused on THIS suite's KB: the
|
||||
single-range behavior is unchanged by the coverage rework (the
|
||||
phase-131 expectation still passes against the coverage-tracked
|
||||
server).
|
||||
|
||||
The app under test keeps the DEFAULT read cap (128 000 chars) —
|
||||
deliberately: NO read truncates in this suite. That is the point —
|
||||
pre-phase, a second range read of a NON-truncated document already
|
||||
in the pair set was refused ``ALREADY_IN_CONTEXT`` (the phase-139
|
||||
suite had to shrink ``BOR_READ_MAX_CHARS`` until the first range
|
||||
slice truncated just to let a second range read through — that
|
||||
workaround is retired by the coverage tracking). The still-refused
|
||||
row (a re-read of FULLY-covered lines) is unit-pinned in tasks
|
||||
02/03 (``tests/unit/test_agent.py`` — the dedupe matrix); this
|
||||
suite's job is the HAPPY two-range path, browser-visible.
|
||||
|
||||
KB fixture — ONE deterministic 200-line text document
|
||||
(``tworanges/two.txt``): fixed-length filler lines (``Line <i>: …``
|
||||
— the line numbers are exact) carrying THREE marker lines —
|
||||
``TWO_RANGES_A_LINE …`` at line 50 (range A), ``TWO_RANGES_B_LINE …``
|
||||
at line 101 (the FIRST line of range B — the read-echo quote is the
|
||||
slice's first 80 chars, so the marker must sit at the range's start
|
||||
to be echoed), and ``LINE_MARKER …`` at line 30 (the phase-131
|
||||
flow's grep target — exactly one line carries the pattern). Each
|
||||
marker line has two contextual neighbor lines repeating its
|
||||
distinctive words (the mock's token-overlap embeddings are
|
||||
diluted by the ~19-line chunks — the neighbors keep the marker
|
||||
chunk's cosine above the mock-calibrated lexical floor, the
|
||||
house gate pattern). All line positions are computed from the very
|
||||
string written to the fixture (never a literal in the
|
||||
assertions), so the suite stays true if the fixture is regenerated.
|
||||
|
||||
The app under test is mock-calibrated (conftest pattern):
|
||||
``BOR_RELEVANCE_THRESHOLD=0.30`` + ``BOR_LEXICAL_SUPPORT_FLOOR=0.15``
|
||||
+ ``BOR_SOURCE_USEFULNESS_FLOOR=0.15`` — both questions' best-chunk
|
||||
cosine (the marker chunk wins — ~0.23 / ~0.19 on the mock's
|
||||
token-overlap distribution, deterministic) clears the floor with
|
||||
FTS hits (``alpha``/``beta``/``beacon`` and ``trap``/``components``
|
||||
in the marker chunks), so every scripted turn stays GROUNDED (the
|
||||
``<tools>`` section is present) and the read document clears the
|
||||
usefulness bar into ``done.sources``.
|
||||
|
||||
Test → observable mapping (Playwright Mapping Rule; the story is
|
||||
the owner's TODO item — one Playwright file per story, A16):
|
||||
1. ``test_two_ranges_both_reach_the_model`` — the ``read two
|
||||
ranges`` question → the SSE ``tool`` frames are EXACTLY the two
|
||||
range reads of the SAME path (``offset`` 1/``limit`` 100, then
|
||||
``offset`` 101/``limit`` 100 — asserted on the frames' ARGS; a
|
||||
refused second read would re-issue the call and add a THIRD
|
||||
frame), NO ``tool_result`` frames (nothing truncated — the
|
||||
default cap), the ``done`` frame is non-deflected with the read
|
||||
document cited (phase 119 A1), the final answer carries
|
||||
``Both parts read.`` + the ``TWO_RANGES_B_LINE`` marker text
|
||||
(the second slice reached the model's context), and NO refusal
|
||||
surface anywhere (neither the ``ALREADY_IN_CONTEXT`` line nor
|
||||
the mock's broken-premise echo is in the answer).
|
||||
2. ``test_single_range_flow_unchanged`` — the phase-131 ``read the
|
||||
lines`` question on this KB → the SSE ``tool`` frames are the
|
||||
grep (the canned pattern) then the range read (``offset`` = the
|
||||
marker's true line, ``limit`` = 30) — the phase-131 expectation
|
||||
unchanged against the coverage-tracked server; the final answer
|
||||
is the grounded echo ``Found it at line <N>`` + the marker text.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Locator, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document
|
||||
from app.rag.agent import ALREADY_IN_CONTEXT
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
from e2e.mock_llm import READ_LINES_LIMIT, READ_LINES_PATTERN
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns
|
||||
# its port in a combined run — this module app binds its own port
|
||||
# instead (a same-port second uvicorn dies on bind and would drive
|
||||
# the wrong server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_TWORANGES", "8156"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
SOURCE = "tworanges" # the local directory's basename = the source name
|
||||
REL = "two.txt"
|
||||
SP = f"{SOURCE}/{REL}" # the combined read identity
|
||||
|
||||
# The fixture's layout (deterministic — see the module docstring):
|
||||
# 200 fixed-length filler lines (60 chars each) carrying three
|
||||
# marker lines, each flanked by two contextual neighbor lines that
|
||||
# repeat its distinctive words (the mock-calibrated gate — the
|
||||
# marker chunk's cosine must clear the 0.15 lexical floor).
|
||||
TOTAL_LINES = 200
|
||||
A_MARKER_LINE = 50 # range A (lines 1–100)
|
||||
B_MARKER_LINE = 101 # range B (lines 101+): the FIRST line of range B
|
||||
LINE_MARKER_LINE = 30 # the phase-131 flow's grep target (range A)
|
||||
|
||||
#: The range A marker line (lines 1–100 carry it — NOT echoed by the
|
||||
#: mock; its presence in the fixture is the two-range premise).
|
||||
A_MARKER_TEXT = (
|
||||
"TWO_RANGES_A_LINE the first range documents the alpha beacon payload"
|
||||
)
|
||||
A_CTX_ABOVE = (
|
||||
"the alpha beacon table starts here: alpha beacon payloads, documented below"
|
||||
)
|
||||
A_CTX_BELOW = (
|
||||
"the alpha beacon entry ends here — alpha beacon payload documented above"
|
||||
)
|
||||
|
||||
#: The range B marker line — the FIRST line of range B, so the
|
||||
#: mock's read-echo quote (the second read result's first 80 chars,
|
||||
#: header lines skipped) carries it: an echo of it proves the second
|
||||
#: slice reached the model.
|
||||
B_MARKER_TEXT = (
|
||||
"TWO_RANGES_B_LINE the second range documents the beta beacon payload"
|
||||
)
|
||||
B_CTX_ABOVE = (
|
||||
"the beta beacon table starts here: beta beacon payloads, documented below"
|
||||
)
|
||||
B_CTX_BELOW = (
|
||||
"the beta beacon entry ends here — beta beacon payload documented above"
|
||||
)
|
||||
|
||||
#: The phase-131 flow's marker line: exactly ONE line in the document
|
||||
#: carries ``READ_LINES_PATTERN`` (imported — the mock can never
|
||||
#: drift from the fixture). Its true line is computed from the very
|
||||
#: string written to the fixture (the determinism pin).
|
||||
LINE_MARKER_TEXT = "LINE_MARKER the trap components are documented here"
|
||||
LM_CTX_ABOVE = (
|
||||
"the trap components table starts here: trap components, documented below"
|
||||
)
|
||||
LM_CTX_BELOW = (
|
||||
"the trap components entry ends here — trap components documented above"
|
||||
)
|
||||
|
||||
_FILLER_WORDS = "quiet mirror vault raid pool chain slot cycle guard probe audit "
|
||||
|
||||
|
||||
def _filler_line(i: int) -> str:
|
||||
"""One fixed-length (60-char) filler line — the line numbers are
|
||||
exact and every line is the SAME length, so the range slices'
|
||||
lengths are deterministic."""
|
||||
prefix = f"Line {i:04d}: "
|
||||
return prefix + (_FILLER_WORDS * 5)[: 60 - len(prefix)]
|
||||
|
||||
|
||||
def _doc() -> str:
|
||||
"""The 200-line fixture document (deterministic — module
|
||||
docstring)."""
|
||||
lines: list[str] = []
|
||||
for i in range(1, TOTAL_LINES + 1):
|
||||
if i == LINE_MARKER_LINE - 1:
|
||||
lines.append(LM_CTX_ABOVE)
|
||||
elif i == LINE_MARKER_LINE:
|
||||
lines.append(LINE_MARKER_TEXT)
|
||||
elif i == LINE_MARKER_LINE + 1:
|
||||
lines.append(LM_CTX_BELOW)
|
||||
elif i == A_MARKER_LINE - 1:
|
||||
lines.append(A_CTX_ABOVE)
|
||||
elif i == A_MARKER_LINE:
|
||||
lines.append(A_MARKER_TEXT)
|
||||
elif i == A_MARKER_LINE + 1:
|
||||
lines.append(A_CTX_BELOW)
|
||||
elif i == B_MARKER_LINE - 1:
|
||||
lines.append(B_CTX_ABOVE)
|
||||
elif i == B_MARKER_LINE:
|
||||
lines.append(B_MARKER_TEXT)
|
||||
elif i == B_MARKER_LINE + 1:
|
||||
lines.append(B_CTX_BELOW)
|
||||
else:
|
||||
lines.append(_filler_line(i))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
DOC = _doc()
|
||||
DOC_LINES = DOC.splitlines()
|
||||
DOC_TOTAL = len(DOC)
|
||||
assert len(DOC_LINES) == TOTAL_LINES, len(DOC_LINES)
|
||||
|
||||
# The determinism pins (computed from the very string written to the
|
||||
# fixture — the assertions use these, never literals): the markers
|
||||
# are unique, the B marker is the FIRST line of range B (the
|
||||
# read-echo quote reaches it), and the range slices are UNDER the
|
||||
# default read cap (nothing truncates — no ``tool_result`` frames).
|
||||
assert DOC.count(LINE_MARKER_TEXT) == 1
|
||||
assert DOC.count(A_MARKER_TEXT) == 1
|
||||
assert DOC.count(B_MARKER_TEXT) == 1
|
||||
assert DOC_LINES[B_MARKER_LINE - 1] == B_MARKER_TEXT
|
||||
assert DOC_LINES[A_MARKER_LINE - 1] == A_MARKER_TEXT
|
||||
assert DOC_LINES[LINE_MARKER_LINE - 1] == LINE_MARKER_TEXT
|
||||
#: The default read cap (128 000 — ``app/config.py``) stays in force
|
||||
#: for this suite; both 100-line slices must stay under it.
|
||||
assert len("\n".join(DOC_LINES[:100])) < 128_000
|
||||
assert len("\n".join(DOC_LINES[100:200])) < 128_000
|
||||
|
||||
# The expected answer fragments (the mock's read-echo shape —
|
||||
# ``Both parts read. <quote>`` with quote the second read result's
|
||||
# content, the ``date:`` / ``lines:`` header lines skipped, first 80
|
||||
# chars). The B marker line (70 chars) IS the slice's first line, so
|
||||
# the full marker text sits inside the 80-char quote.
|
||||
ANSWER_LEAD = "Both parts read."
|
||||
assert len(B_MARKER_TEXT) < 80, len(B_MARKER_TEXT)
|
||||
|
||||
# The scripted questions. Q_A carries ``TWO_RANGES_TRIGGER`` + the
|
||||
# target (the question-carried-call convention — the mock parses
|
||||
# ``read two ranges: read source/path`` from the raw user message);
|
||||
# Q_B carries the phase-131 ``READ_LINES_TRIGGER`` (the grep →
|
||||
# read(offset/limit) dance) on this KB. Both carry their marker
|
||||
# chunks' distinctive words (the mock-calibrated gate stays
|
||||
# GROUNDED — the ``<tools>`` section is present, so both flows
|
||||
# fire).
|
||||
Q_A = (
|
||||
f"Read two ranges: read {SP} — "
|
||||
"what documents the alpha and beta beacon payloads?"
|
||||
)
|
||||
Q_B = f"Read the lines of {SP} — what are the trap components documented here?"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def tworanges_dirs(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The story's local-directory source: one host temp dir (the app
|
||||
server runs on the same host, so the path is visible to it)
|
||||
holding the ONE generated 200-line text document. The directory's
|
||||
basename is the source name (``kind=local``, phase 38)."""
|
||||
root = tmp_path_factory.mktemp("bor_two_ranges") / SOURCE
|
||||
root.mkdir()
|
||||
(root / REL).write_text(DOC, encoding="utf-8")
|
||||
assert (root / REL).read_text(encoding="utf-8") == DOC
|
||||
assert not (root / ".git").exists() # the local-kind story: NOT git
|
||||
return root
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, tworanges_dirs: Path) -> Iterator[str]:
|
||||
"""The real app under test — per-module env (the conftest
|
||||
pattern, cf. ``test_read_line_ranges.py``): NO
|
||||
``BOR_GIT_SOURCES`` (the env fallback is git-only — the source
|
||||
here is a DB-registered local directory), the mock LLM, the
|
||||
mock-calibrated gate (threshold 0.30 + lexical floor 0.15 — the
|
||||
conftest pair). The DEFAULT read cap stays in force — THE
|
||||
SUBJECT: no read truncates in this suite, and the second range
|
||||
read still proceeds (pre-phase, the pair-set dedupe refused a
|
||||
second range read of a NON-truncated document already in
|
||||
context — the owner's stuck turn). The session app is never
|
||||
started in this isolated run, so no port clash."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (conftest pattern): both scripted turns
|
||||
# stay GROUNDED (the questions' best-chunk cosine clears the
|
||||
# 0.15 floor with FTS hits — the marker chunks win).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
# The read document must clear the citation bar into
|
||||
# done.sources (phase 119 A1 — both turns assert it).
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
# Phase 67: instant retry waits + the code-default budget (the
|
||||
# conftest leak-guard pattern).
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's settings (the app reads
|
||||
# it from cwd) — override the source list with an EMPTY value (the
|
||||
# env var beats the .env file): the registry must hold EXACTLY the
|
||||
# local directory this suite registers (a leftover env git list
|
||||
# would pollute the KB the scripted turns run against).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||
# .env cannot leak corpus-specific settings into the app under
|
||||
# test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the E2E
|
||||
suites share one Postgres, so a leftover git_sources row or
|
||||
document would pollute the retrieval the scripted turns run
|
||||
against (the single-document KB's gate margins are pinned
|
||||
against EXACTLY this one document)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
|
||||
"""Poll the (cookie-authenticated) status endpoint until the run
|
||||
reaches a terminal state (the test_ls_tree_drilldown pattern, over
|
||||
plain httpx — this fixture has no browser page yet)."""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def synced_kb(app_server: str, tworanges_dirs: Path) -> None:
|
||||
"""The story's precondition: the one-source KB synced under the
|
||||
deterministic mock. Registers the temp directory through the
|
||||
authenticated API (the ``test_local_directory_sources.py``
|
||||
pattern), runs the REAL in-process sync (``POST /api/sync`` —
|
||||
walk → chunk → embed → overview → version bump), pins the stored
|
||||
content byte-identical to the fixture string."""
|
||||
_truncate_all()
|
||||
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": str(tworanges_dirs)}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
body = _wait_sync_done_http(client)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == 1, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
# The import stored the fixture string BYTE-IDENTICALLY — the
|
||||
# slice extents every assertion uses come from the very string
|
||||
# written to the directory.
|
||||
with SessionLocal() as db:
|
||||
stored = db.scalar(
|
||||
select(Document.content).where(
|
||||
Document.source == SOURCE, Document.path == REL
|
||||
)
|
||||
)
|
||||
assert stored == DOC, "stored content drifted from the fixture"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
"""Per-test query_log isolation (the KB itself is module-scoped —
|
||||
the scripted turns never change it, so the registry persists
|
||||
across the tests of this module)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the test_read_line_ranges / test_read_range_tool_lines
|
||||
# house patterns)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions
|
||||
#: for the ``tool`` / ``tool_result`` frames, independent of the UI
|
||||
#: rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_page_hooks(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _frames(page: Page, deadline_s: float = 120.0) -> list[dict]:
|
||||
"""The SSE frames captured since the last submit (``_submit``
|
||||
clears the buffer), once the hook's background read settles."""
|
||||
deadline = time.monotonic() + deadline_s
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _result_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool_result"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.evaluate("window.__sseFrames = []")
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page, timeout_ms: int = 120_000) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button
|
||||
recovered (the phase-48 settle wait)."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text(
|
||||
"", timeout=timeout_ms
|
||||
)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=timeout_ms)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=timeout_ms)
|
||||
|
||||
|
||||
def _last_brain(page: Page) -> Locator:
|
||||
return page.locator(".msg.brain").last
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. The owner's stuck scenario, browser-visible: two DIFFERENT ranges
|
||||
# of the same document both reach the model (no refusal)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_two_ranges_both_reach_the_model(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(60_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, Q_A)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
# THE SUBJECT on the wire — IN ORDER, asserted on the frames'
|
||||
# ARGS (not a substring): the SAME path, TWO distinct ranges
|
||||
# (offset 1/limit 100, then offset 101/limit 100 — the phase-131
|
||||
# additive fields ride the SAME frame). EXACTLY two: a refused
|
||||
# second read (the pre-phase bug) would re-issue the call and
|
||||
# add a THIRD frame — or land as the mock's broken-premise echo.
|
||||
assert _tool_frames(frames) == [
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "read",
|
||||
"argument": SP,
|
||||
"offset": 1,
|
||||
"limit": 100,
|
||||
},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "read",
|
||||
"argument": SP,
|
||||
"offset": 101,
|
||||
"limit": 100,
|
||||
},
|
||||
], _tool_frames(frames)
|
||||
# NO ``tool_result`` frame: nothing truncated (the default cap —
|
||||
# both 100-line slices are far under it; one frame = one
|
||||
# noteworthy event, the phase-95 additive contract).
|
||||
assert _result_frames(frames) == [], _result_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# The read document is the turn's cited source (phase 119 A1).
|
||||
assert any(
|
||||
s["path"] == REL and s["source"] == SOURCE for s in done["sources"]
|
||||
), done["sources"]
|
||||
|
||||
# The GROUNDED echo — the SECOND range reached the model's
|
||||
# context and landed in the answer: ``Both parts read.`` + the
|
||||
# ``TWO_RANGES_B_LINE`` marker text (the marker is range B's
|
||||
# first line — inside the 80-char read-echo quote).
|
||||
text = _last_brain(page).locator(".bubble").text_content() or ""
|
||||
assert ANSWER_LEAD in text, text
|
||||
assert B_MARKER_TEXT in text, text
|
||||
# NO refusal surface anywhere in the rendered answer: the
|
||||
# ``ALREADY_IN_CONTEXT`` line (the pre-phase false claim — its
|
||||
# full text is the refusal the owner quoted) and the mock's
|
||||
# broken-premise echo are both absent.
|
||||
assert ALREADY_IN_CONTEXT not in text, text
|
||||
assert "Premise broke" not in text, text
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. The single-range behavior is unchanged: the phase-131 grep →
|
||||
# read(offset/limit) dance still lands on the coverage-tracked
|
||||
# server (the phase-131 expectation, this KB)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_single_range_flow_unchanged(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(60_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, Q_B)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
# The phase-131 dance on the wire — IN ORDER, asserted on the
|
||||
# frames' ARGS: the grep (the mock's canned pattern), then the
|
||||
# range read — path = the doc's combined identity, offset = the
|
||||
# marker's TRUE line (the mock parsed it from the grep result's
|
||||
# first match line), limit = READ_LINES_LIMIT (30).
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "grep", "argument": READ_LINES_PATTERN},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "read",
|
||||
"argument": SP,
|
||||
"offset": LINE_MARKER_LINE,
|
||||
"limit": READ_LINES_LIMIT,
|
||||
},
|
||||
], _tool_frames(frames)
|
||||
# The 30-line range slice did NOT truncate (the default cap) —
|
||||
# no ``tool_result`` frame (the phase-131 pinned order contract).
|
||||
assert _result_frames(frames) == [], _result_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
assert any(
|
||||
s["path"] == REL and s["source"] == SOURCE for s in done["sources"]
|
||||
), done["sources"]
|
||||
|
||||
# The GROUNDED echo — the phase-131 answer shape unchanged:
|
||||
# ``Found it at line <N>`` (N == the marker's true line) + the
|
||||
# marker text itself (the single-range read reached the model's
|
||||
# context exactly as before the coverage rework).
|
||||
text = _last_brain(page).locator(".bubble").text_content() or ""
|
||||
assert f"Found it at line {LINE_MARKER_LINE}." in text, text
|
||||
assert LINE_MARKER_TEXT in text, text
|
||||
@@ -25,10 +25,11 @@ drives the turn with the owner's ``BOR_READ_MAX_CHARS=24000`` in force:
|
||||
handbook truncates at 24000 with the REWRITTEN phase-131 notice,
|
||||
which names ``offset``/``limit`` — the dead-end is gone), round 2
|
||||
the grep, round 3 the range read of the ALREADY-TRUNCATED document
|
||||
(the phase-131 dedupe matrix's truncated-then-range row, end-to-end
|
||||
on the real document — it PROCEEDS; the ``REREAD_TRUNCATED``
|
||||
teaching is never needed because the model went straight to the
|
||||
range), round 4 the echo answer.
|
||||
(the phase-141 COVERAGE dedupe matrix's truncated-then-range row,
|
||||
end-to-end on the real document — it PROCEEDS; the
|
||||
``REREAD_PARTIAL`` teaching — the phase-131 ``REREAD_TRUNCATED``
|
||||
line, retired phase 141 — is never needed because the model went
|
||||
straight to the range), round 4 the echo answer.
|
||||
|
||||
The expected find-traps text is DERIVED from the fixture at test time
|
||||
(extract the PDF's text with the phase-126
|
||||
@@ -67,7 +68,7 @@ from app.rag import agent
|
||||
from app.rag.agent import (
|
||||
AGENT_TOOLS,
|
||||
READ_TRUNCATION_NOTICE,
|
||||
REREAD_TRUNCATED,
|
||||
REREAD_PARTIAL,
|
||||
AgentHolder,
|
||||
run_agent,
|
||||
)
|
||||
@@ -191,7 +192,7 @@ class HandbookAgentLLM:
|
||||
* round 1 → plain ``read`` (``path``);
|
||||
* round 2 → ``grep`` (``pattern="find traps"``);
|
||||
* round 3 → ``read`` (``path``, ``offset`` = the first grep-
|
||||
reported line, ``limit`` = 40) — the dedupe matrix's
|
||||
reported line, ``limit`` = 40) — the coverage dedupe matrix's
|
||||
truncated-then-range row on the real document;
|
||||
* round 4 → the answer: the read result's content, verbatim.
|
||||
|
||||
@@ -451,9 +452,13 @@ def test_plain_read_truncates_then_range_read_answers_the_find_traps_question(
|
||||
]
|
||||
|
||||
# Round 3: the range read of the truncated doc — offset is the
|
||||
# grep-reported line (parsed), limit 40 — and the dedupe matrix
|
||||
# lets it PROCEED (no ``REREAD_TRUNCATED`` teaching anywhere in
|
||||
# the loop's tool results).
|
||||
# grep-reported line (parsed), limit 40 — and the coverage dedupe
|
||||
# matrix lets it PROCEED (no ``REREAD_PARTIAL`` teaching anywhere
|
||||
# in the loop's tool results — the partial-teaching line, re-pinned
|
||||
# from the retired phase-131 ``REREAD_TRUNCATED``, is derived from
|
||||
# the REAL document: the truncated plain read covers line 1
|
||||
# through the last line inside the cut, the total is the true line
|
||||
# count).
|
||||
assert llm.calls[1].arguments == {"pattern": PATTERN}
|
||||
grep_line = _first_grep_line(llm.requests[2][0], IDENTITY)
|
||||
read_call = llm.calls[2]
|
||||
@@ -462,7 +467,13 @@ def test_plain_read_truncates_then_range_read_answers_the_find_traps_question(
|
||||
"offset": grep_line,
|
||||
"limit": RANGE_LIMIT,
|
||||
}
|
||||
reread = REREAD_TRUNCATED.format(shown=READ_CAP, source=SOURCE, path=PDF_NAME)
|
||||
covered_lines = len(doc.content[:READ_CAP].splitlines())
|
||||
reread = REREAD_PARTIAL.format(
|
||||
covered=f"1–{covered_lines}",
|
||||
source=SOURCE,
|
||||
path=PDF_NAME,
|
||||
total=len(doc.content.splitlines()),
|
||||
)
|
||||
tool_results = [
|
||||
m["content"]
|
||||
for messages, _tools in llm.requests
|
||||
|
||||
@@ -0,0 +1,536 @@
|
||||
"""Integration: phase 141 (task 03) — the owner's stuck two-range
|
||||
scenario, proven end to end.
|
||||
|
||||
The exact incident from ``TODO.md`` L3: the model was reading
|
||||
different RANGES of one large document (the D&D "Hail of Thorns"
|
||||
spell entry) and every second range dead-ended with
|
||||
``ALREADY_IN_CONTEXT`` — "The system is telling me the document is
|
||||
already in my context, but I haven't actually seen the full text …
|
||||
The read calls keep returning 'Already in your context'". Pre-phase,
|
||||
the pair-set dedupe recorded ANY successful read (a range read
|
||||
included) as if the FULL document were in context, so every attempt
|
||||
to read a DIFFERENT range was refused with a false claim. Phase 141
|
||||
(the coverage rework, tasks 01/02) makes the dedupe decide from the
|
||||
lines each read actually put in the model's context: a range read
|
||||
PROCEEDS unless its lines are fully covered, a plain re-read of a
|
||||
partially-covered doc gets the pinned
|
||||
:data:`~app.rag.agent.REREAD_PARTIAL` teaching (the covered span(s)
|
||||
named) instead of the false full-text claim.
|
||||
|
||||
This suite locks the fix as two scripted reproductions over ONE
|
||||
generated text document (200 lines, two distinct marker lines far
|
||||
apart — the content is GENERATED in the test, no binary fixture: the
|
||||
untracked owner PDF belongs to the separate phase-131 handbook
|
||||
repro, untouched). The document is inserted directly through the
|
||||
house KB pattern (``test_agent_tools.py`` — the ``documents`` table
|
||||
truncated around the test, so the read resolves against exactly this
|
||||
doc; the agent's read path only reads ``documents``), and a scripted
|
||||
fake LLM (the ``tests/unit/test_agent.py`` round-shape pattern)
|
||||
drives ``run_agent``'s loop against the real DB:
|
||||
|
||||
* the **pure dedupe variant** (the doc UNDER the read cap — no
|
||||
truncation anywhere): round 1 ``read(path, offset=1, limit=40)``
|
||||
(range A), round 2 ``read(path, offset=<marker B's line − 10>,
|
||||
limit=30)`` (range B — the PRE-PHASE code refused THIS round with
|
||||
``ALREADY_IN_CONTEXT``), round 3 the EXACT re-read of range A
|
||||
(refused — now with a claim that is true: those lines ARE in the
|
||||
prompt), round 4 the answer echoing the marker lines the delivered
|
||||
tool results carry (the read-echo convention — the model can only
|
||||
echo what reached it; pre-phase, the echo could contain only
|
||||
marker A).
|
||||
* the **truncated variant** (the doc OVER the cap — the phase-131
|
||||
truncated-then-range row, now via coverage): round 1 a plain
|
||||
``read(path)`` (truncated at a cap DERIVED from the generated
|
||||
content — just past marker A's line + a mid-line cut of the next
|
||||
line — and it covers only its cut span), round 2 the SAME range-B
|
||||
read (PROCEEDS — the truncated first read covered only its first
|
||||
lines), round 3 a PLAIN re-read (the generalized teaching —
|
||||
:data:`REREAD_PARTIAL` naming the covered span; the retired
|
||||
phase-131 char-based ``REREAD_TRUNCATED`` line is gone), round 4
|
||||
the echo answer (both markers — the truncated first read carried
|
||||
marker A, the range read carried marker B).
|
||||
|
||||
The marker A line sits INSIDE range A (lines 1–40) — the task's
|
||||
"near line 50" hint is unsatisfiable with the pinned round shape
|
||||
(``offset=1, limit=40``) and the pinned echo assertion (the answer
|
||||
must echo marker A, and the model can only echo what reached it):
|
||||
line 30 keeps it ten lines clear of range A's end while the markers
|
||||
stay far apart (120 lines).
|
||||
|
||||
Requires: podman compose up -d db
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import uuid
|
||||
from collections.abc import AsyncIterator, Iterator
|
||||
from copy import deepcopy
|
||||
from datetime import UTC, datetime
|
||||
from typing import TYPE_CHECKING, Any, cast
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document
|
||||
from app.rag import agent
|
||||
from app.rag.agent import (
|
||||
READ_TRUNCATION_NOTICE,
|
||||
AgentHolder,
|
||||
run_agent,
|
||||
)
|
||||
from app.rag.llm import (
|
||||
LLMClient,
|
||||
RetryPiece,
|
||||
StreamPiece,
|
||||
ToolCallPiece,
|
||||
ToolResultPiece,
|
||||
)
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from app.rag.scaffolding import ScaffoldingFilter
|
||||
|
||||
#: The fixture document's fixed creation date (phase 106, D5) — the
|
||||
#: ``read`` result's second line formats its UTC date part; a fixed
|
||||
#: value keeps the pins deterministic.
|
||||
_FIXTURE_CREATED_AT = datetime(2024, 6, 15, 12, 0, 0, tzinfo=UTC)
|
||||
|
||||
#: The document's identity (the house KB pattern: one source, one
|
||||
#: doc — the owner's spell entry).
|
||||
SOURCE = "Spellbook"
|
||||
DOC_PATH = "hail-of-thorns.txt"
|
||||
IDENTITY = f"{SOURCE}/{DOC_PATH}"
|
||||
|
||||
#: The generated document's shape (see :func:`build_two_range_content`).
|
||||
TOTAL_LINES = 200
|
||||
#: Range A (round 1/3 of the pure-dedupe variant): lines 1–40.
|
||||
RANGE_A_OFFSET = 1
|
||||
RANGE_A_LIMIT = 40
|
||||
#: Range B (round 2 of both variants): 30 lines starting 10 lines
|
||||
#: before marker B (``offset = marker B's line − 10``).
|
||||
RANGE_B_LIMIT = 30
|
||||
#: The marker lines' positions (marker A inside range A — see the
|
||||
#: module docstring for why the task's "near line 50" hint cannot
|
||||
#: stand with the pinned round shape + echo assertion).
|
||||
MARKER_A_LINE = 30
|
||||
MARKER_B_LINE = 150
|
||||
|
||||
#: The two distinct marker lines (each a complete line of the
|
||||
#: generated document — the echo collects whole lines).
|
||||
MARKER_A = "MARKER_A_hail-of-thorns-spell-entry-start"
|
||||
MARKER_B = "MARKER_B_hail-of-thorns-spell-entry-end"
|
||||
|
||||
#: How far past marker A's line the truncated variant's cut runs
|
||||
#: (see :func:`_truncated_first_cap`): the newline + eight chars of
|
||||
#: the following line — a MID-LINE cut (the counted-from-truth rule
|
||||
#: the code uses).
|
||||
TRUNCATED_FIRST_CUT_TAIL = 8
|
||||
|
||||
#: The turn's user message (the owner's question in spirit).
|
||||
QUESTION = "What does the Hail of Thorns spell entry say?"
|
||||
|
||||
|
||||
def build_two_range_content() -> str:
|
||||
"""The generated document (no binary fixture): exactly
|
||||
:data:`TOTAL_LINES` filler lines with the two marker lines in
|
||||
place — marker A on line :data:`MARKER_A_LINE` (inside range A —
|
||||
the round-1 slice must carry it for the echo convention to prove
|
||||
range A reached the model) and marker B on line
|
||||
:data:`MARKER_B_LINE` (the round-2 slice starts 10 lines before
|
||||
it)."""
|
||||
lines = []
|
||||
for i in range(1, TOTAL_LINES + 1):
|
||||
if i == MARKER_A_LINE:
|
||||
lines.append(MARKER_A)
|
||||
elif i == MARKER_B_LINE:
|
||||
lines.append(MARKER_B)
|
||||
else:
|
||||
lines.append(f"line {i:03d} filler filler filler")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _settings(**kwargs: Any) -> Settings:
|
||||
"""The turn's settings: ``_env_file=None`` keeps an operator's
|
||||
local ``.env`` out of the pins; ``read_max_chars`` at the default
|
||||
(the doc fits far under it) unless the test overrides it."""
|
||||
kwargs["_env_file"] = None
|
||||
return Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _truncated_first_cap(content: str) -> int:
|
||||
"""The truncated variant's read cap, DERIVED from the generated
|
||||
*content* (the no-hardcoded-offsets rule): just past marker A's
|
||||
line — the marker must sit fully INSIDE the cut (the echo of both
|
||||
markers needs it in the truncated first read; the marker lines
|
||||
are longer than the filler lines, so a literal char cap would
|
||||
drift) — plus :data:`TRUNCATED_FIRST_CUT_TAIL` chars of the
|
||||
following line (a mid-line cut — the counted-from-truth rule)."""
|
||||
return content.index(MARKER_A) + len(MARKER_A) + 1 + TRUNCATED_FIRST_CUT_TAIL
|
||||
|
||||
|
||||
class TwoRangeAgentLLM:
|
||||
"""The scripted two-range loop (the ``test_agent.py`` round-shape
|
||||
pattern) — deterministic; both variants share one class:
|
||||
|
||||
``truncated_first=False`` (the pure dedupe variant):
|
||||
|
||||
* round 1 → ``read(path, offset=1, limit=40)`` (range A);
|
||||
* round 2 → ``read(path, offset=<marker B's line − 10>,
|
||||
limit=30)`` (range B — the pre-phase false refusal);
|
||||
* round 3 → ``read(path, offset=1, limit=40)`` again (the exact
|
||||
re-covered range A);
|
||||
* round 4 → the answer: the marker lines the delivered tool
|
||||
results carry, in delivery order (the read-echo convention —
|
||||
the model can only echo what reached it).
|
||||
|
||||
``truncated_first=True`` (the truncated variant):
|
||||
|
||||
* round 1 → plain ``read(path)`` (truncates at the cap);
|
||||
* round 2 → the same range-B read (proceeds via coverage);
|
||||
* round 3 → a plain ``read(path)`` (the ``REREAD_PARTIAL``
|
||||
teaching);
|
||||
* round 4 → the echo answer.
|
||||
|
||||
Every ``chat_stream`` request's messages and tools are recorded
|
||||
(``requests``); the scripted :class:`ToolCallPiece` s, in call
|
||||
order, in ``calls``. The echo answer is fed through the round's
|
||||
:class:`ScaffoldingFilter` exactly like
|
||||
``LLMClient.chat_stream`` (the ``test_agent.py`` ScriptedLLM
|
||||
pattern)."""
|
||||
|
||||
def __init__(
|
||||
self, identity: str, *, truncated_first: bool, range_b_offset: int
|
||||
) -> None:
|
||||
self.identity = identity
|
||||
self.truncated_first = truncated_first
|
||||
self.range_b_offset = range_b_offset
|
||||
self.requests: list[
|
||||
tuple[list[dict[str, Any]], list[dict[str, Any]] | None]
|
||||
] = []
|
||||
self.calls: list[ToolCallPiece] = []
|
||||
|
||||
def _next_call(self, index: int) -> ToolCallPiece | None:
|
||||
"""The scripted call for request *index* — ``None`` on the
|
||||
final (echo) round."""
|
||||
range_b = {
|
||||
"path": self.identity,
|
||||
"offset": self.range_b_offset,
|
||||
"limit": RANGE_B_LIMIT,
|
||||
}
|
||||
if not self.truncated_first:
|
||||
if index == 0:
|
||||
return ToolCallPiece(
|
||||
id="call_1",
|
||||
name="read",
|
||||
arguments={
|
||||
"path": self.identity,
|
||||
"offset": RANGE_A_OFFSET,
|
||||
"limit": RANGE_A_LIMIT,
|
||||
},
|
||||
)
|
||||
if index == 1:
|
||||
return ToolCallPiece(id="call_2", name="read", arguments=range_b)
|
||||
if index == 2:
|
||||
return ToolCallPiece(
|
||||
id="call_3",
|
||||
name="read",
|
||||
arguments={
|
||||
"path": self.identity,
|
||||
"offset": RANGE_A_OFFSET,
|
||||
"limit": RANGE_A_LIMIT,
|
||||
},
|
||||
)
|
||||
return None
|
||||
if index == 0:
|
||||
return ToolCallPiece(id="call_1", name="read", arguments={"path": self.identity})
|
||||
if index == 1:
|
||||
return ToolCallPiece(id="call_2", name="read", arguments=range_b)
|
||||
if index == 2:
|
||||
return ToolCallPiece(id="call_3", name="read", arguments={"path": self.identity})
|
||||
return None
|
||||
|
||||
async def chat_stream(
|
||||
self,
|
||||
messages: list[dict[str, str]],
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
scaffolding: ScaffoldingFilter | None = None,
|
||||
) -> AsyncIterator[StreamPiece | ToolCallPiece]:
|
||||
self.requests.append((deepcopy(messages), deepcopy(tools)))
|
||||
call = self._next_call(len(self.requests) - 1)
|
||||
if call is not None:
|
||||
self.calls.append(call)
|
||||
yield call
|
||||
return
|
||||
# The echo round: the marker lines the DELIVERED tool results
|
||||
# carry, in delivery order — the model can only echo what
|
||||
# reached it (pre-phase, range B's slice never did, so the
|
||||
# echo could contain only marker A).
|
||||
marker_lines = [
|
||||
line
|
||||
for m in messages
|
||||
if m.get("role") == "tool" and isinstance(m.get("content"), str)
|
||||
for line in m["content"].splitlines()
|
||||
if MARKER_A in line or MARKER_B in line
|
||||
]
|
||||
content = "\n".join(marker_lines)
|
||||
if scaffolding is None:
|
||||
yield StreamPiece("content", content)
|
||||
return
|
||||
cleaned = scaffolding.feed(content)
|
||||
if cleaned:
|
||||
yield StreamPiece("content", cleaned)
|
||||
tail = scaffolding.flush()
|
||||
if tail:
|
||||
yield StreamPiece("content", tail)
|
||||
|
||||
|
||||
async def _consume(
|
||||
llm: TwoRangeAgentLLM,
|
||||
holder: AgentHolder,
|
||||
settings: Settings,
|
||||
) -> list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]:
|
||||
"""Consume one scripted ``run_agent`` turn. SEC-14-04: a
|
||||
short-lived session per tool call — the same pattern as
|
||||
production and the phase-70/131 integration suites."""
|
||||
out: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece] = []
|
||||
async for piece in run_agent(
|
||||
cast("LLMClient", llm),
|
||||
lambda: SessionLocal(),
|
||||
system_prompt="SYSTEM_PROMPT",
|
||||
user_message=QUESTION,
|
||||
seed_docs=[],
|
||||
settings=settings,
|
||||
holder=holder,
|
||||
):
|
||||
out.append(piece)
|
||||
return out
|
||||
|
||||
|
||||
def _answer(pieces: list[StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece]) -> str:
|
||||
"""The turn's final answer text (the content pieces)."""
|
||||
return "".join(
|
||||
piece.text for piece in pieces if isinstance(piece, StreamPiece) and piece.kind == "content"
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def two_range_doc(db: Session) -> Iterator[Document]:
|
||||
"""The generated two-range document, indexed as a single row
|
||||
(the house KB pattern — ``test_agent_tools.py``: the ``documents``
|
||||
table truncated around the test, so the read resolves against
|
||||
exactly this doc). Inserted directly (no importer — the content
|
||||
is generated in the test, there is no file to import, and the
|
||||
agent's read path only reads ``documents``)."""
|
||||
db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
|
||||
db.commit()
|
||||
doc = Document(
|
||||
id=uuid.uuid4(),
|
||||
source=SOURCE,
|
||||
path=DOC_PATH,
|
||||
full_path=f"/tmp/{SOURCE}/{DOC_PATH}",
|
||||
title="Hail of Thorns",
|
||||
content=build_two_range_content(),
|
||||
content_hash="0" * 64,
|
||||
created_at=_FIXTURE_CREATED_AT,
|
||||
)
|
||||
db.add(doc)
|
||||
db.commit()
|
||||
yield doc
|
||||
db.execute(text("TRUNCATE chunks, documents, folder_summaries"))
|
||||
db.commit()
|
||||
db.expire_all()
|
||||
|
||||
|
||||
def test_two_ranges_of_one_doc_both_reach_the_model(two_range_doc: Document) -> None:
|
||||
"""The owner's stuck loop, fixed — the pure dedupe case (the doc
|
||||
UNDER the read cap: no truncation anywhere): range A (lines 1–40)
|
||||
→ range B (the PRE-PHASE pair-set dedupe refused THIS round with
|
||||
``ALREADY_IN_CONTEXT`` — "the document is already in my context"
|
||||
— false: only range A's slice had reached the model) → the EXACT
|
||||
re-read of range A (refused — the claim in the byte-identical
|
||||
line is now TRUE: those lines ARE in the prompt) → the answer
|
||||
echoing BOTH marker lines (range A's slice carried marker A,
|
||||
range B's slice carried marker B — the read-echo convention: the
|
||||
model can only echo what reached it; pre-phase, the echo could
|
||||
contain only marker A)."""
|
||||
doc = two_range_doc
|
||||
marker_b_line = doc.content.splitlines().index(MARKER_B) + 1
|
||||
llm = TwoRangeAgentLLM(
|
||||
IDENTITY, truncated_first=False, range_b_offset=marker_b_line - 10
|
||||
)
|
||||
holder = AgentHolder()
|
||||
pieces = asyncio.run(_consume(llm, holder, _settings()))
|
||||
|
||||
# Four rounds: range A → range B → re-read range A → echo answer.
|
||||
assert len(llm.requests) == 4
|
||||
assert [call.name for call in llm.calls] == ["read", "read", "read"]
|
||||
assert llm.calls[0].arguments == {
|
||||
"path": IDENTITY,
|
||||
"offset": RANGE_A_OFFSET,
|
||||
"limit": RANGE_A_LIMIT,
|
||||
}
|
||||
assert llm.calls[1].arguments == {
|
||||
"path": IDENTITY,
|
||||
"offset": marker_b_line - 10,
|
||||
"limit": RANGE_B_LIMIT,
|
||||
}
|
||||
assert llm.calls[2].arguments == {
|
||||
"path": IDENTITY,
|
||||
"offset": RANGE_A_OFFSET,
|
||||
"limit": RANGE_A_LIMIT,
|
||||
}
|
||||
|
||||
# Round 1: range A's slice — the true extent on the third line +
|
||||
# marker A inside.
|
||||
range_a_result = llm.requests[1][0][3]
|
||||
assert range_a_result["role"] == "tool"
|
||||
assert isinstance(range_a_result["content"], str)
|
||||
total_lines = len(doc.content.splitlines())
|
||||
assert range_a_result["content"].startswith(
|
||||
f"Document {IDENTITY}:\ndate: 2024-06-15\nlines: 1-40 of {total_lines}\n"
|
||||
)
|
||||
assert MARKER_A in range_a_result["content"]
|
||||
|
||||
# Round 2: range B's slice — NOT a refusal (the owner's bug: the
|
||||
# pre-phase code refused it with ALREADY_IN_CONTEXT), the true
|
||||
# extent + marker B inside.
|
||||
range_b_result = llm.requests[2][0][5]
|
||||
assert range_b_result["role"] == "tool"
|
||||
assert isinstance(range_b_result["content"], str)
|
||||
assert range_b_result["content"] != agent.ALREADY_IN_CONTEXT
|
||||
assert (
|
||||
f"lines: {marker_b_line - 10}-{marker_b_line - 10 + RANGE_B_LIMIT - 1} "
|
||||
f"of {total_lines}\n" in range_b_result["content"]
|
||||
)
|
||||
assert MARKER_B in range_b_result["content"]
|
||||
|
||||
# Round 3: the EXACT re-covered range A — refused (the claim in
|
||||
# the byte-identical line is now true).
|
||||
reread_result = llm.requests[3][0][7]
|
||||
assert reread_result["role"] == "tool"
|
||||
assert reread_result["content"] == agent.ALREADY_IN_CONTEXT
|
||||
|
||||
# Round 4: the answer echoes BOTH markers (the bug: pre-phase,
|
||||
# the echo could contain only marker A — range B's slice never
|
||||
# reached the model).
|
||||
answer = _answer(pieces)
|
||||
assert answer == f"{MARKER_A}\n{MARKER_B}"
|
||||
|
||||
# Bookkeeping: the doc committed to the context exactly once; the
|
||||
# two executed range reads counted (the refusal counted nothing);
|
||||
# no truncation (the doc is under the cap); the coverage spans
|
||||
# are the two ranges, in read order.
|
||||
assert [d.id for d in holder.read_docs] == [doc.id]
|
||||
assert holder.tool_calls == 2
|
||||
assert holder.read_truncations == []
|
||||
assert holder.read_coverage == [
|
||||
(SOURCE, DOC_PATH, RANGE_A_OFFSET, RANGE_A_LIMIT),
|
||||
(SOURCE, DOC_PATH, marker_b_line - 10, marker_b_line - 10 + RANGE_B_LIMIT - 1),
|
||||
]
|
||||
|
||||
|
||||
def test_truncated_plain_read_then_range_b_proceeds_and_plain_reread_teaches(
|
||||
two_range_doc: Document,
|
||||
) -> None:
|
||||
"""The truncated variant (the doc OVER the cap — the phase-131
|
||||
truncated-then-range row, now via coverage): the plain read
|
||||
truncates at the cap (it covers only its cut span — the count
|
||||
from the ACTUAL cut string, the same truth rule the code uses)
|
||||
and carries the pinned notice; the range-B read PROCEEDS (the
|
||||
truncated first read covered only its first lines — pre-phase the
|
||||
row ran through the ``truncated_shown`` map, now through
|
||||
``read_coverage``); the plain re-read gets the generalized
|
||||
``REREAD_PARTIAL`` teaching (BOTH covered spans named in line
|
||||
form — the cut span and the already-read range B; the retired
|
||||
phase-131 char-based ``REREAD_TRUNCATED`` line is gone); the
|
||||
answer echoes both markers (the truncated first read carried
|
||||
marker A, the range read carried marker B)."""
|
||||
doc = two_range_doc
|
||||
marker_b_line = doc.content.splitlines().index(MARKER_B) + 1
|
||||
cap = _truncated_first_cap(doc.content) # derived — see the helper
|
||||
covered = len(doc.content[:cap].splitlines()) # the cut's truth
|
||||
llm = TwoRangeAgentLLM(
|
||||
IDENTITY, truncated_first=True, range_b_offset=marker_b_line - 10
|
||||
)
|
||||
holder = AgentHolder()
|
||||
pieces = asyncio.run(_consume(llm, holder, _settings(read_max_chars=cap)))
|
||||
|
||||
# The cut's shape the assertions below lean on: marker A sits
|
||||
# INSIDE the cut (the echo of both markers needs it in the
|
||||
# truncated first read), marker B does not (the range-B read is
|
||||
# what carries it).
|
||||
assert covered >= MARKER_A_LINE and covered < MARKER_B_LINE
|
||||
|
||||
# Four rounds: plain read → range B → plain re-read → echo.
|
||||
assert len(llm.requests) == 4
|
||||
assert [call.name for call in llm.calls] == ["read", "read", "read"]
|
||||
# Round 1: the PLAIN read (no range arguments at all).
|
||||
assert llm.calls[0].arguments == {"path": IDENTITY}
|
||||
assert llm.calls[1].arguments == {
|
||||
"path": IDENTITY,
|
||||
"offset": marker_b_line - 10,
|
||||
"limit": RANGE_B_LIMIT,
|
||||
}
|
||||
assert llm.calls[2].arguments == {"path": IDENTITY}
|
||||
|
||||
# Round 1's result: cut at the cap, the shared marker + the
|
||||
# pinned notice, marker A inside the cut; the truncation surfaced
|
||||
# to the API layer as a ToolResultPiece (the SSE
|
||||
# ``tool_result`` frame's wire value).
|
||||
plain_result = llm.requests[1][0][3]
|
||||
assert plain_result["role"] == "tool"
|
||||
assert isinstance(plain_result["content"], str)
|
||||
notice = READ_TRUNCATION_NOTICE.format(shown=cap, total=len(doc.content))
|
||||
assert plain_result["content"].startswith(f"Document {IDENTITY}:\n")
|
||||
assert plain_result["content"].endswith(f"\n{TRUNCATION_MARKER}\n{notice}")
|
||||
assert MARKER_A in plain_result["content"]
|
||||
truncations = [p for p in pieces if isinstance(p, ToolResultPiece)]
|
||||
assert [(p.argument, p.chars_shown, p.chars_total) for p in truncations] == [
|
||||
(IDENTITY, cap, len(doc.content))
|
||||
]
|
||||
|
||||
# Round 2: the range-B read of the ALREADY-TRUNCATED doc
|
||||
# PROCEEDS — the slice, not a refusal.
|
||||
range_b_result = llm.requests[2][0][5]
|
||||
assert range_b_result["role"] == "tool"
|
||||
assert isinstance(range_b_result["content"], str)
|
||||
assert range_b_result["content"] != agent.ALREADY_IN_CONTEXT
|
||||
total_lines = len(doc.content.splitlines())
|
||||
assert (
|
||||
f"lines: {marker_b_line - 10}-{marker_b_line - 10 + RANGE_B_LIMIT - 1} "
|
||||
f"of {total_lines}\n" in range_b_result["content"]
|
||||
)
|
||||
assert MARKER_B in range_b_result["content"]
|
||||
|
||||
# Round 3: the plain re-read of the PARTIALLY-covered doc — the
|
||||
# generalized teaching naming BOTH covered spans in line form
|
||||
# (the merged cut span AND the already-read range B — the
|
||||
# multi-span rendering, in span order), the true total from the
|
||||
# resolved doc.
|
||||
reread_result = llm.requests[3][0][7]
|
||||
assert reread_result["role"] == "tool"
|
||||
assert reread_result["content"] == agent.REREAD_PARTIAL.format(
|
||||
covered=f"1–{covered}, {marker_b_line - 10}–{marker_b_line - 10 + RANGE_B_LIMIT - 1}",
|
||||
source=SOURCE,
|
||||
path=DOC_PATH,
|
||||
total=total_lines,
|
||||
)
|
||||
|
||||
# Round 4: the answer echoes both markers.
|
||||
answer = _answer(pieces)
|
||||
assert answer == f"{MARKER_A}\n{MARKER_B}"
|
||||
|
||||
# Bookkeeping: the doc once; the two successful reads counted (the
|
||||
# teaching refusal counted nothing); the truncation record is the
|
||||
# plain read's tuple; the coverage spans — the cut span, then the
|
||||
# range — in read order.
|
||||
assert [d.id for d in holder.read_docs] == [doc.id]
|
||||
assert holder.tool_calls == 2
|
||||
assert holder.read_truncations == [(IDENTITY, cap, len(doc.content))]
|
||||
assert holder.read_coverage == [
|
||||
(SOURCE, DOC_PATH, 1, covered),
|
||||
(SOURCE, DOC_PATH, marker_b_line - 10, marker_b_line - 10 + RANGE_B_LIMIT - 1),
|
||||
]
|
||||
+886
-57
File diff suppressed because it is too large
Load Diff
@@ -72,12 +72,17 @@ def test_high_and_low_bases_byte_locked() -> None:
|
||||
#: phase 118 (task 04, A6, owner directive 2026-09-15): the ``read``
|
||||
#: clause was rewritten for the summary-seed mode (the ``<documents>``
|
||||
#: section holds SUMMARIES — ``read`` adds the full text; only an
|
||||
#: already-read document is refused). Only that clause moved — the
|
||||
#: prefix (the ``ls``-clause opening) and the suffix (the
|
||||
#: discipline-rules ending) survived byte-identical, so they are the
|
||||
#: same anchors as pre-phase-118.
|
||||
TOOLS_SECTION_SHA256 = "87ee80faf0170da6ab1de518177313def6460785613fa074312a2a5f9f071750"
|
||||
TOOLS_SECTION_LEN = 2465
|
||||
#: already-read document is refused). RE-CUT again for phase 141
|
||||
#: (task 02): the re-read sentence was re-cut to the coverage-true
|
||||
#: form — do not re-read a document read IN FULL (answer from it);
|
||||
#: a PARTIAL document (a range read, or a truncated read) is not
|
||||
#: complete in the context until read all — read the other parts with
|
||||
#: ``offset``/``limit``. Only that sentence moved — the prefix (the
|
||||
#: ``ls``-clause opening) and the suffix (the discipline-rules
|
||||
#: ending) survived byte-identical, so they are the same anchors as
|
||||
#: pre-phase-141.
|
||||
TOOLS_SECTION_SHA256 = "3671680bbf4e04efe1186690c18d24869e4974ceb29485cedc3e34a9286bf3b2"
|
||||
TOOLS_SECTION_LEN = 2656
|
||||
TOOLS_SECTION_PREFIX = (
|
||||
"<tools>\n"
|
||||
"You may extend your context with three tools. `ls` lists the "
|
||||
|
||||
@@ -339,23 +339,39 @@ def test_tools_section_phase118_summary_seed_read_clause() -> None:
|
||||
"""Phase 118 (task 04, A6): the ``read`` clause is rewritten for
|
||||
the summary-seed mode — the ``<documents>`` section holds
|
||||
SUMMARIES (a suggested document's full text is not in the prompt
|
||||
until ``read`` adds it); do not re-read an already-read document
|
||||
(answer from the text already in the prompt); if the user asks to
|
||||
until ``read`` adds it); do not re-read a document already read
|
||||
IN FULL (answer from the text already in the prompt); a PARTIAL
|
||||
document (a range read, or a truncated read — re-pinned phase
|
||||
141, task 02: the re-read sentence is the coverage-true form —
|
||||
read the other parts with ``offset``/``limit``, the document is
|
||||
not complete in the context until read all); if the user asks to
|
||||
open or read a suggested document, ``read`` it. The phase-72 "do
|
||||
not call ``read`` for a ``<documents>`` document" copy is retired.
|
||||
The ``ls`` clause (phase 94 drill-down contract), the ``grep``
|
||||
clause, and the discipline rules stay byte-identical — the
|
||||
prompt lock's prefix/suffix anchors survive (see
|
||||
``test_prompt_lock``)."""
|
||||
# The new summary-seed copy (pinned byte-for-byte).
|
||||
# The summary-seed copy (pinned byte-for-byte — the phase-141
|
||||
# coverage-true re-read sentence re-pins the pre-phase-141 line;
|
||||
# the section's other bytes are unchanged).
|
||||
assert (
|
||||
"The <documents> section holds SUMMARIES — the full text of a "
|
||||
"suggested document is not in your prompt until you `read` it. "
|
||||
"Do not re-read a document you have already read — its full "
|
||||
"text is already in your prompt; answer directly from it. If "
|
||||
"the user asks you to open or read a suggested document, "
|
||||
"`read` it — that is the point of the section."
|
||||
"Do not re-read a document you have already read in full — "
|
||||
"its full text is in your prompt, answer from it. If you have "
|
||||
"read only part of a document (a range read, or a truncated "
|
||||
"read), read the other parts with `offset` and `limit` — the "
|
||||
"document is not complete in your context until you have read "
|
||||
"it all. If the user asks you to open or read a suggested "
|
||||
"document, `read` it — that is the point of the section."
|
||||
) in TOOLS_SECTION
|
||||
# The retired pre-phase-141 re-read copy is gone (it claimed the
|
||||
# FULL text for a document that may be only partially in the
|
||||
# context — the owner's stuck-turn bug).
|
||||
assert (
|
||||
"Do not re-read a document you have already read — its full "
|
||||
"text is already in your prompt; answer directly from it"
|
||||
) not in TOOLS_SECTION
|
||||
# The read identity handoff now also names the <documents> summary
|
||||
# blocks (the combined identity is shown there), keeping the
|
||||
# "including the source name" contract.
|
||||
|
||||
Reference in New Issue
Block a user