phase: 141_read_range_dedupe_fix
All gates green and verified. Final report: **Phase 141 final verification pass — complete** - **Verified (tasks 01–04 already in `complete/`):** `AgentHolder.read_coverage` span recording (all 4 read outcomes, refusals record nothing); coverage dedupe matrix (range read of uncovered lines proceeds, fully-covered re-read → `ALREADY_IN_CONTEXT` with a now-true claim, partial plain re-read → `REREAD_PARTIAL` naming merged spans, `REREAD_TRUNCATED` retired, fast path retired per task 02); `TOOLS_SECTION` re-read sentence re-cut to coverage-true form (prompt lock re-pinned); owner-scenario integration repro (pure + truncated variants) and mock `read two ranges` flow + dedicated E2E — all match the task specs exactly; no code defects found. - **Tests/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → exit 0, TOTAL 99% (5498 stmts, `app/rag/agent.py` 99%) — run twice, both green; `uv run pytest tests/e2e/test_read_range_dedupe.py -v --no-cov` → 2 passed in isolation; `uv run ruff check . && uv run pyright` → All checks passed / 0 errors. - **Completion criteria:** (1) two different ranges both reach the model, re-covered re-read refused truthfully — ✅ (integration + E2E); (2) partial plain re-read gets the `offset`/`limit` teaching, false claim gone both directions — ✅; (3) plain-read + phase-131 result shapes byte-identical, existing pins pass unchanged — ✅; (4) all gate commands green, >90% — ✅; (5) no behavior change in completed phases (grep/ls, truncation notice, `read_docs`/`read_truncations` signals) — ✅ (whole project suite green). - **Documentation:** README.md — fixed pre-existing phase-140 gap: `BOR_PDF_PICTORIAL` / `BOR_PDF_PICTORIAL_MIN_CHARS` were in `.env.example`/Settings but undocumented (README claims every `.env.example` var is covered); added the pictorial-PDF-pass bullet to the PDF section + two config-reference rows. AGENTS.md — verified consistent (rules, quick-ref commands, phase refs all check out) — no changes needed. - **Notable:** overview's "fast path survives for holder-state rows" note is superseded by task 02's authoritative decision (every dedupe row resolves — "full" needs the true line count); implementation follows task 02. No phase files touched; changes left in the working tree for the harness commit. - **Next pending phase:** `142_kb_overview_freshness`.
This commit is contained in:
+276
-5
@@ -644,17 +644,76 @@ Implements just enough of the aipi surface:
|
||||
N`` extent) skipped, first 80 chars (the phase-94 read-echo
|
||||
convention — BOTH ranges reached the model's context). NOTE:
|
||||
for the second read to PROCEED (not be refused
|
||||
``ALREADY_IN_CONTEXT``) the document must already be in the
|
||||
agent's truncated set — the story suite boots the app under
|
||||
test with a ``BOR_READ_MAX_CHARS`` small enough that the
|
||||
FIRST (20-line) range slice truncates, which is exactly the
|
||||
large-document premise of the owner's confusion.
|
||||
``ALREADY_IN_CONTEXT``) the document was required to already
|
||||
be in the agent's truncated set — the story suite boots the
|
||||
app under test with a ``BOR_READ_MAX_CHARS`` small enough
|
||||
that the FIRST (20-line) range slice truncates, which is
|
||||
exactly the large-document premise of the owner's confusion.
|
||||
Phase 141 RETIRED that premise: the coverage-tracked dedupe
|
||||
proceeds with a second range read whenever its lines are not
|
||||
fully covered — the suite's small cap stays as the
|
||||
large-document premise, but is no longer LOAD-BEARING for the
|
||||
second read to proceed.
|
||||
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
|
||||
phrases — the phase-71/72/94/118/131 ordering convention; the
|
||||
trigger needs the ``<tools>`` section, so deflected turns never
|
||||
hit it); verified 2026-09-30: no existing E2E question or
|
||||
fixture file contains the phrase, so every other suite is
|
||||
unaffected.
|
||||
- user message containing ``read two ranges``
|
||||
(``TWO_RANGES_TRIGGER``, phase 141 task 04 — the read
|
||||
range-dedupe fix's dedicated story suite
|
||||
``tests/e2e/test_read_range_dedupe.py``) **and** the system
|
||||
prompt carries the ``<tools>`` section -> the owner's stuck
|
||||
scenario (TODO.md L3), end to end: the model range-reads ONE
|
||||
document in TWO different 100-line ranges — pre-phase the
|
||||
second read was refused ``ALREADY_IN_CONTEXT`` ("the document
|
||||
is already in my context, but I haven't actually seen the full
|
||||
text") while ONLY the first slice was in the prompt; the
|
||||
coverage-tracked server proceeds with it, and the answer
|
||||
echoes the SECOND range's content (the marker line lives in
|
||||
range B, so an echo of it proves the slice reached the model).
|
||||
The question carries its own target after the colon (``read
|
||||
two ranges: read source/path`` — the question-carried-call
|
||||
convention of ``READ_CAP_TRIGGER`` / ``SAME_FILE_TRIGGER``),
|
||||
and the step is then discriminated statelessly by the tool
|
||||
results in the messages (the house count pattern — streaming
|
||||
only):
|
||||
* request 1 (``tools`` offered, no tool results yet): a
|
||||
``tool_calls`` delta — ``read`` on the parsed target with
|
||||
``offset`` = ``TWO_RANGES_FIRST_OFFSET`` (1) and ``limit``
|
||||
= ``TWO_RANGES_FIRST_LIMIT`` (100) (id ``call_0``);
|
||||
* EXACTLY ONE ``tool``-role read result is in the messages
|
||||
(content starting with the agent's ``"Document "`` prefix —
|
||||
``_READ_RESULT_PREFIX``) AND it carries the ``lines: 1-100
|
||||
of …`` extent header (the mock's own keying, asserted):
|
||||
the SECOND range read — a ``tool_calls`` delta — ``read``
|
||||
on the SAME target with ``offset`` =
|
||||
``TWO_RANGES_SECOND_OFFSET`` (101) and ``limit`` =
|
||||
``TWO_RANGES_SECOND_LIMIT`` (100) (id ``call_1``) — the
|
||||
flow only works if the second read PROCEEDS (the pre-phase
|
||||
server would have refused it);
|
||||
* TWO or more read results are in the messages and the
|
||||
SECOND carries the ``lines: 101-…`` header (asserted the
|
||||
same way — the mock's keying stays honest): the forced
|
||||
answer, byte-stable, in the existing read-echo shape —
|
||||
``Both parts read. <first 80 chars of the second read
|
||||
result's content, header lines skipped>`` (the
|
||||
:func:`_same_file_quote` convention — the RANGE reached
|
||||
the model's context);
|
||||
* a tool result is in the messages that is not a well-keyed
|
||||
read result for the step (a refusal — the second range
|
||||
read was refused, the pre-phase bug — or an unkeyed extent
|
||||
header): the deterministic broken-premise echo — the
|
||||
offending result VERBATIM under a ``Premise broke:`` line
|
||||
— so a suite fails loudly on it (the read-cap flow's
|
||||
broken-premise convention).
|
||||
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint
|
||||
trigger phrases — the phase-71/72/94/118/131/139 ordering
|
||||
convention; the trigger needs the ``<tools>`` section, so
|
||||
deflected turns never hit it); verified 2026-10-01: no existing
|
||||
E2E question or fixture file contains the phrase, so every
|
||||
other suite is unaffected.
|
||||
|
||||
Failure injection (phase 67, LLM retry, TODO.md L3) — deterministic
|
||||
dead-endpoint behavior for the retry E2E suite (``tests/e2e/
|
||||
@@ -2465,6 +2524,149 @@ def _same_file_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
return ("read1", target)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 141 (task 04): the owner's stuck scenario — the model
|
||||
# range-reads ONE document in TWO different ranges; the
|
||||
# coverage-tracked server proceeds with the second (the pre-phase
|
||||
# pair-set dedupe refused it ``ALREADY_IN_CONTEXT`` while only the
|
||||
# first slice was in the prompt). See the module docstring.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: A user message containing this substring (case-insensitive) —
|
||||
#: combined with the ``<tools>`` section in the system prompt — drives
|
||||
#: the deterministic TWO-RANGES flow (the read range-dedupe fix's
|
||||
#: story suite, ``tests/e2e/test_read_range_dedupe.py`` — the
|
||||
#: owner's stuck scenario, TODO.md L3). Checked BEFORE the plain
|
||||
#: ``TOOLS_TRIGGER`` flow (disjoint trigger phrases — the
|
||||
#: phase-71/72/94/118/131/139 ordering convention); verified
|
||||
#: 2026-10-01: no existing E2E question or fixture file contains
|
||||
#: the phrase, so every other suite is unaffected.
|
||||
TWO_RANGES_TRIGGER = "read two ranges"
|
||||
|
||||
#: The scripted target in the two-ranges question (case-
|
||||
#: insensitive — the suite's question capitalizes the trigger's
|
||||
#: first letter): the verb (``read``) plus the target — a combined
|
||||
#: ``source/path``, parsed from the RAW user message so the target
|
||||
#: keeps its case (the ``_SAME_FILE_CALL_RE`` convention; the target
|
||||
#: is a ``[a-z0-9_./-]`` run, so a `` — `` flavor separator can
|
||||
#: never bleed into it).
|
||||
_TWO_RANGES_CALL_RE = re.compile(
|
||||
r"read two ranges:\s*read\s+(?P<arg>[a-z0-9_./-]+)",
|
||||
re.I,
|
||||
)
|
||||
|
||||
#: The FIRST range read's requested extent — range A (lines 1–100
|
||||
#: of the suite's 200-line fixture document).
|
||||
TWO_RANGES_FIRST_OFFSET = 1
|
||||
TWO_RANGES_FIRST_LIMIT = 100
|
||||
|
||||
#: The SECOND range read's requested extent — range B (lines 101–
|
||||
#: …): the SAME file again, a different range. Pre-phase the server
|
||||
#: refused this call ``ALREADY_IN_CONTEXT`` (the false "full text is
|
||||
#: in your prompt" claim — the owner's stuck turn); the coverage-
|
||||
#: tracked server proceeds with it (its lines are NOT covered by
|
||||
#: the first read's span).
|
||||
TWO_RANGES_SECOND_OFFSET = 101
|
||||
TWO_RANGES_SECOND_LIMIT = 100
|
||||
|
||||
|
||||
def _two_ranges_extent(content: str) -> str | None:
|
||||
"""The ``lines: a-b of N`` extent line of one read result's
|
||||
content (the ``Document …:`` header already stripped by
|
||||
:func:`_read_results`, the phase-106 D5 ``date:`` line skipped)
|
||||
— ``None`` when the result carries no extent line (a plain
|
||||
read). The flow keys its step on this line (the mock's own
|
||||
keying, asserted — a refusal or a differently-shaped result is
|
||||
the broken premise, never a step input).
|
||||
"""
|
||||
body = content
|
||||
if body.startswith("date: "):
|
||||
body = body.partition("\n")[2]
|
||||
if body.startswith("lines: "):
|
||||
return body.partition("\n")[0]
|
||||
return None
|
||||
|
||||
|
||||
def _two_ranges_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
|
||||
"""Classify a phase-141 two-ranges request (see the module
|
||||
docstring). The question carries its own target (``read two
|
||||
ranges: read source/path``); the step is then discriminated
|
||||
statelessly by the tool results in the messages (the house count
|
||||
pattern — the flow only ever issues ``read`` calls, so counting
|
||||
read results is reliable):
|
||||
|
||||
* ``("answer", quote)`` — TWO or more ``tool``-role read results
|
||||
are in the messages and the SECOND carries the ``lines: 101-
|
||||
…`` extent header (asserted — the second range read landed as
|
||||
advertised): the forced answer, byte-stable, in the existing
|
||||
read-echo shape — ``Both parts read. <quote>`` with quote the
|
||||
second result's pure content (header lines skipped, first 80
|
||||
chars — the :func:`_same_file_quote` convention; the marker
|
||||
line lives in range B, so an echo of it proves the slice
|
||||
reached the model).
|
||||
* ``("read2", target)`` — EXACTLY ONE read result is in the
|
||||
messages and it carries the ``lines: 1-100 of …`` extent
|
||||
header (asserted — the first range read landed as advertised):
|
||||
the SECOND range read on the SAME target — ``offset`` =
|
||||
:data:`TWO_RANGES_SECOND_OFFSET`, ``limit`` =
|
||||
:data:`TWO_RANGES_SECOND_LIMIT` (id ``call_1``).
|
||||
* ``("read1", target)`` — ``tools`` are offered and no tool
|
||||
result is in the messages yet: the FIRST range read on the
|
||||
parsed target — ``offset`` = :data:`TWO_RANGES_FIRST_OFFSET`,
|
||||
``limit`` = :data:`TWO_RANGES_FIRST_LIMIT` (id ``call_0``).
|
||||
* ``("bad", result)`` — a tool result is in the messages that is
|
||||
not a well-keyed read result for the step (a refusal — the
|
||||
second range read was refused, the pre-phase bug — or an
|
||||
unkeyed extent header): the deterministic broken-premise echo
|
||||
— the offending result verbatim under ``Premise broke:`` (the
|
||||
read-cap flow's broken-premise convention — a suite fails
|
||||
loudly on it).
|
||||
* ``None`` — not the flow: the trigger is absent, the
|
||||
``<tools>`` section is missing (deflected turns never carry
|
||||
it), the target is unparseable, or ``tools`` are not offered
|
||||
and no tool result is in the messages yet (e.g.
|
||||
``agent_max_rounds=0``).
|
||||
"""
|
||||
user = _user(body)
|
||||
if TWO_RANGES_TRIGGER not in user.lower():
|
||||
return None
|
||||
if "<tools>" not in _system(body):
|
||||
return None
|
||||
match = _TWO_RANGES_CALL_RE.search(user)
|
||||
if match is None:
|
||||
return None
|
||||
target = match.group("arg")
|
||||
reads = _read_results(body)
|
||||
results = _tool_results(body)
|
||||
if len(reads) >= 2:
|
||||
extent = _two_ranges_extent(reads[1][1])
|
||||
if extent is None or not extent.startswith(
|
||||
f"lines: {TWO_RANGES_SECOND_OFFSET}-"
|
||||
):
|
||||
return ("bad", reads[1][1])
|
||||
return ("answer", _same_file_quote(reads[1][1]))
|
||||
if len(reads) == 1:
|
||||
if len(results) > 1:
|
||||
# A NON-read tool result after the read result — a
|
||||
# refusal (the pre-phase bug: the second range read was
|
||||
# refused ``ALREADY_IN_CONTEXT``): the premise broke —
|
||||
# echo it so a suite fails loudly.
|
||||
return ("bad", results[-1])
|
||||
extent = _two_ranges_extent(reads[0][1])
|
||||
if extent is None or not extent.startswith(
|
||||
f"lines: {TWO_RANGES_FIRST_OFFSET}-{TWO_RANGES_FIRST_LIMIT} of "
|
||||
):
|
||||
return ("bad", reads[0][1])
|
||||
return ("read2", target)
|
||||
if results:
|
||||
# A tool result that is NOT a read result before any read
|
||||
# landed — the first read was refused: the premise broke.
|
||||
return ("bad", results[-1])
|
||||
if not body.get("tools"):
|
||||
return None
|
||||
return ("read1", target)
|
||||
|
||||
|
||||
def long_answer() -> str:
|
||||
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
|
||||
a unique final line that must survive the stream untruncated."""
|
||||
@@ -3773,6 +3975,75 @@ def chat_completions(body: dict[str, Any]) -> Any:
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 141 (task 04): the owner's stuck scenario (the read
|
||||
# range-dedupe fix's story suite,
|
||||
# tests/e2e/test_read_range_dedupe.py — TODO.md L3): the
|
||||
# model range-reads ONE document in TWO different ranges —
|
||||
# the second PROCEEDS on the coverage-tracked server (the
|
||||
# pre-phase pair-set dedupe refused it ``ALREADY_IN_
|
||||
# CONTEXT``), and the answer echoes the second range's
|
||||
# content. Checked BEFORE the plain TOOLS_TRIGGER flow
|
||||
# (disjoint trigger phrases — the phase-71/72/94/118/131/
|
||||
# 139 ordering convention; the trigger needs the ``<tools>``
|
||||
# section, so deflected turns never hit it).
|
||||
two_ranges = _two_ranges_flow(body)
|
||||
if two_ranges is not None:
|
||||
if two_ranges[0] == "read1":
|
||||
# The FIRST range read on the parsed target — range
|
||||
# A (lines 1–100 of the suite's document).
|
||||
stream = _tool_call_stream(
|
||||
"read",
|
||||
{
|
||||
"path": two_ranges[1],
|
||||
"offset": TWO_RANGES_FIRST_OFFSET,
|
||||
"limit": TWO_RANGES_FIRST_LIMIT,
|
||||
},
|
||||
"call_0",
|
||||
)
|
||||
elif two_ranges[0] == "read2":
|
||||
# The SECOND range read — the SAME file again, a
|
||||
# different range (range B, lines 101–…). It only
|
||||
# PROCEEDS on the coverage-tracked server: its lines
|
||||
# are NOT covered by the first read's span (the pre-
|
||||
# phase pair-set dedupe refused it — the owner's
|
||||
# stuck turn).
|
||||
stream = _tool_call_stream(
|
||||
"read",
|
||||
{
|
||||
"path": two_ranges[1],
|
||||
"offset": TWO_RANGES_SECOND_OFFSET,
|
||||
"limit": TWO_RANGES_SECOND_LIMIT,
|
||||
},
|
||||
"call_1",
|
||||
)
|
||||
elif two_ranges[0] == "bad":
|
||||
# The premise broke (a refusal — the second range
|
||||
# read was refused — or an unkeyed extent header):
|
||||
# echo the offending result verbatim so a suite
|
||||
# fails loudly on it (the read-cap flow's broken-
|
||||
# premise convention).
|
||||
stream = _sse_stream(
|
||||
_apply_max_tokens(
|
||||
f"Premise broke:\n{two_ranges[1]}",
|
||||
body.get("max_tokens"),
|
||||
),
|
||||
0.0,
|
||||
)
|
||||
else: # "answer" — the forced read-echo of range B
|
||||
# (the marker line lives in range B — the echo of it
|
||||
# proves the second slice reached the model).
|
||||
stream = _sse_stream(
|
||||
_apply_max_tokens(
|
||||
f"Both parts read. {two_ranges[1]}",
|
||||
body.get("max_tokens"),
|
||||
),
|
||||
0.0,
|
||||
)
|
||||
return StreamingResponse(
|
||||
stream,
|
||||
media_type="text/event-stream",
|
||||
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
|
||||
)
|
||||
# Phase 124 (task 07, image chips and answers, TODO L3): the
|
||||
# scripted SHOW_IMAGE flow — the mock shows the seeded image
|
||||
# document (the ``<documents>`` block with the image-
|
||||
|
||||
@@ -0,0 +1,665 @@
|
||||
"""Phase 141 task 04 E2E (Playwright, mock-only): the read
|
||||
range-dedupe fix's dedicated story suite — the owner's stuck
|
||||
scenario, end to end through the SSE stream.
|
||||
|
||||
TODO.md L3: "From a chat I was having, the LLM is trying to read
|
||||
different ranges of a document and keeps saying: 'The system is
|
||||
telling me the document is already in my context, but I haven't
|
||||
actually seen the full text … The read calls keep returning
|
||||
"Already in your context"…' This seems like a critical bug, fix
|
||||
it." Pre-phase, ``AgentHolder``'s pair-set dedupe recorded ANY
|
||||
successful read — including a RANGE read — as if the FULL document
|
||||
were in context, so a second read of a DIFFERENT range of the same
|
||||
document was refused with a claim that was false. Phase 141 tracks
|
||||
the covered line SPANS (``AgentHolder.read_coverage``) and refuses
|
||||
only genuinely-covered re-reads (tasks 01–03, unit/integration
|
||||
pinned) — this suite is the browser-visible proof: the
|
||||
deterministic mock LLM drives a two-range turn, and the assertions
|
||||
show both ``read`` tool frames succeeding and the final answer
|
||||
carrying content from the SECOND range.
|
||||
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_read_range_dedupe.py -v --no-cov
|
||||
|
||||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — this story's
|
||||
gate is the deterministic SCRIPTED flow in
|
||||
``tests/e2e/mock_llm.py`` (the mock is the only E2E lens on the
|
||||
LLM's context):
|
||||
|
||||
* the two-range turn — ``TWO_RANGES_TRIGGER``: a user message
|
||||
containing ``read two ranges`` **and** the system prompt carrying
|
||||
the HIGH prompt's ``<tools>`` section. Request 1 reads
|
||||
``offset=1, limit=100`` (range A) on the question-carried target
|
||||
(``read two ranges: read source/path``); request 2 (one
|
||||
``tool``-role read result in the messages, carrying the
|
||||
``lines: 1-100 of …`` extent header — the mock's own keying,
|
||||
asserted) issues the SECOND read ``offset=101, limit=100`` (range
|
||||
B) on the SAME target — pre-phase the server refused it
|
||||
``ALREADY_IN_CONTEXT`` while only range A was in the prompt; the
|
||||
flow only works if the second read PROCEEDS. Request 3 (two read
|
||||
results in the messages, the second carrying the ``lines: 101-…``
|
||||
header — asserted the same way) answers ``Both parts read.
|
||||
<first 80 chars of the second read result's content, header lines
|
||||
skipped>`` (the existing read-echo shape) — the marker line lives
|
||||
in range B, so an echo of it proves the slice reached the model.
|
||||
A broken premise (a refusal in the messages — the pre-phase bug —
|
||||
or an unkeyed extent header) lands as the deterministic
|
||||
``Premise broke:`` echo, so a suite fails loudly on it.
|
||||
* the single-range turn — the mock's EXISTING ``READ_LINES_TRIGGER``
|
||||
flow (``read the lines``, phase 131 — the grep →
|
||||
read(offset/limit) dance) reused on THIS suite's KB: the
|
||||
single-range behavior is unchanged by the coverage rework (the
|
||||
phase-131 expectation still passes against the coverage-tracked
|
||||
server).
|
||||
|
||||
The app under test keeps the DEFAULT read cap (128 000 chars) —
|
||||
deliberately: NO read truncates in this suite. That is the point —
|
||||
pre-phase, a second range read of a NON-truncated document already
|
||||
in the pair set was refused ``ALREADY_IN_CONTEXT`` (the phase-139
|
||||
suite had to shrink ``BOR_READ_MAX_CHARS`` until the first range
|
||||
slice truncated just to let a second range read through — that
|
||||
workaround is retired by the coverage tracking). The still-refused
|
||||
row (a re-read of FULLY-covered lines) is unit-pinned in tasks
|
||||
02/03 (``tests/unit/test_agent.py`` — the dedupe matrix); this
|
||||
suite's job is the HAPPY two-range path, browser-visible.
|
||||
|
||||
KB fixture — ONE deterministic 200-line text document
|
||||
(``tworanges/two.txt``): fixed-length filler lines (``Line <i>: …``
|
||||
— the line numbers are exact) carrying THREE marker lines —
|
||||
``TWO_RANGES_A_LINE …`` at line 50 (range A), ``TWO_RANGES_B_LINE …``
|
||||
at line 101 (the FIRST line of range B — the read-echo quote is the
|
||||
slice's first 80 chars, so the marker must sit at the range's start
|
||||
to be echoed), and ``LINE_MARKER …`` at line 30 (the phase-131
|
||||
flow's grep target — exactly one line carries the pattern). Each
|
||||
marker line has two contextual neighbor lines repeating its
|
||||
distinctive words (the mock's token-overlap embeddings are
|
||||
diluted by the ~19-line chunks — the neighbors keep the marker
|
||||
chunk's cosine above the mock-calibrated lexical floor, the
|
||||
house gate pattern). All line positions are computed from the very
|
||||
string written to the fixture (never a literal in the
|
||||
assertions), so the suite stays true if the fixture is regenerated.
|
||||
|
||||
The app under test is mock-calibrated (conftest pattern):
|
||||
``BOR_RELEVANCE_THRESHOLD=0.30`` + ``BOR_LEXICAL_SUPPORT_FLOOR=0.15``
|
||||
+ ``BOR_SOURCE_USEFULNESS_FLOOR=0.15`` — both questions' best-chunk
|
||||
cosine (the marker chunk wins — ~0.23 / ~0.19 on the mock's
|
||||
token-overlap distribution, deterministic) clears the floor with
|
||||
FTS hits (``alpha``/``beta``/``beacon`` and ``trap``/``components``
|
||||
in the marker chunks), so every scripted turn stays GROUNDED (the
|
||||
``<tools>`` section is present) and the read document clears the
|
||||
usefulness bar into ``done.sources``.
|
||||
|
||||
Test → observable mapping (Playwright Mapping Rule; the story is
|
||||
the owner's TODO item — one Playwright file per story, A16):
|
||||
1. ``test_two_ranges_both_reach_the_model`` — the ``read two
|
||||
ranges`` question → the SSE ``tool`` frames are EXACTLY the two
|
||||
range reads of the SAME path (``offset`` 1/``limit`` 100, then
|
||||
``offset`` 101/``limit`` 100 — asserted on the frames' ARGS; a
|
||||
refused second read would re-issue the call and add a THIRD
|
||||
frame), NO ``tool_result`` frames (nothing truncated — the
|
||||
default cap), the ``done`` frame is non-deflected with the read
|
||||
document cited (phase 119 A1), the final answer carries
|
||||
``Both parts read.`` + the ``TWO_RANGES_B_LINE`` marker text
|
||||
(the second slice reached the model's context), and NO refusal
|
||||
surface anywhere (neither the ``ALREADY_IN_CONTEXT`` line nor
|
||||
the mock's broken-premise echo is in the answer).
|
||||
2. ``test_single_range_flow_unchanged`` — the phase-131 ``read the
|
||||
lines`` question on this KB → the SSE ``tool`` frames are the
|
||||
grep (the canned pattern) then the range read (``offset`` = the
|
||||
marker's true line, ``limit`` = 30) — the phase-131 expectation
|
||||
unchanged against the coverage-tracked server; the final answer
|
||||
is the grounded echo ``Found it at line <N>`` + the marker text.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from playwright.sync_api import Locator, Page, expect
|
||||
from sqlalchemy import select, text
|
||||
|
||||
from app.config import Settings as _Settings
|
||||
from app.db import SessionLocal
|
||||
from app.models import Document
|
||||
from app.rag.agent import ALREADY_IN_CONTEXT
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
from e2e.mock_llm import READ_LINES_LIMIT, READ_LINES_PATTERN
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
# Phase 79 (task 04, full inventory): the conftest session app owns
|
||||
# its port in a combined run — this module app binds its own port
|
||||
# instead (a same-port second uvicorn dies on bind and would drive
|
||||
# the wrong server). Env-overridable.
|
||||
APP_PORT = int(os.environ.get("E2E_APP_PORT_TWORANGES", "8156"))
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
SOURCE = "tworanges" # the local directory's basename = the source name
|
||||
REL = "two.txt"
|
||||
SP = f"{SOURCE}/{REL}" # the combined read identity
|
||||
|
||||
# The fixture's layout (deterministic — see the module docstring):
|
||||
# 200 fixed-length filler lines (60 chars each) carrying three
|
||||
# marker lines, each flanked by two contextual neighbor lines that
|
||||
# repeat its distinctive words (the mock-calibrated gate — the
|
||||
# marker chunk's cosine must clear the 0.15 lexical floor).
|
||||
TOTAL_LINES = 200
|
||||
A_MARKER_LINE = 50 # range A (lines 1–100)
|
||||
B_MARKER_LINE = 101 # range B (lines 101+): the FIRST line of range B
|
||||
LINE_MARKER_LINE = 30 # the phase-131 flow's grep target (range A)
|
||||
|
||||
#: The range A marker line (lines 1–100 carry it — NOT echoed by the
|
||||
#: mock; its presence in the fixture is the two-range premise).
|
||||
A_MARKER_TEXT = (
|
||||
"TWO_RANGES_A_LINE the first range documents the alpha beacon payload"
|
||||
)
|
||||
A_CTX_ABOVE = (
|
||||
"the alpha beacon table starts here: alpha beacon payloads, documented below"
|
||||
)
|
||||
A_CTX_BELOW = (
|
||||
"the alpha beacon entry ends here — alpha beacon payload documented above"
|
||||
)
|
||||
|
||||
#: The range B marker line — the FIRST line of range B, so the
|
||||
#: mock's read-echo quote (the second read result's first 80 chars,
|
||||
#: header lines skipped) carries it: an echo of it proves the second
|
||||
#: slice reached the model.
|
||||
B_MARKER_TEXT = (
|
||||
"TWO_RANGES_B_LINE the second range documents the beta beacon payload"
|
||||
)
|
||||
B_CTX_ABOVE = (
|
||||
"the beta beacon table starts here: beta beacon payloads, documented below"
|
||||
)
|
||||
B_CTX_BELOW = (
|
||||
"the beta beacon entry ends here — beta beacon payload documented above"
|
||||
)
|
||||
|
||||
#: The phase-131 flow's marker line: exactly ONE line in the document
|
||||
#: carries ``READ_LINES_PATTERN`` (imported — the mock can never
|
||||
#: drift from the fixture). Its true line is computed from the very
|
||||
#: string written to the fixture (the determinism pin).
|
||||
LINE_MARKER_TEXT = "LINE_MARKER the trap components are documented here"
|
||||
LM_CTX_ABOVE = (
|
||||
"the trap components table starts here: trap components, documented below"
|
||||
)
|
||||
LM_CTX_BELOW = (
|
||||
"the trap components entry ends here — trap components documented above"
|
||||
)
|
||||
|
||||
_FILLER_WORDS = "quiet mirror vault raid pool chain slot cycle guard probe audit "
|
||||
|
||||
|
||||
def _filler_line(i: int) -> str:
|
||||
"""One fixed-length (60-char) filler line — the line numbers are
|
||||
exact and every line is the SAME length, so the range slices'
|
||||
lengths are deterministic."""
|
||||
prefix = f"Line {i:04d}: "
|
||||
return prefix + (_FILLER_WORDS * 5)[: 60 - len(prefix)]
|
||||
|
||||
|
||||
def _doc() -> str:
|
||||
"""The 200-line fixture document (deterministic — module
|
||||
docstring)."""
|
||||
lines: list[str] = []
|
||||
for i in range(1, TOTAL_LINES + 1):
|
||||
if i == LINE_MARKER_LINE - 1:
|
||||
lines.append(LM_CTX_ABOVE)
|
||||
elif i == LINE_MARKER_LINE:
|
||||
lines.append(LINE_MARKER_TEXT)
|
||||
elif i == LINE_MARKER_LINE + 1:
|
||||
lines.append(LM_CTX_BELOW)
|
||||
elif i == A_MARKER_LINE - 1:
|
||||
lines.append(A_CTX_ABOVE)
|
||||
elif i == A_MARKER_LINE:
|
||||
lines.append(A_MARKER_TEXT)
|
||||
elif i == A_MARKER_LINE + 1:
|
||||
lines.append(A_CTX_BELOW)
|
||||
elif i == B_MARKER_LINE - 1:
|
||||
lines.append(B_CTX_ABOVE)
|
||||
elif i == B_MARKER_LINE:
|
||||
lines.append(B_MARKER_TEXT)
|
||||
elif i == B_MARKER_LINE + 1:
|
||||
lines.append(B_CTX_BELOW)
|
||||
else:
|
||||
lines.append(_filler_line(i))
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
DOC = _doc()
|
||||
DOC_LINES = DOC.splitlines()
|
||||
DOC_TOTAL = len(DOC)
|
||||
assert len(DOC_LINES) == TOTAL_LINES, len(DOC_LINES)
|
||||
|
||||
# The determinism pins (computed from the very string written to the
|
||||
# fixture — the assertions use these, never literals): the markers
|
||||
# are unique, the B marker is the FIRST line of range B (the
|
||||
# read-echo quote reaches it), and the range slices are UNDER the
|
||||
# default read cap (nothing truncates — no ``tool_result`` frames).
|
||||
assert DOC.count(LINE_MARKER_TEXT) == 1
|
||||
assert DOC.count(A_MARKER_TEXT) == 1
|
||||
assert DOC.count(B_MARKER_TEXT) == 1
|
||||
assert DOC_LINES[B_MARKER_LINE - 1] == B_MARKER_TEXT
|
||||
assert DOC_LINES[A_MARKER_LINE - 1] == A_MARKER_TEXT
|
||||
assert DOC_LINES[LINE_MARKER_LINE - 1] == LINE_MARKER_TEXT
|
||||
#: The default read cap (128 000 — ``app/config.py``) stays in force
|
||||
#: for this suite; both 100-line slices must stay under it.
|
||||
assert len("\n".join(DOC_LINES[:100])) < 128_000
|
||||
assert len("\n".join(DOC_LINES[100:200])) < 128_000
|
||||
|
||||
# The expected answer fragments (the mock's read-echo shape —
|
||||
# ``Both parts read. <quote>`` with quote the second read result's
|
||||
# content, the ``date:`` / ``lines:`` header lines skipped, first 80
|
||||
# chars). The B marker line (70 chars) IS the slice's first line, so
|
||||
# the full marker text sits inside the 80-char quote.
|
||||
ANSWER_LEAD = "Both parts read."
|
||||
assert len(B_MARKER_TEXT) < 80, len(B_MARKER_TEXT)
|
||||
|
||||
# The scripted questions. Q_A carries ``TWO_RANGES_TRIGGER`` + the
|
||||
# target (the question-carried-call convention — the mock parses
|
||||
# ``read two ranges: read source/path`` from the raw user message);
|
||||
# Q_B carries the phase-131 ``READ_LINES_TRIGGER`` (the grep →
|
||||
# read(offset/limit) dance) on this KB. Both carry their marker
|
||||
# chunks' distinctive words (the mock-calibrated gate stays
|
||||
# GROUNDED — the ``<tools>`` section is present, so both flows
|
||||
# fire).
|
||||
Q_A = (
|
||||
f"Read two ranges: read {SP} — "
|
||||
"what documents the alpha and beta beacon payloads?"
|
||||
)
|
||||
Q_B = f"Read the lines of {SP} — what are the trap components documented here?"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def tworanges_dirs(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The story's local-directory source: one host temp dir (the app
|
||||
server runs on the same host, so the path is visible to it)
|
||||
holding the ONE generated 200-line text document. The directory's
|
||||
basename is the source name (``kind=local``, phase 38)."""
|
||||
root = tmp_path_factory.mktemp("bor_two_ranges") / SOURCE
|
||||
root.mkdir()
|
||||
(root / REL).write_text(DOC, encoding="utf-8")
|
||||
assert (root / REL).read_text(encoding="utf-8") == DOC
|
||||
assert not (root / ".git").exists() # the local-kind story: NOT git
|
||||
return root
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, tworanges_dirs: Path) -> Iterator[str]:
|
||||
"""The real app under test — per-module env (the conftest
|
||||
pattern, cf. ``test_read_line_ranges.py``): NO
|
||||
``BOR_GIT_SOURCES`` (the env fallback is git-only — the source
|
||||
here is a DB-registered local directory), the mock LLM, the
|
||||
mock-calibrated gate (threshold 0.30 + lexical floor 0.15 — the
|
||||
conftest pair). The DEFAULT read cap stays in force — THE
|
||||
SUBJECT: no read truncates in this suite, and the second range
|
||||
read still proceeds (pre-phase, the pair-set dedupe refused a
|
||||
second range read of a NON-truncated document already in
|
||||
context — the owner's stuck turn). The session app is never
|
||||
started in this isolated run, so no port clash."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1"
|
||||
if USE_REAL_LLM
|
||||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated gate (conftest pattern): both scripted turns
|
||||
# stay GROUNDED (the questions' best-chunk cosine clears the
|
||||
# 0.15 floor with FTS hits — the marker chunks win).
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||
# The read document must clear the citation bar into
|
||||
# done.sources (phase 119 A1 — both turns assert it).
|
||||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||
# Phase 67: instant retry waits + the code-default budget (the
|
||||
# conftest leak-guard pattern).
|
||||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's settings (the app reads
|
||||
# it from cwd) — override the source list with an EMPTY value (the
|
||||
# env var beats the .env file): the registry must hold EXACTLY the
|
||||
# local directory this suite registers (a leftover env git list
|
||||
# would pollute the KB the scripted turns run against).
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||||
# .env cannot leak corpus-specific settings into the app under
|
||||
# test.
|
||||
env["BOR_DOCS_REPO"] = ""
|
||||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||
_Settings.model_fields["suggestions"].default
|
||||
)
|
||||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB (the E2E isolation pattern): the E2E
|
||||
suites share one Postgres, so a leftover git_sources row or
|
||||
document would pollute the retrieval the scripted turns run
|
||||
against (the single-document KB's gate margins are pinned
|
||||
against EXACTLY this one document)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(
|
||||
text(
|
||||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||
"kb_overview, git_sources, folder_summaries"
|
||||
)
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
|
||||
"""Poll the (cookie-authenticated) status endpoint until the run
|
||||
reaches a terminal state (the test_ls_tree_drilldown pattern, over
|
||||
plain httpx — this fixture has no browser page yet)."""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = client.get("/api/sync/status")
|
||||
assert r.status_code == 200, r.text
|
||||
body = r.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def synced_kb(app_server: str, tworanges_dirs: Path) -> None:
|
||||
"""The story's precondition: the one-source KB synced under the
|
||||
deterministic mock. Registers the temp directory through the
|
||||
authenticated API (the ``test_local_directory_sources.py``
|
||||
pattern), runs the REAL in-process sync (``POST /api/sync`` —
|
||||
walk → chunk → embed → overview → version bump), pins the stored
|
||||
content byte-identical to the fixture string."""
|
||||
_truncate_all()
|
||||
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, r.text
|
||||
r = client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": str(tworanges_dirs)}
|
||||
)
|
||||
assert r.status_code == 201, r.text
|
||||
r = client.post("/api/sync")
|
||||
assert r.status_code == 202, r.text
|
||||
body = _wait_sync_done_http(client)
|
||||
assert body["state"] == "success", body
|
||||
detail = body["detail"]
|
||||
assert detail["added"] == 1, detail
|
||||
assert detail["pruned"] == 0, detail
|
||||
# The import stored the fixture string BYTE-IDENTICALLY — the
|
||||
# slice extents every assertion uses come from the very string
|
||||
# written to the directory.
|
||||
with SessionLocal() as db:
|
||||
stored = db.scalar(
|
||||
select(Document.content).where(
|
||||
Document.source == SOURCE, Document.path == REL
|
||||
)
|
||||
)
|
||||
assert stored == DOC, "stored content drifted from the fixture"
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
"""Per-test query_log isolation (the KB itself is module-scoped —
|
||||
the scripted turns never change it, so the registry persists
|
||||
across the tests of this module)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
yield
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE query_log"))
|
||||
db.commit()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# Page helpers (the test_read_line_ranges / test_read_range_tool_lines
|
||||
# house patterns)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||||
#: (a response clone read in the background) — wire-level assertions
|
||||
#: for the ``tool`` / ``tool_result`` frames, independent of the UI
|
||||
#: rendering.
|
||||
SSE_HOOK = """
|
||||
() => {
|
||||
if (window.__sseInstalled) return;
|
||||
window.__sseInstalled = true;
|
||||
window.__sseFrames = [];
|
||||
const origFetch = window.fetch;
|
||||
window.fetch = async function (...args) {
|
||||
const res = await origFetch.apply(this, args);
|
||||
try {
|
||||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||||
if (url.includes('/api/chat')) {
|
||||
res.clone().text().then((bodyText) => {
|
||||
for (const block of bodyText.split('\\n\\n')) {
|
||||
const line = block.trim();
|
||||
if (line.startsWith('data: ')) {
|
||||
window.__sseFrames.push(line.slice(6));
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
} catch (e) { /* non-clonable responses: ignored */ }
|
||||
return res;
|
||||
};
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
def _install_page_hooks(page: Page) -> None:
|
||||
page.evaluate(SSE_HOOK)
|
||||
|
||||
|
||||
def _frames(page: Page, deadline_s: float = 120.0) -> list[dict]:
|
||||
"""The SSE frames captured since the last submit (``_submit``
|
||||
clears the buffer), once the hook's background read settles."""
|
||||
deadline = time.monotonic() + deadline_s
|
||||
while True:
|
||||
raw = page.evaluate("() => window.__sseFrames || []")
|
||||
parsed = [json.loads(line) for line in raw if line]
|
||||
if any(f.get("type") == "done" for f in parsed):
|
||||
return parsed
|
||||
if time.monotonic() > deadline:
|
||||
raise AssertionError(
|
||||
f"SSE hook captured no `done` frame (frames so far: "
|
||||
f"{len(parsed)}) — hook install failed?"
|
||||
)
|
||||
time.sleep(0.05)
|
||||
|
||||
|
||||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool"]
|
||||
|
||||
|
||||
def _result_frames(frames: list[dict]) -> list[dict]:
|
||||
return [f for f in frames if f.get("type") == "tool_result"]
|
||||
|
||||
|
||||
def _submit(page: Page, question: str) -> None:
|
||||
page.evaluate("window.__sseFrames = []")
|
||||
page.fill("#message-input", question)
|
||||
page.click("#send-btn")
|
||||
# The user bubble lands synchronously with the submit handler.
|
||||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||||
|
||||
|
||||
def _wait_settled(page: Page, timeout_ms: int = 120_000) -> None:
|
||||
"""The turn is complete: answer text in the bubble, button
|
||||
recovered (the phase-48 settle wait)."""
|
||||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text(
|
||||
"", timeout=timeout_ms
|
||||
)
|
||||
expect(page.locator("#send-btn")).to_be_enabled(timeout=timeout_ms)
|
||||
expect(page.locator("#send-label")).to_have_text("Send", timeout=timeout_ms)
|
||||
|
||||
|
||||
def _last_brain(page: Page) -> Locator:
|
||||
return page.locator(".msg.brain").last
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 1. The owner's stuck scenario, browser-visible: two DIFFERENT ranges
|
||||
# of the same document both reach the model (no refusal)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_two_ranges_both_reach_the_model(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(60_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, Q_A)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
# THE SUBJECT on the wire — IN ORDER, asserted on the frames'
|
||||
# ARGS (not a substring): the SAME path, TWO distinct ranges
|
||||
# (offset 1/limit 100, then offset 101/limit 100 — the phase-131
|
||||
# additive fields ride the SAME frame). EXACTLY two: a refused
|
||||
# second read (the pre-phase bug) would re-issue the call and
|
||||
# add a THIRD frame — or land as the mock's broken-premise echo.
|
||||
assert _tool_frames(frames) == [
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "read",
|
||||
"argument": SP,
|
||||
"offset": 1,
|
||||
"limit": 100,
|
||||
},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "read",
|
||||
"argument": SP,
|
||||
"offset": 101,
|
||||
"limit": 100,
|
||||
},
|
||||
], _tool_frames(frames)
|
||||
# NO ``tool_result`` frame: nothing truncated (the default cap —
|
||||
# both 100-line slices are far under it; one frame = one
|
||||
# noteworthy event, the phase-95 additive contract).
|
||||
assert _result_frames(frames) == [], _result_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
# The read document is the turn's cited source (phase 119 A1).
|
||||
assert any(
|
||||
s["path"] == REL and s["source"] == SOURCE for s in done["sources"]
|
||||
), done["sources"]
|
||||
|
||||
# The GROUNDED echo — the SECOND range reached the model's
|
||||
# context and landed in the answer: ``Both parts read.`` + the
|
||||
# ``TWO_RANGES_B_LINE`` marker text (the marker is range B's
|
||||
# first line — inside the 80-char read-echo quote).
|
||||
text = _last_brain(page).locator(".bubble").text_content() or ""
|
||||
assert ANSWER_LEAD in text, text
|
||||
assert B_MARKER_TEXT in text, text
|
||||
# NO refusal surface anywhere in the rendered answer: the
|
||||
# ``ALREADY_IN_CONTEXT`` line (the pre-phase false claim — its
|
||||
# full text is the refusal the owner quoted) and the mock's
|
||||
# broken-premise echo are both absent.
|
||||
assert ALREADY_IN_CONTEXT not in text, text
|
||||
assert "Premise broke" not in text, text
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# 2. The single-range behavior is unchanged: the phase-131 grep →
|
||||
# read(offset/limit) dance still lands on the coverage-tracked
|
||||
# server (the phase-131 expectation, this KB)
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_single_range_flow_unchanged(
|
||||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||||
) -> None:
|
||||
page.set_default_timeout(60_000)
|
||||
login(page, app_url, next="/")
|
||||
_install_page_hooks(page)
|
||||
|
||||
_submit(page, Q_B)
|
||||
_wait_settled(page)
|
||||
|
||||
frames = _frames(page)
|
||||
# The phase-131 dance on the wire — IN ORDER, asserted on the
|
||||
# frames' ARGS: the grep (the mock's canned pattern), then the
|
||||
# range read — path = the doc's combined identity, offset = the
|
||||
# marker's TRUE line (the mock parsed it from the grep result's
|
||||
# first match line), limit = READ_LINES_LIMIT (30).
|
||||
assert _tool_frames(frames) == [
|
||||
{"type": "tool", "name": "grep", "argument": READ_LINES_PATTERN},
|
||||
{
|
||||
"type": "tool",
|
||||
"name": "read",
|
||||
"argument": SP,
|
||||
"offset": LINE_MARKER_LINE,
|
||||
"limit": READ_LINES_LIMIT,
|
||||
},
|
||||
], _tool_frames(frames)
|
||||
# The 30-line range slice did NOT truncate (the default cap) —
|
||||
# no ``tool_result`` frame (the phase-131 pinned order contract).
|
||||
assert _result_frames(frames) == [], _result_frames(frames)
|
||||
done = next(f for f in frames if f.get("type") == "done")
|
||||
assert done["deflected"] is False, done
|
||||
assert any(
|
||||
s["path"] == REL and s["source"] == SOURCE for s in done["sources"]
|
||||
), done["sources"]
|
||||
|
||||
# The GROUNDED echo — the phase-131 answer shape unchanged:
|
||||
# ``Found it at line <N>`` (N == the marker's true line) + the
|
||||
# marker text itself (the single-range read reached the model's
|
||||
# context exactly as before the coverage rework).
|
||||
text = _last_brain(page).locator(".bubble").text_content() or ""
|
||||
assert f"Found it at line {LINE_MARKER_LINE}." in text, text
|
||||
assert LINE_MARKER_TEXT in text, text
|
||||
Reference in New Issue
Block a user