All gates green and verified. Final report: **Phase 141 final verification pass — complete** - **Verified (tasks 01–04 already in `complete/`):** `AgentHolder.read_coverage` span recording (all 4 read outcomes, refusals record nothing); coverage dedupe matrix (range read of uncovered lines proceeds, fully-covered re-read → `ALREADY_IN_CONTEXT` with a now-true claim, partial plain re-read → `REREAD_PARTIAL` naming merged spans, `REREAD_TRUNCATED` retired, fast path retired per task 02); `TOOLS_SECTION` re-read sentence re-cut to coverage-true form (prompt lock re-pinned); owner-scenario integration repro (pure + truncated variants) and mock `read two ranges` flow + dedicated E2E — all match the task specs exactly; no code defects found. - **Tests/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → exit 0, TOTAL 99% (5498 stmts, `app/rag/agent.py` 99%) — run twice, both green; `uv run pytest tests/e2e/test_read_range_dedupe.py -v --no-cov` → 2 passed in isolation; `uv run ruff check . && uv run pyright` → All checks passed / 0 errors. - **Completion criteria:** (1) two different ranges both reach the model, re-covered re-read refused truthfully — ✅ (integration + E2E); (2) partial plain re-read gets the `offset`/`limit` teaching, false claim gone both directions — ✅; (3) plain-read + phase-131 result shapes byte-identical, existing pins pass unchanged — ✅; (4) all gate commands green, >90% — ✅; (5) no behavior change in completed phases (grep/ls, truncation notice, `read_docs`/`read_truncations` signals) — ✅ (whole project suite green). - **Documentation:** README.md — fixed pre-existing phase-140 gap: `BOR_PDF_PICTORIAL` / `BOR_PDF_PICTORIAL_MIN_CHARS` were in `.env.example`/Settings but undocumented (README claims every `.env.example` var is covered); added the pictorial-PDF-pass bullet to the PDF section + two config-reference rows. AGENTS.md — verified consistent (rules, quick-ref commands, phase refs all check out) — no changes needed. - **Notable:** overview's "fast path survives for holder-state rows" note is superseded by task 02's authoritative decision (every dedupe row resolves — "full" needs the true line count); implementation follows task 02. No phase files touched; changes left in the working tree for the harness commit. - **Next pending phase:** `142_kb_overview_freshness`.
666 lines
28 KiB
Python
666 lines
28 KiB
Python
"""Phase 141 task 04 E2E (Playwright, mock-only): the read
|
||
range-dedupe fix's dedicated story suite — the owner's stuck
|
||
scenario, end to end through the SSE stream.
|
||
|
||
TODO.md L3: "From a chat I was having, the LLM is trying to read
|
||
different ranges of a document and keeps saying: 'The system is
|
||
telling me the document is already in my context, but I haven't
|
||
actually seen the full text … The read calls keep returning
|
||
"Already in your context"…' This seems like a critical bug, fix
|
||
it." Pre-phase, ``AgentHolder``'s pair-set dedupe recorded ANY
|
||
successful read — including a RANGE read — as if the FULL document
|
||
were in context, so a second read of a DIFFERENT range of the same
|
||
document was refused with a claim that was false. Phase 141 tracks
|
||
the covered line SPANS (``AgentHolder.read_coverage``) and refuses
|
||
only genuinely-covered re-reads (tasks 01–03, unit/integration
|
||
pinned) — this suite is the browser-visible proof: the
|
||
deterministic mock LLM drives a two-range turn, and the assertions
|
||
show both ``read`` tool frames succeeding and the final answer
|
||
carrying content from the SECOND range.
|
||
|
||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||
|
||
uv run pytest tests/e2e/test_read_range_dedupe.py -v --no-cov
|
||
|
||
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — this story's
|
||
gate is the deterministic SCRIPTED flow in
|
||
``tests/e2e/mock_llm.py`` (the mock is the only E2E lens on the
|
||
LLM's context):
|
||
|
||
* the two-range turn — ``TWO_RANGES_TRIGGER``: a user message
|
||
containing ``read two ranges`` **and** the system prompt carrying
|
||
the HIGH prompt's ``<tools>`` section. Request 1 reads
|
||
``offset=1, limit=100`` (range A) on the question-carried target
|
||
(``read two ranges: read source/path``); request 2 (one
|
||
``tool``-role read result in the messages, carrying the
|
||
``lines: 1-100 of …`` extent header — the mock's own keying,
|
||
asserted) issues the SECOND read ``offset=101, limit=100`` (range
|
||
B) on the SAME target — pre-phase the server refused it
|
||
``ALREADY_IN_CONTEXT`` while only range A was in the prompt; the
|
||
flow only works if the second read PROCEEDS. Request 3 (two read
|
||
results in the messages, the second carrying the ``lines: 101-…``
|
||
header — asserted the same way) answers ``Both parts read.
|
||
<first 80 chars of the second read result's content, header lines
|
||
skipped>`` (the existing read-echo shape) — the marker line lives
|
||
in range B, so an echo of it proves the slice reached the model.
|
||
A broken premise (a refusal in the messages — the pre-phase bug —
|
||
or an unkeyed extent header) lands as the deterministic
|
||
``Premise broke:`` echo, so a suite fails loudly on it.
|
||
* the single-range turn — the mock's EXISTING ``READ_LINES_TRIGGER``
|
||
flow (``read the lines``, phase 131 — the grep →
|
||
read(offset/limit) dance) reused on THIS suite's KB: the
|
||
single-range behavior is unchanged by the coverage rework (the
|
||
phase-131 expectation still passes against the coverage-tracked
|
||
server).
|
||
|
||
The app under test keeps the DEFAULT read cap (128 000 chars) —
|
||
deliberately: NO read truncates in this suite. That is the point —
|
||
pre-phase, a second range read of a NON-truncated document already
|
||
in the pair set was refused ``ALREADY_IN_CONTEXT`` (the phase-139
|
||
suite had to shrink ``BOR_READ_MAX_CHARS`` until the first range
|
||
slice truncated just to let a second range read through — that
|
||
workaround is retired by the coverage tracking). The still-refused
|
||
row (a re-read of FULLY-covered lines) is unit-pinned in tasks
|
||
02/03 (``tests/unit/test_agent.py`` — the dedupe matrix); this
|
||
suite's job is the HAPPY two-range path, browser-visible.
|
||
|
||
KB fixture — ONE deterministic 200-line text document
|
||
(``tworanges/two.txt``): fixed-length filler lines (``Line <i>: …``
|
||
— the line numbers are exact) carrying THREE marker lines —
|
||
``TWO_RANGES_A_LINE …`` at line 50 (range A), ``TWO_RANGES_B_LINE …``
|
||
at line 101 (the FIRST line of range B — the read-echo quote is the
|
||
slice's first 80 chars, so the marker must sit at the range's start
|
||
to be echoed), and ``LINE_MARKER …`` at line 30 (the phase-131
|
||
flow's grep target — exactly one line carries the pattern). Each
|
||
marker line has two contextual neighbor lines repeating its
|
||
distinctive words (the mock's token-overlap embeddings are
|
||
diluted by the ~19-line chunks — the neighbors keep the marker
|
||
chunk's cosine above the mock-calibrated lexical floor, the
|
||
house gate pattern). All line positions are computed from the very
|
||
string written to the fixture (never a literal in the
|
||
assertions), so the suite stays true if the fixture is regenerated.
|
||
|
||
The app under test is mock-calibrated (conftest pattern):
|
||
``BOR_RELEVANCE_THRESHOLD=0.30`` + ``BOR_LEXICAL_SUPPORT_FLOOR=0.15``
|
||
+ ``BOR_SOURCE_USEFULNESS_FLOOR=0.15`` — both questions' best-chunk
|
||
cosine (the marker chunk wins — ~0.23 / ~0.19 on the mock's
|
||
token-overlap distribution, deterministic) clears the floor with
|
||
FTS hits (``alpha``/``beta``/``beacon`` and ``trap``/``components``
|
||
in the marker chunks), so every scripted turn stays GROUNDED (the
|
||
``<tools>`` section is present) and the read document clears the
|
||
usefulness bar into ``done.sources``.
|
||
|
||
Test → observable mapping (Playwright Mapping Rule; the story is
|
||
the owner's TODO item — one Playwright file per story, A16):
|
||
1. ``test_two_ranges_both_reach_the_model`` — the ``read two
|
||
ranges`` question → the SSE ``tool`` frames are EXACTLY the two
|
||
range reads of the SAME path (``offset`` 1/``limit`` 100, then
|
||
``offset`` 101/``limit`` 100 — asserted on the frames' ARGS; a
|
||
refused second read would re-issue the call and add a THIRD
|
||
frame), NO ``tool_result`` frames (nothing truncated — the
|
||
default cap), the ``done`` frame is non-deflected with the read
|
||
document cited (phase 119 A1), the final answer carries
|
||
``Both parts read.`` + the ``TWO_RANGES_B_LINE`` marker text
|
||
(the second slice reached the model's context), and NO refusal
|
||
surface anywhere (neither the ``ALREADY_IN_CONTEXT`` line nor
|
||
the mock's broken-premise echo is in the answer).
|
||
2. ``test_single_range_flow_unchanged`` — the phase-131 ``read the
|
||
lines`` question on this KB → the SSE ``tool`` frames are the
|
||
grep (the canned pattern) then the range read (``offset`` = the
|
||
marker's true line, ``limit`` = 30) — the phase-131 expectation
|
||
unchanged against the coverage-tracked server; the final answer
|
||
is the grounded echo ``Found it at line <N>`` + the marker text.
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import json
|
||
import os
|
||
import subprocess
|
||
import sys
|
||
import time
|
||
from collections.abc import Iterator
|
||
from pathlib import Path
|
||
from typing import Any
|
||
|
||
import httpx
|
||
import pytest
|
||
from playwright.sync_api import Locator, Page, expect
|
||
from sqlalchemy import select, text
|
||
|
||
from app.config import Settings as _Settings
|
||
from app.db import SessionLocal
|
||
from app.models import Document
|
||
from app.rag.agent import ALREADY_IN_CONTEXT
|
||
from e2e.auth_helpers import login
|
||
from e2e.conftest import (
|
||
ADMIN_PASSWORD,
|
||
SESSION_SECRET,
|
||
USE_REAL_LLM,
|
||
_wait_http,
|
||
)
|
||
from e2e.mock_llm import READ_LINES_LIMIT, READ_LINES_PATTERN
|
||
|
||
REPO = Path(__file__).resolve().parents[2]
|
||
|
||
# Phase 79 (task 04, full inventory): the conftest session app owns
|
||
# its port in a combined run — this module app binds its own port
|
||
# instead (a same-port second uvicorn dies on bind and would drive
|
||
# the wrong server). Env-overridable.
|
||
APP_PORT = int(os.environ.get("E2E_APP_PORT_TWORANGES", "8156"))
|
||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||
|
||
SOURCE = "tworanges" # the local directory's basename = the source name
|
||
REL = "two.txt"
|
||
SP = f"{SOURCE}/{REL}" # the combined read identity
|
||
|
||
# The fixture's layout (deterministic — see the module docstring):
|
||
# 200 fixed-length filler lines (60 chars each) carrying three
|
||
# marker lines, each flanked by two contextual neighbor lines that
|
||
# repeat its distinctive words (the mock-calibrated gate — the
|
||
# marker chunk's cosine must clear the 0.15 lexical floor).
|
||
TOTAL_LINES = 200
|
||
A_MARKER_LINE = 50 # range A (lines 1–100)
|
||
B_MARKER_LINE = 101 # range B (lines 101+): the FIRST line of range B
|
||
LINE_MARKER_LINE = 30 # the phase-131 flow's grep target (range A)
|
||
|
||
#: The range A marker line (lines 1–100 carry it — NOT echoed by the
|
||
#: mock; its presence in the fixture is the two-range premise).
|
||
A_MARKER_TEXT = (
|
||
"TWO_RANGES_A_LINE the first range documents the alpha beacon payload"
|
||
)
|
||
A_CTX_ABOVE = (
|
||
"the alpha beacon table starts here: alpha beacon payloads, documented below"
|
||
)
|
||
A_CTX_BELOW = (
|
||
"the alpha beacon entry ends here — alpha beacon payload documented above"
|
||
)
|
||
|
||
#: The range B marker line — the FIRST line of range B, so the
|
||
#: mock's read-echo quote (the second read result's first 80 chars,
|
||
#: header lines skipped) carries it: an echo of it proves the second
|
||
#: slice reached the model.
|
||
B_MARKER_TEXT = (
|
||
"TWO_RANGES_B_LINE the second range documents the beta beacon payload"
|
||
)
|
||
B_CTX_ABOVE = (
|
||
"the beta beacon table starts here: beta beacon payloads, documented below"
|
||
)
|
||
B_CTX_BELOW = (
|
||
"the beta beacon entry ends here — beta beacon payload documented above"
|
||
)
|
||
|
||
#: The phase-131 flow's marker line: exactly ONE line in the document
|
||
#: carries ``READ_LINES_PATTERN`` (imported — the mock can never
|
||
#: drift from the fixture). Its true line is computed from the very
|
||
#: string written to the fixture (the determinism pin).
|
||
LINE_MARKER_TEXT = "LINE_MARKER the trap components are documented here"
|
||
LM_CTX_ABOVE = (
|
||
"the trap components table starts here: trap components, documented below"
|
||
)
|
||
LM_CTX_BELOW = (
|
||
"the trap components entry ends here — trap components documented above"
|
||
)
|
||
|
||
_FILLER_WORDS = "quiet mirror vault raid pool chain slot cycle guard probe audit "
|
||
|
||
|
||
def _filler_line(i: int) -> str:
|
||
"""One fixed-length (60-char) filler line — the line numbers are
|
||
exact and every line is the SAME length, so the range slices'
|
||
lengths are deterministic."""
|
||
prefix = f"Line {i:04d}: "
|
||
return prefix + (_FILLER_WORDS * 5)[: 60 - len(prefix)]
|
||
|
||
|
||
def _doc() -> str:
|
||
"""The 200-line fixture document (deterministic — module
|
||
docstring)."""
|
||
lines: list[str] = []
|
||
for i in range(1, TOTAL_LINES + 1):
|
||
if i == LINE_MARKER_LINE - 1:
|
||
lines.append(LM_CTX_ABOVE)
|
||
elif i == LINE_MARKER_LINE:
|
||
lines.append(LINE_MARKER_TEXT)
|
||
elif i == LINE_MARKER_LINE + 1:
|
||
lines.append(LM_CTX_BELOW)
|
||
elif i == A_MARKER_LINE - 1:
|
||
lines.append(A_CTX_ABOVE)
|
||
elif i == A_MARKER_LINE:
|
||
lines.append(A_MARKER_TEXT)
|
||
elif i == A_MARKER_LINE + 1:
|
||
lines.append(A_CTX_BELOW)
|
||
elif i == B_MARKER_LINE - 1:
|
||
lines.append(B_CTX_ABOVE)
|
||
elif i == B_MARKER_LINE:
|
||
lines.append(B_MARKER_TEXT)
|
||
elif i == B_MARKER_LINE + 1:
|
||
lines.append(B_CTX_BELOW)
|
||
else:
|
||
lines.append(_filler_line(i))
|
||
return "\n".join(lines)
|
||
|
||
|
||
DOC = _doc()
|
||
DOC_LINES = DOC.splitlines()
|
||
DOC_TOTAL = len(DOC)
|
||
assert len(DOC_LINES) == TOTAL_LINES, len(DOC_LINES)
|
||
|
||
# The determinism pins (computed from the very string written to the
|
||
# fixture — the assertions use these, never literals): the markers
|
||
# are unique, the B marker is the FIRST line of range B (the
|
||
# read-echo quote reaches it), and the range slices are UNDER the
|
||
# default read cap (nothing truncates — no ``tool_result`` frames).
|
||
assert DOC.count(LINE_MARKER_TEXT) == 1
|
||
assert DOC.count(A_MARKER_TEXT) == 1
|
||
assert DOC.count(B_MARKER_TEXT) == 1
|
||
assert DOC_LINES[B_MARKER_LINE - 1] == B_MARKER_TEXT
|
||
assert DOC_LINES[A_MARKER_LINE - 1] == A_MARKER_TEXT
|
||
assert DOC_LINES[LINE_MARKER_LINE - 1] == LINE_MARKER_TEXT
|
||
#: The default read cap (128 000 — ``app/config.py``) stays in force
|
||
#: for this suite; both 100-line slices must stay under it.
|
||
assert len("\n".join(DOC_LINES[:100])) < 128_000
|
||
assert len("\n".join(DOC_LINES[100:200])) < 128_000
|
||
|
||
# The expected answer fragments (the mock's read-echo shape —
|
||
# ``Both parts read. <quote>`` with quote the second read result's
|
||
# content, the ``date:`` / ``lines:`` header lines skipped, first 80
|
||
# chars). The B marker line (70 chars) IS the slice's first line, so
|
||
# the full marker text sits inside the 80-char quote.
|
||
ANSWER_LEAD = "Both parts read."
|
||
assert len(B_MARKER_TEXT) < 80, len(B_MARKER_TEXT)
|
||
|
||
# The scripted questions. Q_A carries ``TWO_RANGES_TRIGGER`` + the
|
||
# target (the question-carried-call convention — the mock parses
|
||
# ``read two ranges: read source/path`` from the raw user message);
|
||
# Q_B carries the phase-131 ``READ_LINES_TRIGGER`` (the grep →
|
||
# read(offset/limit) dance) on this KB. Both carry their marker
|
||
# chunks' distinctive words (the mock-calibrated gate stays
|
||
# GROUNDED — the ``<tools>`` section is present, so both flows
|
||
# fire).
|
||
Q_A = (
|
||
f"Read two ranges: read {SP} — "
|
||
"what documents the alpha and beta beacon payloads?"
|
||
)
|
||
Q_B = f"Read the lines of {SP} — what are the trap components documented here?"
|
||
|
||
|
||
# --------------------------------------------------------------------------
|
||
# Fixtures
|
||
# --------------------------------------------------------------------------
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def tworanges_dirs(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||
"""The story's local-directory source: one host temp dir (the app
|
||
server runs on the same host, so the path is visible to it)
|
||
holding the ONE generated 200-line text document. The directory's
|
||
basename is the source name (``kind=local``, phase 38)."""
|
||
root = tmp_path_factory.mktemp("bor_two_ranges") / SOURCE
|
||
root.mkdir()
|
||
(root / REL).write_text(DOC, encoding="utf-8")
|
||
assert (root / REL).read_text(encoding="utf-8") == DOC
|
||
assert not (root / ".git").exists() # the local-kind story: NOT git
|
||
return root
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def app_server(mock_llm: int, tworanges_dirs: Path) -> Iterator[str]:
|
||
"""The real app under test — per-module env (the conftest
|
||
pattern, cf. ``test_read_line_ranges.py``): NO
|
||
``BOR_GIT_SOURCES`` (the env fallback is git-only — the source
|
||
here is a DB-registered local directory), the mock LLM, the
|
||
mock-calibrated gate (threshold 0.30 + lexical floor 0.15 — the
|
||
conftest pair). The DEFAULT read cap stays in force — THE
|
||
SUBJECT: no read truncates in this suite, and the second range
|
||
read still proceeds (pre-phase, the pair-set dedupe refused a
|
||
second range read of a NON-truncated document already in
|
||
context — the owner's stuck turn). The session app is never
|
||
started in this isolated run, so no port clash."""
|
||
env = dict(os.environ)
|
||
env.pop("DEBUGPY", None)
|
||
env["BOR_ENVIRONMENT"] = "e2e"
|
||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||
env["BOR_LLM_BASE_URL"] = (
|
||
"https://aipi.reeseapps.com/v1"
|
||
if USE_REAL_LLM
|
||
else f"http://127.0.0.1:{mock_llm}/v1"
|
||
)
|
||
# Mock-calibrated gate (conftest pattern): both scripted turns
|
||
# stay GROUNDED (the questions' best-chunk cosine clears the
|
||
# 0.15 floor with FTS hits — the marker chunks win).
|
||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||
# The read document must clear the citation bar into
|
||
# done.sources (phase 119 A1 — both turns assert it).
|
||
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||
# Phase 67: instant retry waits + the code-default budget (the
|
||
# conftest leak-guard pattern).
|
||
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
|
||
env.setdefault(
|
||
"BOR_DATABASE_URL",
|
||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||
)
|
||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||
# The repo's .env file carries the owner's settings (the app reads
|
||
# it from cwd) — override the source list with an EMPTY value (the
|
||
# env var beats the .env file): the registry must hold EXACTLY the
|
||
# local directory this suite registers (a leftover env git list
|
||
# would pollute the KB the scripted turns run against).
|
||
env["BOR_GIT_SOURCES"] = ""
|
||
# Leak guards (conftest pattern): an operator's local (gitignored)
|
||
# .env cannot leak corpus-specific settings into the app under
|
||
# test.
|
||
env["BOR_DOCS_REPO"] = ""
|
||
env["BOR_SUGGESTIONS"] = json.dumps(
|
||
_Settings.model_fields["suggestions"].default
|
||
)
|
||
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
|
||
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
|
||
proc = subprocess.Popen(
|
||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||
cwd=REPO,
|
||
env=env,
|
||
)
|
||
try:
|
||
_wait_http(f"{APP_URL}/api/health")
|
||
yield APP_URL
|
||
finally:
|
||
proc.terminate()
|
||
try:
|
||
proc.wait(timeout=10)
|
||
except subprocess.TimeoutExpired:
|
||
proc.kill()
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def app_url(app_server: str) -> str:
|
||
return app_server
|
||
|
||
|
||
def _truncate_all() -> None:
|
||
"""Fresh registry + KB (the E2E isolation pattern): the E2E
|
||
suites share one Postgres, so a leftover git_sources row or
|
||
document would pollute the retrieval the scripted turns run
|
||
against (the single-document KB's gate margins are pinned
|
||
against EXACTLY this one document)."""
|
||
with SessionLocal() as db:
|
||
db.execute(
|
||
text(
|
||
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||
"kb_overview, git_sources, folder_summaries"
|
||
)
|
||
)
|
||
db.commit()
|
||
|
||
|
||
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
|
||
"""Poll the (cookie-authenticated) status endpoint until the run
|
||
reaches a terminal state (the test_ls_tree_drilldown pattern, over
|
||
plain httpx — this fixture has no browser page yet)."""
|
||
deadline = time.monotonic() + timeout_s
|
||
body: dict[str, Any] = {}
|
||
while time.monotonic() < deadline:
|
||
r = client.get("/api/sync/status")
|
||
assert r.status_code == 200, r.text
|
||
body = r.json()
|
||
if body["state"] in ("success", "failed"):
|
||
return body
|
||
time.sleep(0.5)
|
||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||
|
||
|
||
@pytest.fixture(scope="module")
|
||
def synced_kb(app_server: str, tworanges_dirs: Path) -> None:
|
||
"""The story's precondition: the one-source KB synced under the
|
||
deterministic mock. Registers the temp directory through the
|
||
authenticated API (the ``test_local_directory_sources.py``
|
||
pattern), runs the REAL in-process sync (``POST /api/sync`` —
|
||
walk → chunk → embed → overview → version bump), pins the stored
|
||
content byte-identical to the fixture string."""
|
||
_truncate_all()
|
||
with httpx.Client(base_url=app_server, timeout=30.0) as client:
|
||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||
assert r.status_code == 204, r.text
|
||
r = client.post(
|
||
"/api/git-sources", json={"kind": "local", "path": str(tworanges_dirs)}
|
||
)
|
||
assert r.status_code == 201, r.text
|
||
r = client.post("/api/sync")
|
||
assert r.status_code == 202, r.text
|
||
body = _wait_sync_done_http(client)
|
||
assert body["state"] == "success", body
|
||
detail = body["detail"]
|
||
assert detail["added"] == 1, detail
|
||
assert detail["pruned"] == 0, detail
|
||
# The import stored the fixture string BYTE-IDENTICALLY — the
|
||
# slice extents every assertion uses come from the very string
|
||
# written to the directory.
|
||
with SessionLocal() as db:
|
||
stored = db.scalar(
|
||
select(Document.content).where(
|
||
Document.source == SOURCE, Document.path == REL
|
||
)
|
||
)
|
||
assert stored == DOC, "stored content drifted from the fixture"
|
||
|
||
|
||
@pytest.fixture(autouse=True)
|
||
def _clean(db_ready: None) -> Iterator[None]:
|
||
"""Per-test query_log isolation (the KB itself is module-scoped —
|
||
the scripted turns never change it, so the registry persists
|
||
across the tests of this module)."""
|
||
with SessionLocal() as db:
|
||
db.execute(text("TRUNCATE query_log"))
|
||
db.commit()
|
||
yield
|
||
with SessionLocal() as db:
|
||
db.execute(text("TRUNCATE query_log"))
|
||
db.commit()
|
||
|
||
|
||
# --------------------------------------------------------------------------
|
||
# Page helpers (the test_read_line_ranges / test_read_range_tool_lines
|
||
# house patterns)
|
||
# --------------------------------------------------------------------------
|
||
|
||
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
|
||
#: (a response clone read in the background) — wire-level assertions
|
||
#: for the ``tool`` / ``tool_result`` frames, independent of the UI
|
||
#: rendering.
|
||
SSE_HOOK = """
|
||
() => {
|
||
if (window.__sseInstalled) return;
|
||
window.__sseInstalled = true;
|
||
window.__sseFrames = [];
|
||
const origFetch = window.fetch;
|
||
window.fetch = async function (...args) {
|
||
const res = await origFetch.apply(this, args);
|
||
try {
|
||
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
|
||
if (url.includes('/api/chat')) {
|
||
res.clone().text().then((bodyText) => {
|
||
for (const block of bodyText.split('\\n\\n')) {
|
||
const line = block.trim();
|
||
if (line.startsWith('data: ')) {
|
||
window.__sseFrames.push(line.slice(6));
|
||
}
|
||
}
|
||
});
|
||
}
|
||
} catch (e) { /* non-clonable responses: ignored */ }
|
||
return res;
|
||
};
|
||
}
|
||
"""
|
||
|
||
|
||
def _install_page_hooks(page: Page) -> None:
|
||
page.evaluate(SSE_HOOK)
|
||
|
||
|
||
def _frames(page: Page, deadline_s: float = 120.0) -> list[dict]:
|
||
"""The SSE frames captured since the last submit (``_submit``
|
||
clears the buffer), once the hook's background read settles."""
|
||
deadline = time.monotonic() + deadline_s
|
||
while True:
|
||
raw = page.evaluate("() => window.__sseFrames || []")
|
||
parsed = [json.loads(line) for line in raw if line]
|
||
if any(f.get("type") == "done" for f in parsed):
|
||
return parsed
|
||
if time.monotonic() > deadline:
|
||
raise AssertionError(
|
||
f"SSE hook captured no `done` frame (frames so far: "
|
||
f"{len(parsed)}) — hook install failed?"
|
||
)
|
||
time.sleep(0.05)
|
||
|
||
|
||
def _tool_frames(frames: list[dict]) -> list[dict]:
|
||
return [f for f in frames if f.get("type") == "tool"]
|
||
|
||
|
||
def _result_frames(frames: list[dict]) -> list[dict]:
|
||
return [f for f in frames if f.get("type") == "tool_result"]
|
||
|
||
|
||
def _submit(page: Page, question: str) -> None:
|
||
page.evaluate("window.__sseFrames = []")
|
||
page.fill("#message-input", question)
|
||
page.click("#send-btn")
|
||
# The user bubble lands synchronously with the submit handler.
|
||
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
|
||
|
||
|
||
def _wait_settled(page: Page, timeout_ms: int = 120_000) -> None:
|
||
"""The turn is complete: answer text in the bubble, button
|
||
recovered (the phase-48 settle wait)."""
|
||
expect(page.locator(".msg.brain .bubble").last).not_to_have_text(
|
||
"", timeout=timeout_ms
|
||
)
|
||
expect(page.locator("#send-btn")).to_be_enabled(timeout=timeout_ms)
|
||
expect(page.locator("#send-label")).to_have_text("Send", timeout=timeout_ms)
|
||
|
||
|
||
def _last_brain(page: Page) -> Locator:
|
||
return page.locator(".msg.brain").last
|
||
|
||
|
||
# --------------------------------------------------------------------------
|
||
# 1. The owner's stuck scenario, browser-visible: two DIFFERENT ranges
|
||
# of the same document both reach the model (no refusal)
|
||
# --------------------------------------------------------------------------
|
||
|
||
|
||
def test_two_ranges_both_reach_the_model(
|
||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||
) -> None:
|
||
page.set_default_timeout(60_000)
|
||
login(page, app_url, next="/")
|
||
_install_page_hooks(page)
|
||
|
||
_submit(page, Q_A)
|
||
_wait_settled(page)
|
||
|
||
frames = _frames(page)
|
||
# THE SUBJECT on the wire — IN ORDER, asserted on the frames'
|
||
# ARGS (not a substring): the SAME path, TWO distinct ranges
|
||
# (offset 1/limit 100, then offset 101/limit 100 — the phase-131
|
||
# additive fields ride the SAME frame). EXACTLY two: a refused
|
||
# second read (the pre-phase bug) would re-issue the call and
|
||
# add a THIRD frame — or land as the mock's broken-premise echo.
|
||
assert _tool_frames(frames) == [
|
||
{
|
||
"type": "tool",
|
||
"name": "read",
|
||
"argument": SP,
|
||
"offset": 1,
|
||
"limit": 100,
|
||
},
|
||
{
|
||
"type": "tool",
|
||
"name": "read",
|
||
"argument": SP,
|
||
"offset": 101,
|
||
"limit": 100,
|
||
},
|
||
], _tool_frames(frames)
|
||
# NO ``tool_result`` frame: nothing truncated (the default cap —
|
||
# both 100-line slices are far under it; one frame = one
|
||
# noteworthy event, the phase-95 additive contract).
|
||
assert _result_frames(frames) == [], _result_frames(frames)
|
||
done = next(f for f in frames if f.get("type") == "done")
|
||
assert done["deflected"] is False, done
|
||
# The read document is the turn's cited source (phase 119 A1).
|
||
assert any(
|
||
s["path"] == REL and s["source"] == SOURCE for s in done["sources"]
|
||
), done["sources"]
|
||
|
||
# The GROUNDED echo — the SECOND range reached the model's
|
||
# context and landed in the answer: ``Both parts read.`` + the
|
||
# ``TWO_RANGES_B_LINE`` marker text (the marker is range B's
|
||
# first line — inside the 80-char read-echo quote).
|
||
text = _last_brain(page).locator(".bubble").text_content() or ""
|
||
assert ANSWER_LEAD in text, text
|
||
assert B_MARKER_TEXT in text, text
|
||
# NO refusal surface anywhere in the rendered answer: the
|
||
# ``ALREADY_IN_CONTEXT`` line (the pre-phase false claim — its
|
||
# full text is the refusal the owner quoted) and the mock's
|
||
# broken-premise echo are both absent.
|
||
assert ALREADY_IN_CONTEXT not in text, text
|
||
assert "Premise broke" not in text, text
|
||
|
||
|
||
# --------------------------------------------------------------------------
|
||
# 2. The single-range behavior is unchanged: the phase-131 grep →
|
||
# read(offset/limit) dance still lands on the coverage-tracked
|
||
# server (the phase-131 expectation, this KB)
|
||
# --------------------------------------------------------------------------
|
||
|
||
|
||
def test_single_range_flow_unchanged(
|
||
page: Page, app_url: str, synced_kb: None, db_ready: None
|
||
) -> None:
|
||
page.set_default_timeout(60_000)
|
||
login(page, app_url, next="/")
|
||
_install_page_hooks(page)
|
||
|
||
_submit(page, Q_B)
|
||
_wait_settled(page)
|
||
|
||
frames = _frames(page)
|
||
# The phase-131 dance on the wire — IN ORDER, asserted on the
|
||
# frames' ARGS: the grep (the mock's canned pattern), then the
|
||
# range read — path = the doc's combined identity, offset = the
|
||
# marker's TRUE line (the mock parsed it from the grep result's
|
||
# first match line), limit = READ_LINES_LIMIT (30).
|
||
assert _tool_frames(frames) == [
|
||
{"type": "tool", "name": "grep", "argument": READ_LINES_PATTERN},
|
||
{
|
||
"type": "tool",
|
||
"name": "read",
|
||
"argument": SP,
|
||
"offset": LINE_MARKER_LINE,
|
||
"limit": READ_LINES_LIMIT,
|
||
},
|
||
], _tool_frames(frames)
|
||
# The 30-line range slice did NOT truncate (the default cap) —
|
||
# no ``tool_result`` frame (the phase-131 pinned order contract).
|
||
assert _result_frames(frames) == [], _result_frames(frames)
|
||
done = next(f for f in frames if f.get("type") == "done")
|
||
assert done["deflected"] is False, done
|
||
assert any(
|
||
s["path"] == REL and s["source"] == SOURCE for s in done["sources"]
|
||
), done["sources"]
|
||
|
||
# The GROUNDED echo — the phase-131 answer shape unchanged:
|
||
# ``Found it at line <N>`` (N == the marker's true line) + the
|
||
# marker text itself (the single-range read reached the model's
|
||
# context exactly as before the coverage rework).
|
||
text = _last_brain(page).locator(".bubble").text_content() or ""
|
||
assert f"Found it at line {LINE_MARKER_LINE}." in text, text
|
||
assert LINE_MARKER_TEXT in text, text
|