phase: 118_summary_seed_context
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 14s

**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.

- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
This commit is contained in:
2026-09-16 06:57:49 -04:00
parent 21aad84a6d
commit 9820c361b0
80 changed files with 4690 additions and 1302 deletions
+211 -46
View File
@@ -55,14 +55,20 @@ Implements just enough of the aipi surface:
- system prompt containing ``<knowledge_base>`` (phase 31, KB overview)
-> the composed answer ends with `` (kb: <first bullet line>)`` —
the same echo convention for the overview's prompt injection.
- user message containing ``show the end of your notes`` (phase 24,
whole-document context) -> the answer quotes the **last 160 chars of
the ``<documents>`` block** — a tail echo, byte-stable across runs, so
a sentinel placed at the *end* of a document appears in the rendered
answer iff the whole document was in the prompt. (Phase 37: the HIGH
prompt now ends with a ``<tools>`` section after ``</documents>``, so
the echo targets the block itself; its tail still includes the
closing tag — same sentinel semantics.)
- user message containing ``show the end of your notes`` (phase 24;
phase 118 re-targeted — the story's dedicated suite
``tests/e2e/test_summary_seed_context.py``) -> the answer quotes
the **last 160 chars of the ``<documents>`` block** — a tail echo,
byte-stable across runs. (Phase 37: the HIGH prompt now ends with
a ``<tools>`` section after ``</documents>``, so the echo targets
the block itself; its tail still includes the closing tag.)
Phase 118 (A6): the block carries the suggested documents'
SUMMARIES (never full texts), so the echoed tail is the LAST
suggested document's SUMMARY tail (its digest + the
``Source: <source>/<path>`` pointer line) — a sentinel on a
document's *last line* appears in the rendered answer iff the
FULL content (not the summary) was in the prompt, which under the
summary-seed contract is only through a ``read`` tool result.
- user message containing ``use your tools`` (phase 37, agent document
tools; phase 70: the flow emits the harness-aligned names — ``ls``
/ ``read`` with the combined ``source/path`` identity; phase 94:
@@ -235,16 +241,19 @@ Implements just enough of the aipi surface:
content>`` (the phase-37 single-read shape — the grounded-
turn citation contract);
* the last tool result is the agent's ALREADY_IN_CONTEXT dedupe
refusal (the read target is already a top-2 retrieval
document — with the drill fixture that is DETERMINISTIC:
the read question names the file's path, so the file
self-matches the hybrid gate and its FULL text is in the
``<documents>`` prompt): the model answers FROM THE PROMPT —
the deterministic answer ``Already in context: Read
<source/path>. <first 80 chars of the target document's text
as it appears in the ``<documents>`` block>`` (same citation
shape as the read-result branch — the document text reached
the model either way, and the answer proves it);
refusal (phase 118: the read target is a document ALREADY
READ into full-text context earlier in the same turn — the
seeds are summaries, so a first read of any document
succeeds and only a re-read is refused): the model answers
FROM THE PROMPT — the deterministic answer ``Already in
context: Read <source/path>. <first 80 chars of the target
document's text as it appears in the ``<documents>`` block>``
(same citation shape as the read-result branch — the document
text reached the model either way, and the answer proves it;
the block it quotes now carries the document's SUMMARY).
No suite exercises this branch today (the drill questions
read once per turn) — it is kept for the still-real
already-read refusal;
* any other last result (a top-level or folder LISTING landed):
the deterministic ECHO answer ``Here's the level I listed:\n
<the listing, verbatim>`` — the mock echoes what it received
@@ -287,6 +296,38 @@ Implements just enough of the aipi surface:
needs the ``<tools>`` section, so deflected turns never hit it);
verified 2026-09-10: no existing E2E question or fixture file
contains the phrase, so every other suite is unaffected.
- user message containing ``read the suggested document``
(``SUMMARY_SEED_READ_TRIGGER``, phase 118 task 06 — the
summary-seed context's dedicated story suite
``tests/e2e/test_summary_seed_context.py``) **and** the system
prompt carries the ``<tools>`` section -> the deterministic
SCRIPTED SUMMARY-READ flow: the question carries its own tool call
after the colon — ``read the suggested document: read source/path``
— parsed by ``_SUMMARY_SEED_READ_CALL_RE`` from the RAW user
message (the target keeps its case), then discriminated
statelessly from the tool results (streaming only):
* request 1 (``tools`` offered, no ``tool``-role result in the
messages yet): the scripted call — ``read`` with the parsed
target (synthetic id ``call_0``);
* a ``tool``-role result is in the messages: the deterministic
ECHO — the answer carries the LAST tool result VERBATIM
(``Here's what the read returned:\n<result>``): under the
phase-118 summary-seed contract a first ``read`` of ANY
document succeeds (the seeds are summaries, not full text), so
a read result (``"Document <source/path>:…`` — header + the
phase-106 D5 ``date:`` line + the FULL content) lands in the
answer with its tail intact — the story suite's lens on the
full text the ``read`` tool delivered (a tail sentinel on the
document's last line appears in the answer iff the full content
reached the model through the read, not the seed); a refusal
(the premise broke) lands just as visibly, so the suite fails
loudly on it. The mock is the only E2E lens on the LLM's
context, so the echo is the assertion surface.
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
phrases — the phase-71/72/94 ordering convention; the trigger
needs the ``<tools>`` section, so deflected turns never hit it);
no existing E2E question or fixture file contains the phrase, so
every other suite is unaffected.
- user message containing ``what are the correct llama.cpp
arguments`` (``GREP_TEACH_TRIGGER``, the 2026-09-05 incident —
the harness prior is that grep takes a REGEX; this app's grep is a
@@ -802,10 +843,12 @@ _DRILL_CALL_RE = re.compile(
_NOT_A_FOLDER_MARKER = "is not a folder"
#: The stable substring of the harness-owned dedupe refusal the
#: drill-down flow's ``ctx_answer`` branch keys on (a read target that
#: is already a top-2 retrieval document — the mock then answers from
#: the document's text in the ``<documents>`` prompt block, exactly
#: what the refusal instructs). Keyed on a substring (not the whole
#: drill-down flow's ``ctx_answer`` branch keys on (phase 118: a read
#: target ALREADY READ into full-text context in the same turn — the
#: seeds are summaries, so a first read of any document succeeds and
#: only a re-read is refused; the mock then answers from the document's
#: summary in the ``<documents>`` prompt block, exactly what the
#: refusal instructs). Keyed on a substring (not the whole
#: constant) so a re-wrap of the constant cannot silently re-route the
#: mock; the module-level assert below fails loudly if the substring
#: ever leaves the constant (the mock must never drift from
@@ -819,11 +862,13 @@ assert _ALREADY_IN_CONTEXT_MARKER in ALREADY_IN_CONTEXT, (
#: section (``app.rag.prompts.build_high_prompt``): the block is the
#: document identity (``source``/``path``/``title`` attributes — plus,
#: since phase 106 D5, the ``date`` attribute, the row's ``created_at``
#: UTC date part, APPENDED after ``title``) plus the document's FULL
#: text (never truncated on the retrieval path, owner-locked A7)
#: between the tags. The ``date`` group is OPTIONAL so the mock
#: tolerates the pre- and post-phase block shapes (house rule: the
#: marker/regex lands with the prompt change).
#: UTC date part, APPENDED after ``title``) plus the document's
#: SUMMARY (phase 118, A6 — the summary-seed contract: the seeded
#: blocks are summaries, never full text; full text enters the context
#: only through the capped ``read`` tool) between the tags. The
#: ``date`` group is OPTIONAL so the mock tolerates the pre- and
#: post-phase block shapes (house rule: the marker/regex lands with
#: the prompt change).
_DOCUMENT_BLOCK_RE = re.compile(
r'<document source="(?P<source>[^"]+)" path="(?P<path>[^"]+)" '
r'title="[^"]*"(\sdate="[^"]*")?>\n(?P<content>.*?)\n</document>',
@@ -835,10 +880,14 @@ def _document_block(system: str, source: str, path: str) -> str | None:
"""The stored text of one ``<document>`` block (or ``None``).
The drill-down flow's ``ctx_answer`` branch: when the agent's read
of a top-2 retrieval document gets the ALREADY_IN_CONTEXT dedupe,
the document's full text is in the ``<documents>`` prompt — the
mock (the model) extracts it by the block's identity attributes
and quotes it, answering from the prompt as the refusal instructs.
of a document ALREADY READ in the same turn gets the
ALREADY_IN_CONTEXT dedupe (phase 118 — the seeds are summaries, so
a first read of any document succeeds and only a re-read is
refused), the mock (the model) extracts the block's text by the
identity attributes and quotes it, answering from the prompt as
the refusal instructs. The block carries the document's SUMMARY
(phase 118 A6) — no suite exercises the branch today; it is kept
for the still-real already-read refusal.
"""
for block in _DOCUMENT_BLOCK_RE.finditer(system):
if block.group("source") == source and block.group("path") == path:
@@ -1377,16 +1426,22 @@ def _drill_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
last = results[-1]
if last.startswith(_READ_RESULT_PREFIX):
head, _, content = last.partition("\n")
# The read result is ``"Document <source/path>:\n<content>"`` —
# the head carries the server's appended ``:`` (removed here;
# a document path never legitimately ends with one).
# The read result is ``"Document <source/path>:\ndate: …\n<content>"``
# — the head carries the server's appended ``:`` (removed here;
# a document path never legitimately ends with one); the
# phase-106 D5 ``date:`` line sits between the header and the
# document text — skipped so the quote stays pure document
# content (the suite's byte-identical pin).
combined = head[len(_READ_RESULT_PREFIX):].strip().removesuffix(":")
if content.startswith("date: "):
content = content.partition("\n")[2]
return ("read_answer", combined, content[:80])
if _ALREADY_IN_CONTEXT_MARKER in last and verb == "read" and "/" in target:
# The dedupe fired: the read target is already a top-2
# retrieval document, so its FULL text is in the
# ``<documents>`` prompt — answer from the prompt (the
# refusal's instruction), quoting the block's text.
# Phase 118: the dedupe fires only for a document ALREADY READ
# in the same turn (the seeds are summaries — a first read of
# a suggested document succeeds). Answer from the prompt (the
# refusal's instruction), quoting the block's text (now the
# document's summary).
src, _, p = target.partition("/")
content = _document_block(_system(body), src, p)
if content is not None:
@@ -1469,6 +1524,81 @@ def _read_cap_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
return ("echo", results[-1])
# ---------------------------------------------------------------------------
# Phase 118 (task 06, the summary-seed context's dedicated story suite):
# the deterministic SCRIPTED summary read — see the module docstring
# ---------------------------------------------------------------------------
#: A user message containing this substring (case-insensitive) —
#: combined with the ``<tools>`` section in the system prompt — drives
#: the scripted SUMMARY-READ flow (the summary-seed context's story
#: suite, ``tests/e2e/test_summary_seed_context.py``): the question
#: carries its own tool call after the colon — ``read the suggested
#: document: read source/path`` — the mock emits the scripted ``read``
#: (phase 118: the seeds are summaries, so a first read of a suggested
#: document SUCCEEDS — the full text arrives through the read), then
#: ECHOES the ENTIRE tool result into its answer (the house
#: scripted-turn lens on the LLM's context — the full content's tail
#: reaches the rendered answer iff the read delivered it). Checked
#: BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger phrases —
#: the phase-71/72/94 ordering convention); no existing E2E question
#: or fixture file contains the phrase, so every other suite is
#: unaffected.
SUMMARY_SEED_READ_TRIGGER = "read the suggested document"
#: The scripted call in the summary-read question (case-insensitive —
#: the suite's questions capitalize the trigger's first letter): the
#: verb (``read``) plus the target — a combined ``source/path``, parsed
#: from the RAW user message so the target keeps its case. The target
#: is a ``[a-z0-9_./-]`` run (case-insensitively), so the suite's
#: `` — `` flavor separator (em dash) can never bleed into it (the
#: read-cap/drill convention, ``_READ_CAP_CALL_RE`` / ``_DRILL_CALL_RE``).
_SUMMARY_SEED_READ_CALL_RE = re.compile(
r"read the suggested document:\s*read\s+(?P<arg>[a-z0-9_./-]+)",
re.I,
)
def _summary_seed_read_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
"""Classify a phase-118 scripted summary-read request (see the
module docstring). The question carries the scripted call (``read
the suggested document: read source/path``); the step is then
discriminated statelessly from the tool results, like the other
marker flows:
* ``("call", target, "call_0")`` — ``tools`` are offered and no
``tool``-role result is in the messages yet: the scripted
``read`` on the parsed target (synthetic id ``call_0``).
* ``("echo", result)`` — a ``tool``-role result is in the messages:
the deterministic ECHO — the answer carries the LAST tool result
VERBATIM (``Here's what the read returned:\n<result>``): a read
result (``"Document <source/path>:…`` — header + date line +
FULL content) lands in the answer with its tail intact (the
full text the ``read`` tool delivered — the phase-118 contract:
a first read of a suggested document succeeds, the seed was a
summary); a refusal (the premise broke) lands just as visibly,
so the suite fails loudly on it.
* ``None`` — not the flow: the trigger is absent, the ``<tools>``
section is missing (deflected turns never carry it), the scripted
call is unparseable, or ``tools`` are not offered and no tool
results are in the messages yet (e.g. ``agent_max_rounds=0``).
"""
user = _user(body)
if SUMMARY_SEED_READ_TRIGGER not in user.lower():
return None
if "<tools>" not in _system(body):
return None
match = _SUMMARY_SEED_READ_CALL_RE.search(user)
if match is None:
return None
results = _tool_results(body)
if not results:
if not body.get("tools"):
return None
return ("call", match.group("arg"), "call_0")
return ("echo", results[-1])
def long_answer() -> str:
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
a unique final line that must survive the stream untruncated."""
@@ -1924,14 +2054,19 @@ def compose_answer(body: dict[str, Any]) -> str:
"You've got this!"
)
elif END_OF_NOTES_TRIGGER in user.lower():
# Whole-document-context story (phase 24): echo the tail of the
# document context. Byte-stable across runs — a sentinel on the
# document's last line appears in the answer iff the whole
# document was in the prompt. (The tail includes the closing
# </documents> — harmless for the E2E sentinel assertions.)
# Phase 37: the HIGH prompt now ends with the <tools> section
# after </documents>, so the echo targets the <documents> block
# itself — the sentinel semantics are unchanged.
# Phase 24 (whole-document context) — phase 118 re-targeted
# (the summary-seed context's story suite): echo the tail of
# the <documents> block. Byte-stable across runs — under the
# phase-118 summary-seed contract the block carries the
# suggested docs' SUMMARIES, so the echoed tail is the LAST
# suggested doc's summary tail (digest + Source: pointer line),
# and a sentinel on a document's last line appears in the
# answer only if the FULL content reached the model (through a
# read tool result — never the seed). (The tail includes the
# closing </documents> — harmless for the E2E sentinel
# assertions.) Phase 37: the HIGH prompt ends with the <tools>
# section after </documents>, so the echo targets the
# <documents> block itself.
block = _DOCUMENTS_BLOCK_RE.search(_system(body))
tail_source = block.group(0) if block else _context(body)
answer = (
@@ -2461,6 +2596,36 @@ def chat_completions(body: dict[str, Any]) -> Any:
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
# Phase 118 (task 06): the deterministic SCRIPTED summary read
# (the question carries its own call — ``read the suggested
# document: read source/path``): the scripted ``read`` (phase
# 118: a first read of a suggested document succeeds — the
# seeds are summaries), then the answer that ECHOES the whole
# tool result (the full text's tail reaches the rendered answer
# iff the read delivered it — the suite's lens on the LLM's
# context). Checked BEFORE the plain TOOLS_TRIGGER flow
# (disjoint trigger phrases — the phase-71/72/94 ordering
# convention; the trigger needs the ``<tools>`` section, so
# deflected turns never hit it).
seed_read = _summary_seed_read_flow(body)
if seed_read is not None:
if seed_read[0] == "call":
stream = _tool_call_stream(
"read", {"path": seed_read[1]}, seed_read[2]
)
else: # "echo" — the last tool result verbatim (the lens)
stream = _sse_stream(
_apply_max_tokens(
f"Here's what the read returned:\n{seed_read[1]}",
body.get("max_tokens"),
),
0.0,
)
return StreamingResponse(
stream,
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
# Phase 109 (task 03, never-frozen-turn story suite): the
# deterministic REPORTED-REPRO turn (delta → tool →
# thinking-after-delta — TODO.md L3): the scripted first answer,
+7 -2
View File
@@ -44,8 +44,13 @@ reseelink.json" but does not include it):
(a ``documents`` row: it is in the agent's catalog, readable, and a
source-chip target) but seeded WITHOUT chunks. In a real
hundreds-of-document KB the file would simply fail to rank into the
top-2 context; with a two-document corpus every chunk would rank, so
"not in context" is expressed as "no retrieval candidates". Its
suggested (summary-seeded) context; with a two-document corpus every
chunk would rank, so "not in context" is expressed as "no retrieval
candidates". (Phase 118, A6: a read of a suggested doc now succeeds
anyway — the ALREADY_IN_CONTEXT refusal fires only for a document
already READ in the turn — so the catalog-only design stands on the
mock's first-catalog-line parse, not on the retired seed-read
refusal.) Its
``(source, path)`` also sorts FIRST in the catalog
(``Deployments`` < ``Homelab``) — which is exactly the line the mock
parses out of the listing and reads.
+72 -31
View File
@@ -46,9 +46,14 @@ Test → contract mapping (six tests, one per contract bullet):
the ``Indexed`` badge (D8 — the date at the top of the clicked
document), with the source/format/indexed/chunks badges intact.
4. ``test_old_correct_beats_new_similar`` — THE OWNER SCENARIO end to
end: the real retriever + the DEFAULT recency boost (0.0007 / 365 d)
over the mock's token-overlap embeddings ranks the OLDER correct
document as the first cited source over the newer similar one.
end (phase 118, A4): the real retriever + the DEFAULT recency boost
(0.0007 / 365 d) over the mock's token-overlap embeddings ranks the
OLDER correct document first in the suggested tier — the chip row IS
the suggested tier (top-5, NO floor; this four-doc KB therefore
chips ALL four docs in rank order: retention, draft, forward,
old-doc) and the related row is absent (no rank-6+ doc exists).
The newer similar doc (the boost's intended beneficiary) stays
second — the boost never lets it outrank the one that answers.
5. ``test_date_edit_and_sync_preserves`` — the admin-only editor in
the real UI: set → Save → the badge re-renders from the RESPONSE
(never the optimistic input) → the API round-trips; a re-import
@@ -69,9 +74,12 @@ deterministic for fixed text — measured with
``app.rag.retriever._vector_candidates`` / ``_lexical_candidates`` /
``retrieve()`` against a real Postgres + the mock:
* ``retention.md`` (correct, 2020) — rank 1 in BOTH lists (cosine
0.6222; the lexical tsquery ``how|did|i|configure|backup|retention|
policy`` after stopword removal matches it most densely).
* ``retention.md`` (correct, 2020) — rank 1 in BOTH lists (content
chunk cosine 0.6222; the lexical tsquery ``how|did|i|configure|
backup|retention|policy`` after stopword removal matches it most
densely). Phase 118 (A2): its EMBEDDED summary is a retrieval
candidate too (best-chunk cosine 0.7133 — the seeded summary ranks,
which is the point of seeding it).
* ``retention-draft.md`` (similar, now) — rank 4 in the vector list
(cosine 0.1443) and rank 2 in the lexical list. Its wording was
tuned for exactly this: it shares ONLY the three "backup retention
@@ -85,14 +93,18 @@ deterministic for fixed text — measured with
(0.1875) rank 2–3 in the vector list (unrelated content) and match
the tsquery not at all.
Fused (RRF k=60) + the default boost (0.0007 · exp(−age/365d)):
retention.md 1/61+1/61 = 0.0327878 (+ ≈ 0, 6.7 half-lives old) vs
retention-draft.md 1/64+1/62 = 0.0317540 (+ the FULL zero-age 0.0007
= 0.0324540) → the older correct doc wins by 0.000334 WITH the boost
on (it would win by 0.001034 with the boost off — the scenario holds
both ways; the boost never lets the newer similar doc outrank the one
that answers the question). ``select_documents`` (top-2) cites
retention.md first, the draft second. The boost defaults are owned by
Fused (RRF k=60) + the default boost (0.0007 · exp(−age/365d)),
probe-verified against the current candidate set (the embedded summary
chunks join the walk — phase 118 A2): retention.md 0.032523 vs
retention-draft.md 0.031498 (the full zero-age boost included) → the
older correct doc wins by ≈ 0.001025 WITH the boost on (the margin
only grew once the summary chunks ranked — the scenario holds both
ways; the boost never lets the newer similar doc outrank the one that
answers the question). The phase-118 suggested tier (top-5, NO floor)
carries all four docs in that rank order — retention.md first, the
draft second, then forward.md (0.016325) and old-doc.md (0.015874)
(no floor filters the unrelated docs); the related tier is EMPTY
(no rank-6+ doc in a four-doc KB). The boost defaults are owned by
task 07 — untouched here.
DB isolation: every test TRUNCATEs the KB tables (the
@@ -529,11 +541,14 @@ def test_old_correct_beats_new_similar(
) -> None:
"""``How did I configure the backup retention policy?`` → grounded,
and the FIRST cited source is the OLDER correct doc (2020) — the
NEWER similar one (now, "under review") is cited second. The real
hybrid retriever + the DEFAULT recency boost (0.0007 / 365 d) over
the mock's token-overlap embeddings (the module docstring records
the measured fused scores: 0.0327878 vs 0.0324540 — margin
0.000334 WITH the full zero-age boost on the newer doc)."""
NEWER similar one (now, "under review") is cited second. Phase 118
(A4): the chip row IS the suggested tier (top-5, NO floor) — this
four-doc KB chips ALL four docs in rank order — and the related row
is absent (no rank-6+ doc). The real hybrid retriever + the DEFAULT
recency boost (0.0007 / 365 d) over the mock's token-overlap
embeddings (the module docstring records the measured fused scores:
0.032523 vs 0.031498 — margin ≈ 0.001025 WITH the full zero-age
boost on the newer doc)."""
_reset_db(mock_llm, dates_tree)
page.set_default_timeout(30_000)
login(page, app_url, next="/") # phase 79: chat is require_user-gated
@@ -548,22 +563,34 @@ def test_old_correct_beats_new_similar(
# top_score 0.6222 ≥ the e2e threshold 0.30).
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
# The OLDER correct doc is the FIRST source chip; the NEWER similar
# one (the boost's intended beneficiary) is cited — but second.
# Phase 118 (A4): the chip row IS the suggested tier — top-5, NO
# floor, and this KB has exactly four docs, so ALL four are
# suggested (chipped) in fused rank order: the OLDER correct doc
# first, the NEWER similar one (the boost's intended beneficiary)
# second, then the two unrelated docs (no floor filters them — the
# LLM decides what the summaries earn).
chips = page.locator(".msg.brain .source-chip")
expect(chips).to_have_count(2, timeout=30_000)
expect(chips).to_have_count(4, timeout=30_000)
assert chips.nth(0).inner_text() == f"{dates_tree.name}/{RETENTION_MD}"
assert chips.nth(1).inner_text() == f"{dates_tree.name}/{DRAFT_MD}"
assert chips.nth(2).inner_text() == f"{dates_tree.name}/{FORWARD_MD}"
assert chips.nth(3).inner_text() == f"{dates_tree.name}/{OLDDOC_MD}"
# No rank-6+ doc exists in this four-doc KB → the related row is
# absent (the de-emphasized row renders only when it has entries).
expect(page.locator(".msg.brain .related-docs")).to_have_count(0)
# Durable record: one row, grounded, both docs cited in rank order.
# Durable record: one row, grounded, the FULL retrieval (suggested
# tier + related + read, deduped — here: all four docs) in rank
# order (LOCKED A3).
with SessionLocal() as db:
row = db.scalars(select(QueryLog)).one()
assert row.question == QUESTION
assert row.deflected is False
assert row.top_score >= 0.30 # the e2e mock-calibrated threshold
assert (row.fts_hits or 0) >= 1
assert f"{dates_tree.name}/{RETENTION_MD}" in row.sources
assert row.sources.index(RETENTION_MD) < row.sources.index(DRAFT_MD)
assert row.sources == ", ".join(
f"{dates_tree.name}/{p}" for p in (RETENTION_MD, DRAFT_MD, FORWARD_MD, OLDDOC_MD)
), row.sources
# ---------------------------------------------------------------------------
@@ -728,12 +755,20 @@ def test_anonymous_gate_and_editor_a11y(
" === document.querySelector('#doc-modal .doc-date-input')"
)
# Keyboard traversal runs through the editor's controls (the modal
# focus trap): Tab from Save → Cancel, and Tab from Revert wraps to
# the panel's first control (the trap's edge behavior). (The CDP
# key dispatch of headless Chromium does NOT perform the native
# focus move off <input type="date"> itself — a harness artifact,
# not a product defect: the same Tab works from every text input
# and button in the editor, pinned here through the buttons.)
# focus trap — the visible focusable order is "Full page" → Close →
# the content's controls, in DOM order): Tab from Save → Cancel,
# Tab from Revert → the summary editor's button (phase 118, A2: the
# markdown doc is summarized TOO, so the modal's summary-edit
# button is visible and joins the trap after the date editor — it
# did not exist in the focusable set when this pin was written,
# which is why the wrap below is now reached from IT), and Tab from
# the summary editor wraps to the panel's FIRST control, the
# "Full page" link (the trap's edge behavior the pin originally
# carried). (The CDP key dispatch of headless Chromium does NOT
# perform the native focus move off <input type="date"> itself — a
# harness artifact, not a product defect: the same Tab works from
# every text input and button in the editor, pinned here through
# the buttons.)
page.evaluate("() => document.querySelector('#doc-modal .doc-date-save').focus()")
page.keyboard.press("Tab")
assert page.evaluate(
@@ -742,6 +777,12 @@ def test_anonymous_gate_and_editor_a11y(
)
page.evaluate("() => document.querySelector('#doc-modal .doc-date-revert').focus()")
page.keyboard.press("Tab")
assert page.evaluate(
"() => !!document.activeElement"
" && document.activeElement.classList.contains('doc-summary-edit')"
)
page.evaluate("() => document.querySelector('#doc-modal .doc-summary-edit').focus()")
page.keyboard.press("Tab")
assert page.evaluate(
"() => !!document.activeElement"
" && document.activeElement.classList.contains('doc-modal-open')"
+90 -47
View File
@@ -1,4 +1,5 @@
"""Phase 30 E2E (Playwright): a summary hit delivers the full source doc.
"""Phase 30 E2E (Playwright) — phase 118 re-targeted (A2/A6): a summary
hit seeds the SUMMARY into the prompt — never the full source doc.
Story: ``.agents/user_stories/document-summaries.md``
Run in isolation (DB must be up: ``podman compose up -d db``):
@@ -7,27 +8,33 @@ Run in isolation (DB must be up: ``podman compose up -d db``):
The fixture KB is a story-dedicated directory
(``tests/fixtures/summary_kb/`` — the shared ``tests/fixtures/docs/``
stays at its 9 pinned files) with two documents:
stays at its pinned files) with two documents:
* ``quadlet/qwen-llamacpp.yaml`` — a non-markdown A9 doc. At import the
mock ``lite`` model (``SUMMARY_MODE`` marker, ``tests/e2e/mock_llm.py``)
reduces it to a deterministic 24-token digest, stored on
``documents.summary`` and indexed as one ``is_summary`` chunk. The raw
yaml body is deliberately token-diluted, so the document's best fused
chunk is its summary chunk. The sentinel ``RESE-SUMMARY-SENTINEL-7f3a``
sits on the document's LAST line — outside the 24-token digest,
unreachable from the summary.
* ``notes/qwen-llamacpp-notes.md`` — a markdown control doc (never
summarized) that ranks first, which puts the yaml document LAST inside
``<documents>``.
* ``quadlet/qwen-llamacpp.yaml`` — at import the mock ``lite`` model
(``SUMMARY_MODE`` marker, ``tests/e2e/mock_llm.py``) reduces it to a
deterministic 24-token digest, stored on ``documents.summary`` and
indexed as one ``is_summary`` chunk. The raw yaml body is deliberately
token-diluted, so the document's top fused chunks are the summary and
the one lexical-hitting raw chunk. The sentinel
``RESE-SUMMARY-SENTINEL-7f3a`` sits on the document's LAST line —
outside the 24-token digest, unreachable from the summary.
* ``notes/qwen-llamacpp-notes.md`` — a markdown doc (phase 118 A2: it is
summarized TOO — the phase-30 non-markdown-only scope is retired) that
ranks first, which puts the yaml document LAST inside
``<documents>`` (a two-doc KB → both docs are suggested, the related
tier is empty).
The mock LLM's tail-echo trigger (``END_OF_NOTES_TRIGGER``) makes the
answer quote the last 160 chars of the document context — the tail of
the LAST selected document. The sentinel therefore appears in the
rendered answer **iff the entire yaml source document (not the summary
digest) reached the LLM prompt** — the summary→parent-document resolution
through the unchanged chunk→document mapping (A7 revised: never
truncated), which is what this story is about.
answer quote the last 160 chars of the seeded ``<documents>`` block —
under the phase-118 summary-seed contract (A6) that block carries the
suggested docs' SUMMARIES, never their full texts, so the echoed tail is
the LAST suggested doc's summary (the yaml doc's byte-stable digest tail
+ ``Source:`` pointer line). The sentinel therefore appears in the
rendered answer **only if the entire yaml source document (not the
summary) reached the LLM prompt** — under the locked contract it must be
ABSENT (full text enters the context only through the capped ``read``
tool), which is the inverse of the retired phase-30/24 full-text pin
and what this story is about now.
"""
from __future__ import annotations
@@ -47,7 +54,7 @@ from app.models import Document, QueryLog
from app.rag.chunker import chunk_document
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from app.rag.retriever import RetrievedChunk, retrieve
from app.rag.retriever import RetrievedChunk, retrieve, select_suggested
from e2e.auth_helpers import login
from tests.e2e.mock_llm import TOKEN_RE, embed_text
@@ -156,17 +163,20 @@ def _chunks_by_path(chunks: Sequence[RetrievedChunk], path: str) -> list[Retriev
# --- Story tests -------------------------------------------------------------
def test_summary_hit_retrieves_full_source_document(
def test_summary_hit_seeds_the_summary_not_the_full_text(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""A question whose best yaml match is the summary chunk yields an
answer grounded in the FULL yaml source document: its tail sentinel —
which the summary digest cannot contain — is echoed back, and the
source chip cites the yaml path (deflected: false)."""
"""A question whose best yaml match is its summary chunk yields a
grounded answer seeded from the SUMMARY — the mock's tail echo
quotes the last suggested doc's summary tail (digest + pointer), and
the sentinel (the yaml doc's last line, outside the digest) is
ABSENT: the full source doc never reached the prompt (A6). Both
fixture docs are suggested (two-doc KB, no floor) and both chips
render (deflected: false)."""
_reset_db()
summary = _run_in_thread(_import_fixtures(mock_llm))
assert summary.added == 2 # yaml + md control
assert summary.summaries == 1 and summary.summary_errors == 0
assert summary.summaries == 2 and summary.summary_errors == 0 # A2: md too
assert summary.errors == 0
# Import state: exactly one embedded ``is_summary`` chunk (position
@@ -182,30 +192,48 @@ def test_summary_hit_retrieves_full_source_document(
assert schunks[0].embedding is not None
assert yaml_doc.summary == _expected_summary(yaml_content, SOURCE, YAML_PATH)
# Retrieval state: the summary chunk is the yaml document's best fused
# chunk — the document enters the context through its summary, not
# through the diluted raw yaml chunks.
# Retrieval state (phase 118, A2/A6): the EMBEDDED summary chunk is
# a retrieval candidate (the seeded text ranks on its own), and the
# suggested tier is the two docs in rank order — md first, yaml
# LAST (so the yaml's summary is the tail of the <documents> block,
# the mock echo's target). The document enters the prompt through
# its SUMMARY block, not through the diluted raw yaml chunks.
with SessionLocal() as db:
chunks = retrieve(db, QUESTION, embed_text(QUESTION))
yaml_chunks = _chunks_by_path(chunks, YAML_PATH)
best_yaml = max(yaml_chunks, key=lambda c: c.score)
assert best_yaml.is_summary
assert any(c.is_summary for c in yaml_chunks) # the embedded summary ranks
assert len(yaml_chunks) >= 2 # summary + at least one raw candidate
assert [d.path for d in select_suggested(chunks)] == [MD_PATH, YAML_PATH]
bubble = _ask(page, app_url, QUESTION)
# The tail sentinel exists only on the document's last line and
# cannot be in the summary digest — its presence proves the entire
# source document was in the LLM prompt (summary→parent resolution).
expect(bubble).to_contain_text(SENTINEL, timeout=30_000)
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
# Phase 118 (A6): the seed is the SUMMARY, not the full text — the
# mock's tail echo quotes the last 160 chars of the <documents>
# block, which end in the LAST suggested doc's summary: the yaml
# doc's byte-stable digest tail + pointer line. The sentinel
# (document's last line, outside the digest) is therefore ABSENT —
# the full source doc never reached the prompt (full text enters
# only through the capped read tool; the inverse of the retired
# phase-30/24 full-text pin). The bubble renders the answer as
# markdown, which collapses the summary's newline — so pin each
# LINE separately (the digest line's tail sits inside the echoed
# 160 chars; the pointer line is single-line too).
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
yaml_summary = _expected_summary(yaml_content, SOURCE, YAML_PATH)
yaml_digest_line = yaml_summary.split("\n", 1)[0]
expect(bubble).to_contain_text(f"Source: {SOURCE}/{YAML_PATH}")
expect(bubble).to_contain_text(yaml_digest_line[-80:])
expect(bubble).not_to_contain_text(SENTINEL)
# Grounded: the yaml source chip renders (the md doc ranks first, so
# both fixtures are cited).
# Grounded: both fixture docs are suggested (two-doc KB — no floor)
# and both chips render, in rank order (md first, yaml last).
chip = page.locator(".msg.brain .source-chip", has_text=YAML_PATH)
expect(chip).to_have_count(1)
expect(chip.first).to_contain_text(f"{SOURCE}/{YAML_PATH}")
expect(page.locator(".msg.brain .source-chip", has_text=MD_PATH)).to_have_count(1)
expect(page.locator(".msg.brain .source-chip")).to_have_count(2)
# No rank-6+ doc in a two-doc KB → the related row is absent.
expect(page.locator(".msg.brain .related-docs")).to_have_count(0)
# Button recovers (never stale) and the turn was grounded, not
# deflected.
@@ -214,20 +242,26 @@ def test_summary_hit_retrieves_full_source_document(
row = _last_query_log()
assert row.question == QUESTION
assert row.deflected is False
assert f"{SOURCE}/{YAML_PATH}" in row.sources
assert f"{SOURCE}/{MD_PATH}" in row.sources
# The durable record: suggested + related + read (deduped) — here
# exactly the two suggested docs, in rank order (LOCKED A3).
assert row.sources == (
f"{SOURCE}/{MD_PATH}, {SOURCE}/{YAML_PATH}"
), row.sources
def test_markdown_control_doc_gets_no_summary_chunk(
def test_markdown_control_doc_gets_a_summary_chunk(
mock_llm: int, db_ready: None
) -> None:
"""Control: in the same KB the markdown doc gets no summary at all —
its chunk count is exactly the raw chunks; the yaml doc has exactly
one ``is_summary`` row and its raw chunk count is untouched by the
summary."""
"""Phase 118 (A2): in the same KB the markdown doc gets a summary TOO
— the phase-30 non-markdown-only scope is retired (markdown docs
backfill + summarize like every other doc): the same byte-stable
digest + pointer line, exactly one ``is_summary`` chunk, and the raw
chunk count untouched. The yaml doc keeps its exactly-one summary
row with its raw chunks untouched."""
_reset_db()
summary = _run_in_thread(_import_fixtures(mock_llm))
assert summary.added == 2
assert summary.summaries == 2 # A2: the markdown doc is summarized too
md_content = (FIXTURES / MD_PATH).read_text(encoding="utf-8")
yaml_content = (FIXTURES / YAML_PATH).read_text(encoding="utf-8")
@@ -235,11 +269,20 @@ def test_markdown_control_doc_gets_no_summary_chunk(
md_doc = _doc(db, MD_PATH)
yaml_doc = _doc(db, YAML_PATH)
md_chunks = [c for c in md_doc.chunks if not c.is_summary]
md_summary = [c for c in md_doc.chunks if c.is_summary]
yaml_raw = [c for c in yaml_doc.chunks if not c.is_summary]
yaml_summary = [c for c in yaml_doc.chunks if c.is_summary]
# Markdown: never summarized (phase 30 scope — A9 non-markdown only).
assert md_doc.summary is None
# Markdown: NOW summarized (phase 118 A2) — the same byte-stable
# digest + deterministic pointer line, exactly one is_summary chunk
# (position −1, embedded), raw chunks untouched.
expected_md_summary = _expected_summary(md_content, SOURCE, MD_PATH)
assert md_doc.summary == expected_md_summary
assert len(md_summary) == 1
assert md_summary[0].position == -1
assert md_summary[0].embedding is not None
assert md_summary[0].content == expected_md_summary
assert expected_md_summary.endswith(f"\nSource: {SOURCE}/{MD_PATH}")
assert len(md_chunks) == len(
chunk_document(md_content, MD_PATH, CHUNK_TARGET, CHUNK_OVERLAP)
)
+7 -1
View File
@@ -201,7 +201,13 @@ def test_viewer_header_content_still_fits(
assert _box_height(page, "#doc-title") < 30 # one line at either size
# Meta row: badges + path present and visible, single line.
assert _box_height(page, ".doc-meta") < 26
# The bound catches a WRAP (a second line would be ≥ ~44px),
# not a pixel-exact line height: the row is content-sized by
# its tallest child — the native type=date input (phase 106
# D8), whose metrics are Chromium/font-dependent and render
# the single line at ~26px on this host (the pre-phase 26px
# pin red-lined on the 26.125px measurement, 2026-09-16).
assert _box_height(page, ".doc-meta") < 34
expect(page.locator(".doc-source-badge", has_text="docs")).to_be_visible()
expect(page.locator(".format-badge", has_text="md")).to_be_visible()
expect(page.locator(".doc-path", has_text="homelab/kubernetes.md")).to_be_visible()
+30 -21
View File
@@ -57,14 +57,18 @@ Test → story mapping (Playwright Mapping Rule):
``source: X | path: Y | title: Z`` file lines), and the grounded
``read(alpha/two/two-a.md)`` (the ``📄 Reading …`` line + the answer
citing the document, the phase-37 assertion pattern). The read
target is DELIBERATELY a top-2 retrieval document for its question
(the question names the file's path, so the file self-matches the
hybrid gate deterministically): the agent's phase-72 dedupe returns
``ALREADY_IN_CONTEXT`` (a refusal — counts in nothing), and the
mock answers FROM THE ``<documents>`` PROMPT with the same citation
shape (``Already in context: Read <sp>. <first 80 chars>``) — the
document text reached the model either way, and the prefix pins
that the dedupe notice itself reached it.
target is DELIBERATELY a suggested (summary-seeded) document for
its question (the question names the file's path, so the file
self-matches the hybrid gate and takes rank 1 deterministically):
phase 118 (A6) — the seed is a SUMMARY, not full text, so the read
SUCCEEDS (the phase-72 ALREADY_IN_CONTEXT dedupe fires only for a
document already READ in the turn — the retired top-2 seed-read
refusal is gone) and the full text arrives through the ``read``
tool; the mock answers from the READ RESULT with the phase-37
citation shape (``Read <sp>. <first 80 chars of the document
content>`` — the mock skips the phase-106 D5 ``date:`` line, so the
quote is pure content, byte-identical to the retired
answer-from-prompt quote).
2. ``test_wide_folder_holds_the_fifty_line_cap`` — ``ls(alpha/wide)``
on the 51-file folder: the mock's echo carries exactly 50 file lines
+ the ``…and 1 more documents in this folder — use grep (pattern)…``
@@ -226,12 +230,16 @@ READ_QUESTION = f"Drill down the tree: read {READ_SP} — read the file"
WIDE_QUESTION = f"Drill down the tree: ls {ALPHA}/wide — how many files does this folder hold?"
NOPE_QUESTION = f"Drill down the tree: ls {ALPHA}/nope — is there such a folder?"
#: The read target is a top-2 retrieval document for its question (the
#: question names the path — the file self-matches the hybrid gate
#: deterministically), so the read gets the phase-72 ALREADY_IN_CONTEXT
#: dedupe and the mock answers from the ``<documents>`` prompt with the
#: same citation shape, prefixed (the suite pins the dedupe path).
READ_ANSWER_PREFIX = f"Already in context: Read {READ_SP}."
#: The read target is a suggested (summary-seeded) document for its
#: question (the question names the path — the file self-matches the
#: hybrid gate and takes rank 1 deterministically). Phase 118 (A6):
#: the seed is a summary, not full text — a first read of a suggested
#: document SUCCEEDS, the full text arrives through the read tool, and
#: the mock answers from the READ RESULT with the phase-37 citation
#: shape (the mock skips the phase-106 D5 ``date:`` line, so the quote
#: is pure document content — byte-identical to the retired
#: answer-from-prompt quote, which the suite pins).
READ_ANSWER_PREFIX = f"Read {READ_SP}."
READ_ANSWER_QUOTE = TWO_A_CONTENT[:80]
# --------------------------------------------------------------------------
@@ -658,14 +666,15 @@ def test_drill_down_sources_folders_files_and_read(
{"type": "tool", "name": "read", "argument": READ_SP}
], _tool_frames(frames)
bubble = _last_brain(page).locator(".bubble")
# The dedupe notice reached the model (the prefix pins it — the read
# was refused as ALREADY_IN_CONTEXT because the target is a top-2
# retrieval document)…
# Phase 118 (A6): the read SUCCEEDED — the seeds are summaries,
# not full text, so the phase-72 ALREADY_IN_CONTEXT refusal (only
# for a document already READ in the turn) did not fire —
expect(bubble).to_contain_text(READ_ANSWER_PREFIX)
# …and the answer still cites the document: the mock quotes the
# FIRST 80 chars of the target's text from the ``<documents>``
# prompt (the refusal's instruction — answer from that text; the
# quote is newline-free, pinned above).
expect(bubble).not_to_contain_text("Already in context")
# — and the answer cites the document from the READ RESULT: the mock
# quotes the FIRST 80 chars of the document's own content (the full
# text arrived through the read tool; the quote is newline-free,
# pinned above).
expect(bubble).to_contain_text(READ_ANSWER_QUOTE)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
+5 -2
View File
@@ -623,8 +623,11 @@ def test_api_created_chats_carry_the_bor_chat_v1_shape(
assert body["message_count"] == 2
# The bor.chat.v1 record shape: who/text present; only the schema's
# optional keys may accompany them (explicit nulls are preserved —
# the plain model_dump round-trips byte-identical).
allowed = {"who", "text", "sources", "deflected", "suggestions",
# the plain model_dump round-trips byte-identical). ``related`` is
# the phase-113 related-doc tier the UI persists with grounded
# brain records (accepted by the ChatMessage schema — its earlier
# absence 422'd the done-time auto-save, phase-118 verification fix).
allowed = {"who", "text", "sources", "related", "deflected", "suggestions",
"thinking", "tools", "stopped"}
assert all({"who", "text"} <= set(m) <= allowed for m in body["messages"])
# The History row renders it (the Open link's text IS the title).
+30 -52
View File
@@ -36,16 +36,14 @@ the same host) registered as a local-directory source (the
``test_local_directory_sources.py`` registration + real-Sync pattern —
registration through the authenticated API, the real in-process
``POST /api/sync`` pipeline; no git anywhere), with THREE documents
whose bodies are token-controlled so the phase-72 ALREADY_IN_CONTEXT
dedupe (the read target is refused when it is a top-2 retrieval seed)
NEVER fires — each scripted read target must actually execute, not be
refused:
whose bodies are token-controlled so every scripted turn stays
GROUNDED (the hybrid gate HIGH — the mock's read-cap flow needs the
``<tools>`` section):
* ``anchor-2024.md`` — the retrieval ANCHOR: a digit-bearing name.
Both questions name "anchor 2024", whose joined normalized token
(``anchor2024``) name-hits this document — the name-hit list LEADS
the lexical side, so the anchor is the #1 seed of BOTH turns
(grounding: the name hit's ``fts_hit`` keeps the gate HIGH, the
(``anchor2024``) name-hits this document — the name hit's
``fts_hit`` keeps the gate HIGH for every turn (grounding: the
``<tools>`` section is present, and the anchor is never a read
target);
* ``zz-capped.md`` — the SUBJECT: ~3 100 chars of varied rotation
@@ -53,18 +51,19 @@ refused:
``chars_total`` the assertions use is ``len(CAP_DOC)`` of this very
string, and ``synced_kb`` pins the stored content byte-identical to
it). Its body avoids EVERY token of both questions, so on its own
(turn 1) question it has zero lexical hits and only the common-word
cosine — it is the #3 fused candidate, NOT a seed;
question it has zero lexical hits and only the common-word cosine —
it ranks below the anchor (irrelevant to the cap's contract: phase
118, A6 — the seeds are SUMMARIES, so a first ``read`` of ANY
document, suggested or not, succeeds, and the cap applies to
everything ``read`` returns — the retired top-2 seed-read refusal
that used to constrain this design is gone);
* ``aa-short.md`` — the CONTROL: ~200 chars, under the cap.
The turn-specific flavor words pick the #2 seed: turn 1's question
carries ``note, long form`` (planted in the SHORT doc's body) and turn
2's carries ``quick pass`` (planted in the CAPPED doc's body), so each
turn's NON-target document wins the second seed slot on its own
question and the target stays #3/#6. ``synced_kb`` pins this design
with the app's real hybrid retrieval (``_assert_target_not_seed`` — a
fixture-text regression that makes a target a seed fails at setup with
a clear message, not at the wire assertions).
The turn-specific flavor words (``note, long form`` planted in the
SHORT doc's body, ``quick pass`` in the CAPPED doc's) are inert
leftovers of the retired not-a-seed design — they no longer select a
seed slot (phase 118: a read of a suggested document succeeds, so the
target's seed status never changes the wire contract).
Test → story mapping (Playwright Mapping Rule; the story is the owner
TODO item — one Playwright file per story, A16):
@@ -116,7 +115,6 @@ from e2e.conftest import (
USE_REAL_LLM,
_wait_http,
)
from e2e.mock_llm import embed_text
REPO = Path(__file__).resolve().parents[2]
@@ -163,9 +161,10 @@ def _cap_doc() -> str:
every token of BOTH questions (``read``, ``capped``, ``document``,
``capkb``, ``zz``, ``md``, ``anchor``, ``2024``, ``note``,
``long``, ``form``, ``aa``, ``short``, ``quick``, ``pass``,
``wire``, ``save``, ``control``, ``check``) — the target must have
zero lexical hits on its own turn so the dedupe cannot refuse the
read."""
``wire``, ``save``, ``control``, ``check``) — the target has zero
lexical hits on its own turn, so it ranks below the anchor (phase
118: its seed status is irrelevant — a first read of any document
succeeds; the anchor keeps the gate HIGH)."""
topics = [
"vault", "mirror", "raid", "pool", "drive",
"chain", "slot", "cycle", "guard", "probe",
@@ -226,7 +225,8 @@ NOTICE = READ_TRUNCATION_NOTICE.format(shown=READ_CAP, total=CAP_DOC_TOTAL)
# The scripted turns (the mock's ``READ_CAP_TRIGGER`` questions — each
# carries its own tool call after the colon; the turn-specific flavor
# words are the #2-seed selectors, see the module docstring).
# words are inert leftovers of the retired #2-seed design — see the
# module docstring).
Q_WIRE = (
f"Read the capped document: read {CAPPED_SP} — "
"anchor 2024 note, long form, wire check"
@@ -359,26 +359,6 @@ def _truncate_all() -> None:
db.commit()
def _assert_target_not_seed(question: str, target_rel: str) -> None:
"""Pin the retrieval-anchor design (see the module docstring) with
the app's REAL hybrid retrieval over the mock's embeddings
(deterministic): the scripted read target must NOT be a top-2 seed
for its own question — the phase-72 ALREADY_IN_CONTEXT dedupe would
refuse the read and the cap would never fire (the turn would echo
the refusal instead). A fixture-text regression that breaks this
fails here, at setup, with a clear message."""
from app.rag.retriever import retrieve, select_documents
with SessionLocal() as db:
seeds = select_documents(retrieve(db, question, embed_text(question)))
paths = [f"{d.source}/{d.path}" for d in seeds]
assert f"{SOURCE}/{target_rel}" not in paths, (
f"the read target {SOURCE}/{target_rel} is a top-2 seed for its own "
f"question — the ALREADY_IN_CONTEXT dedupe would refuse the read and "
f"the cap would never fire (seeds: {paths})"
)
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
"""Poll the (cookie-authenticated) status endpoint until the run
reaches a terminal state (the test_ls_tree_drilldown pattern, over
@@ -403,8 +383,10 @@ def synced_kb(app_server: str, cap_dirs: Path) -> None:
runs the REAL in-process sync (``POST /api/sync`` — walk → chunk →
embed → overview → version bump), pins the stored content
byte-identical to the fixture strings (the ``chars_total``
assumption), and pins the retrieval-anchor design for all three
scripted questions (the dedupe never fires)."""
assumption). Phase 118: the retrieval-anchor design only pins the
gate's grounding (the anchor keeps every turn HIGH) — a read of a
suggested document succeeds, so the targets' seed status no longer
constrains the suite."""
_truncate_all()
with httpx.Client(base_url=app_server, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
@@ -436,11 +418,6 @@ def synced_kb(app_server: str, cap_dirs: Path) -> None:
)
)
assert stored == expected, f"stored content drifted for {rel}"
# The retrieval-anchor design (the module docstring): every scripted
# read target stays OUT of its own question's top-2 seeds.
_assert_target_not_seed(Q_WIRE, CAPPED_REL)
_assert_target_not_seed(Q_SAVE, CAPPED_REL)
_assert_target_not_seed(Q_CONTROL, SHORT_REL)
@pytest.fixture(autouse=True)
@@ -681,9 +658,10 @@ def test_truncated_read_frame_order_live_marker_and_llm_notice(
assert i_tool < i_result < i_delta, (i_tool, i_result, i_delta)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The read document is the turn's cited source even though it was
# NOT a retrieval seed (the anchor design) — retrieval + agent-read,
# deduped (the grounded-turn record).
# The read document is the turn's cited source (retrieval +
# agent-read, deduped — the grounded-turn record; phase 118: the
# target may be a suggested (summary-seeded) document — its seed
# status never changes the cap's wire contract).
assert any(
s["path"] == CAPPED_REL and s["source"] == SOURCE for s in done["sources"]
), done["sources"]
+4 -3
View File
@@ -346,9 +346,10 @@ def test_anonymous_shared_view(
expect(think.first).not_to_have_attribute("open")
# Source chips are PLAIN TEXT: the on-topic turn carries its
# source chips (top-2 docs — the hybrid retrieval), but every
# one as a <span>: zero <a.source-chip> anywhere (a guest
# cannot open documents; the documents API is admin-only).
# source chips (the suggested docs — the hybrid retrieval's
# summary-seed tier, phase 118), but every one as a <span>:
# zero <a.source-chip> anywhere (a guest cannot open documents;
# the documents API is admin-only).
assert (
anon.locator(".msg.brain .source-chip").count() >= 1
), "the grounded turn must carry its source chips"
+96 -59
View File
@@ -1,35 +1,44 @@
"""Phase 113 E2E (Playwright): the source-chip quality contract (TODO L5 +
L2c) — the usefulness bar + the de-emphasized related-docs row, as VISIBLE
chip counts.
"""Phase 113 E2E (Playwright) — phase 118 re-targeted: the citation-surface
contract (LOCKED A4) as VISIBLE chip counts — the summary-seed contract
replaced the phase-112/113 usefulness bar: the chip row is the
SUGGESTED tier (top-5 distinct docs, NO floor) + agent reads (none on
these turns), the related row is rank 6+ (capped at
``related_max_docs`` = 2), and a deflected turn still cites nothing
(done.sources = [] — its weak hits are suggested for the durable record
but are never citation chips).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_source_chip_quality.py -v --no-cov
Acceptance (TODO L144–146): "for a single-document question, the turn
shows one citation chip"; a weak 2nd doc renders only in the
de-emphasized related row (``.related-doc`` links, NEVER ``.source-chip``);
a deflected turn shows zero citation chips (its weak hits, if any, live in
the related row).
Acceptance (TODO L144–146, phase-118 shape): a grounded turn's chip row
is the suggested tier — the 5 docs the model was seeded with (its
"start here" set — the old "one chip per bar-clearing doc" is retired
with the full-text seeds; the LLM decides what the summaries earn);
a deflected turn shows zero citation chips (its weak hits, if any, live
in the de-emphasized related row — ``.related-doc`` links, NEVER
``.source-chip``; L2c: never render weak hits as answer citations).
The fixture KB's tier shapes are deterministic under the E2E mock's
bag-of-words embeddings + the mock-calibrated bar (conftest:
``BOR_SOURCE_USEFULNESS_FLOOR=0.15``, half the 0.30 threshold — like the
lexical floor):
bag-of-words embeddings + the fused rank walk (probe-verified, pinned
here against the wire):
* **single-source question** — "What SSH aliases do I have?":
``ssh_aliases.txt`` is the ONLY doc whose best-chunk cosine clears the
bar (0.352 ≥ 0.15; grounded at 0.352 ≥ 0.30) → the done frame carries
exactly ONE cited ref; the two below-bar docs (gitlab-compose.yaml
0.025, uptime_probe.py 0.113) ride the related tier. This IS the
strong+weak two-tier shape on the wire (the four OBSERVED live shapes
are unit-pinned at plan level in
``tests/unit/test_source_chip_quality.py`` — the fixture KB reproduces
the same shape live, so no docstring caveat is needed).
* **grounded question** — "What SSH aliases do I have?": best cosine
0.352 ≥ 0.30 → grounded; the suggested tier (top-5, NO floor) is
ssh_aliases.txt > gitlab-compose.yaml > uptime_probe.py > kubernetes.md
> backups.md (fused rank — the lexical-only cosine-0.0 docs rank when
they rank: no floor filters them, LOCKED A3) and the done frame
carries EXACTLY those five cited refs; the rank-6+ remainder
(compose.container, static-dns.json) rides the related row, capped at
two. (The four OBSERVED live shapes are unit-pinned at plan level in
``tests/unit/test_source_chip_quality.py`` — the fixture KB
reproduces the tiered shape live, so no docstring caveat is needed.)
* **deflected question** — "How do I bake sourdough bread?": best cosine
0.109 < 0.30 and zero FTS hits → honest deflection; nothing clears the
bar → zero cited refs; the weak hits (new-service.md 0.109,
ssh_aliases.txt 0.050) ride the related row.
0.109 < 0.30 and zero FTS hits → honest deflection → zero cited refs
(done.sources = []); the weak hits ARE suggested (no floor) for the
durable record — new-service.md > ssh_aliases.txt > lan.network >
uptime_probe.py > compose.container — and the rank-6+ remainder
(backups.md, kubernetes.md) rides the related row.
"""
from __future__ import annotations
@@ -97,12 +106,14 @@ def _ask(page: Page, message: str) -> None:
def test_single_source_question_shows_exactly_one_citation_chip(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""The acceptance pin: a single-document question → the done bubble
carries EXACTLY ONE ``.source-chip`` (the bar-clearing doc) and the
below-bar docs render only in the de-emphasized ``.related-docs`` row
(``.related-doc`` links — never ``.source-chip``), labeled
"Nearby docs, in case:". The durable record keeps the FULL retrieval
(LOCKED A3)."""
"""The acceptance pin (phase 118, A4): a grounded question → the done
bubble carries EXACTLY the suggested tier as ``.source-chip``s (the
five docs the model was seeded with — top-5, NO floor: the old
usefulness-bar "one chip" is retired with the full-text seeds) and
the rank-6+ remainder renders only in the de-emphasized
``.related-docs`` row (``.related-doc`` links — never
``.source-chip``), labeled "Nearby docs, in case:". The durable
record keeps the FULL retrieval (LOCKED A3)."""
_reset_db(mock_llm)
page.set_default_timeout(30_000)
login(page, app_url, next="/") # phase 79: chat is require_user-gated
@@ -116,52 +127,64 @@ def test_single_source_question_shows_exactly_one_citation_chip(
# Grounded: no deflected bubble at all.
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
# EXACTLY ONE citation chip — the bar-clearing doc (the 2nd chip of
# the pre-phase turn demoted to the row; the acceptance criterion).
# The citation surface IS the suggested tier (LOCKED A4 — top-5,
# NO floor): five chips, in fused rank order — the model was
# seeded with exactly these five docs' summaries.
chip = page.locator(".msg.brain .source-chip")
expect(chip).to_have_count(1, timeout=30_000)
expect(chip.first).to_contain_text("ssh_aliases.txt")
expect(chip).to_have_count(5, timeout=30_000)
expect(chip.nth(0)).to_contain_text("ssh_aliases.txt")
expect(chip.nth(1)).to_contain_text("gitlab-compose.yaml")
expect(chip.nth(2)).to_contain_text("uptime_probe.py")
expect(chip.nth(3)).to_contain_text("kubernetes.md")
expect(chip.nth(4)).to_contain_text("backups.md")
# The below-bar docs ride the related row: a labeled, de-emphasized
# list — one .related-doc link per doc (rank order, capped at
# related_max_docs = 2), never a .source-chip.
# The rank-6+ remainder rides the related row: a labeled,
# de-emphasized list — one .related-doc link per doc (rank order,
# capped at related_max_docs = 2), never a .source-chip.
row = page.locator(".msg.brain .related-docs")
expect(row).to_have_count(1)
expect(row.first).to_have_attribute("aria-label", "Nearby docs, in case")
expect(row.first.locator(".related-docs-label")).to_have_text("Nearby docs, in case:")
links = page.locator(".msg.brain .related-docs .related-doc")
expect(links).to_have_count(2, timeout=30_000)
expect(links.nth(0)).to_contain_text("gitlab-compose.yaml")
expect(links.nth(1)).to_contain_text("uptime_probe.py")
expect(links.nth(0)).to_contain_text("compose.container")
expect(links.nth(1)).to_contain_text("static-dns.json")
expect(page.locator(".msg.brain .related-docs .source-chip")).to_have_count(0)
# The related links keep the chip's /document.html href + identity.
expect(links.first).to_have_attribute(
"title", "docs/homelab/container_gitlab/gitlab-compose.yaml"
"title", "docs/homelab/quadlet/compose.container"
)
# Durable record: not deflected; the FULL retrieval (cited + related)
# is logged — query_log records retrieval, not citations (LOCKED A3).
# Durable record: not deflected; the FULL retrieval (suggested +
# related + read, deduped) is logged — query_log records retrieval,
# not citations (LOCKED A3).
with SessionLocal() as db:
row_log = db.scalars(select(QueryLog)).one()
assert row_log.question == SINGLE_SOURCE_QUESTION
assert row_log.deflected is False
for path in (
"homelab/ssh/ssh_aliases.txt",
"homelab/container_gitlab/gitlab-compose.yaml",
"homelab/scripts/uptime_probe.py",
):
assert path in row_log.sources
# Suggested tier + related remainder, in the logged order.
assert row_log.sources == (
"docs/homelab/ssh/ssh_aliases.txt, "
"docs/homelab/container_gitlab/gitlab-compose.yaml, "
"docs/homelab/scripts/uptime_probe.py, "
"docs/homelab/kubernetes.md, "
"docs/homelab/backups.md, "
"docs/homelab/quadlet/compose.container, "
"docs/homelab/networking/static-dns.json"
), row_log.sources
def test_deflected_question_shows_zero_citation_chips(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""The acceptance pin: a deflected turn (known-out-of-KB) → ZERO
``.source-chip`` elements under the bubble; the weak hits live in the
related row only — its links are ``.related-doc``, never
``.source-chip`` (L2c: "at minimum: never render them as answer
citations"). The "Maybe try" chips and the durable record are
unchanged."""
"""The acceptance pin (phase 118, A4): a deflected turn
(known-out-of-KB) → ZERO ``.source-chip`` elements under the bubble
(done.sources = [] — a deflected answer cites nothing; the weak hits
are suggested for the durable record but are scored docs, not
citations); the rank-6+ weak hits live in the related row only —
its links are ``.related-doc``, never ``.source-chip`` (L2c: "at
minimum: never render them as answer citations"). The "Maybe try"
chips are unchanged."""
_reset_db(mock_llm)
page.set_default_timeout(30_000)
login(page, app_url, next="/") # phase 79: chat is require_user-gated
@@ -181,21 +204,35 @@ def test_deflected_question_shows_zero_citation_chips(
row = page.locator(".msg.brain .related-docs")
expect(row).to_have_count(1, timeout=30_000)
links = page.locator(".msg.brain .related-docs .related-doc")
expect(links).to_have_count(2) # the weak hits, rank order, capped at 2
expect(links.nth(0)).to_contain_text("new-service.md")
expect(links.nth(1)).to_contain_text("ssh_aliases.txt")
# The rank-6+ remainder of the weak hits, rank order, capped at 2 —
# the top five weak hits are suggested (no floor) for the durable
# record, but a deflected turn cites nothing, so they never render
# as chips.
expect(links).to_have_count(2)
expect(links.nth(0)).to_contain_text("backups.md")
expect(links.nth(1)).to_contain_text("kubernetes.md")
# ZERO citation chips under the bubble — the weak hits are scored
# docs, not citations (the phase-112/113 contract on the wire).
expect(page.locator(".msg.brain .source-chip")).to_have_count(0)
expect(page.locator(".msg.brain .related-docs .source-chip")).to_have_count(0)
# Durable record: deflected, weak top score — the retrieval stays
# logged for threshold tuning (LOCKED A3: observability unchanged).
# Durable record: deflected, weak top score — the FULL retrieval
# (the weak hits ARE suggested — no floor — plus the rank-6+ related
# remainder) stays logged for threshold tuning (LOCKED A3:
# observability unchanged).
with SessionLocal() as db:
row_log = db.scalars(select(QueryLog)).one()
assert row_log.question == OFF_TOPIC
assert row_log.deflected is True
assert 0.0 < row_log.top_score < get_settings().relevance_threshold
assert row_log.fts_hits == 0
assert row_log.sources # the weak-hit paths, for threshold tuning
assert row_log.sources == (
"docs/deployments/new-service.md, "
"docs/homelab/ssh/ssh_aliases.txt, "
"docs/homelab/quadlet/lan.network, "
"docs/homelab/scripts/uptime_probe.py, "
"docs/homelab/quadlet/compose.container, "
"docs/homelab/backups.md, "
"docs/homelab/kubernetes.md"
), row_log.sources
+45 -24
View File
@@ -16,9 +16,10 @@ reused here as closely as possible):
``documents.summary``. The sentinel ``RESE-SUMMARY-SENTINEL-7f3a`` sits
on the document's LAST line — **outside** the 24-token digest — so it
is a marker for "the original, not the summary".
* ``notes/qwen-llamacpp-notes.md`` — a markdown control doc, never
summarized (phase 30 scope): the viewer must render it exactly as
before, with no Summary panel.
* ``notes/qwen-llamacpp-notes.md`` — a markdown doc that phase 118 (A2)
summarizes TOO (the phase-30 non-markdown-only scope is retired): the
viewer must show its Summary panel exactly like the yaml's, above the
rendered markdown.
This phase only surfaces the stored field: the content endpoint returns
``summary`` (task 01) and the shared ``renderDocument`` core draws the
@@ -27,8 +28,8 @@ labeled ``.doc-summary`` panel above the content on BOTH surfaces (task
exactly that contract. Phase 79 supersedes the phase-16 soft rule: the
content endpoint is ``require_user``-gated, so the viewer surfaces and
the API shape pin run under a signed-in session (the shape itself —
``summary`` for the yaml, ``null`` for the markdown control — is
unchanged).
``summary`` for BOTH fixture docs, phase 118 A2 — is what the pins
carry).
"""
from __future__ import annotations
@@ -98,13 +99,14 @@ def _run_in_thread(coro: Any) -> Any:
def _reset_db_and_import(mock_llm: int) -> ImportSummary:
"""Truncate the KB (and query log + steering) and re-import the
summary_kb fixtures — yaml summarized, md control not."""
summary_kb fixtures — both docs summarized (phase 118, A2: the
markdown doc too)."""
with SessionLocal() as db:
db.execute(text("TRUNCATE chunks, documents, query_log, steering_notes"))
db.commit()
summary = _run_in_thread(_import_fixtures(mock_llm))
assert summary.added == 2 # yaml + md control
assert summary.summaries == 1 and summary.summary_errors == 0
assert summary.added == 2 # yaml + md
assert summary.summaries == 2 and summary.summary_errors == 0 # A2: md too
assert summary.errors == 0
return summary
@@ -279,19 +281,30 @@ def test_modal_shows_panel_and_full_page_agrees(
# ---------------------------------------------------------------------------
def test_markdown_doc_has_no_summary_panel(
def test_markdown_doc_has_a_summary_panel(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""Control doc (markdown — never summarized): NO .doc-summary
element on either surface, and the content renders exactly as
before (first child of the content container is the doc body)."""
"""Phase 118 (A2): the markdown doc is summarized TOO — the SAME
labeled ``.doc-summary`` panel (the deterministic mock digest +
pointer line) renders above the rendered markdown on BOTH surfaces
(the phase-30 "markdown never summarized" control is retired with
the non-markdown-only scope)."""
_reset_db_and_import(mock_llm)
page.set_default_timeout(30_000)
login(page, app_url, next="/") # phase 79: the viewer content is gated
# Full page: no panel, markdown column untouched.
digest_line, pointer_line = _summary_lines(SOURCE, MD_PATH)
# Full page: the labeled panel above the markdown column (the same
# contract as the yaml — one shared renderer).
page.goto(f"{app_url}/document.html?source={SOURCE}&path={MD_URL_PATH}")
expect(page.locator(".doc-summary")).to_have_count(0)
panel = page.locator(".doc-summary")
expect(panel).to_have_count(1)
expect(panel).to_be_visible()
expect(panel).to_have_attribute("aria-label", "Summary")
expect(panel.locator(".doc-summary-title")).to_have_text("Summary")
expect(panel).to_contain_text(digest_line)
expect(panel).to_contain_text(pointer_line)
expect(page.locator("#doc-content .doc-md")).to_have_count(1)
expect(page.locator("#doc-content")).to_contain_text(
"came out of a week of"
@@ -300,10 +313,10 @@ def test_markdown_doc_has_no_summary_panel(
"() => [...document.querySelector('#doc-content').children]"
".map((el) => el.className)"
)
assert order == ["doc-md"], f"markdown doc gained children: {order}"
assert order == ["doc-summary", "doc-md"], f"panel not first: {order}"
# Modal: same story — no panel, .doc-md is the sole content child.
# (the session is already signed in — the form login above)
# Modal: same story — the panel above .doc-md (the session is
# already signed in — the form login above).
page.goto(f"{app_url}/sources.html")
# Phase 97: the re-mount lands on the tree's top level — drill to
# the notes level where the row lives (the drill is the only
@@ -314,7 +327,11 @@ def test_markdown_doc_has_no_summary_panel(
expect(row).to_have_count(1)
row.locator("td:nth-child(2) a.doc-link").click()
expect(page.locator(".doc-modal")).to_be_visible()
expect(page.locator("#doc-modal .doc-summary")).to_have_count(0)
modal_panel = page.locator("#doc-modal .doc-summary")
expect(modal_panel).to_have_count(1)
expect(modal_panel).to_be_visible()
expect(modal_panel).to_contain_text(digest_line)
expect(modal_panel).to_contain_text(pointer_line)
expect(page.locator("#doc-modal .doc-md")).to_have_count(1)
expect(page.locator("#doc-modal-content")).to_contain_text(
"came out of a week of"
@@ -323,7 +340,7 @@ def test_markdown_doc_has_no_summary_panel(
"() => [...document.querySelector('#doc-modal-content').children]"
".map((el) => el.className)"
)
assert order == ["doc-md"], f"markdown modal gained children: {order}"
assert order == ["doc-summary", "doc-md"], f"markdown modal order: {order}"
# ---------------------------------------------------------------------------
@@ -336,10 +353,9 @@ def test_content_api_summary_shape(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""``GET /api/documents/content`` carries ``summary`` — the string
for the summarized yaml, ``null`` for the markdown control. Phase 79
superseded the phase-16 soft rule: the endpoint is require_user, so
the pin runs under the form-login session (the shape itself is
unchanged)."""
for BOTH fixture docs (phase 118, A2: the markdown doc too). Phase
79 superseded the phase-16 soft rule: the endpoint is require_user,
so the pin runs under the form-login session."""
_reset_db_and_import(mock_llm)
login(page, app_url, next="/")
@@ -364,5 +380,10 @@ def test_content_api_summary_shape(
assert resp_md.status == 200
md_body = resp_md.json()
assert md_body["format"] == "md"
assert md_body["summary"] is None, "markdown docs never carry a summary"
# Phase 118 (A2): the markdown doc carries its summary too — the
# same byte-stable digest + pointer line.
md_digest_line, _ = _summary_lines(SOURCE, MD_PATH)
assert md_body["summary"] == (
f"{md_digest_line}\nSource: {SOURCE}/{MD_PATH}"
)
assert "came out of a week of" in md_body["content"]
+671
View File
@@ -0,0 +1,671 @@
"""Phase 118 task 06 E2E (Playwright, mock-only): the summary-seed
context — grounded turns seed SUMMARIES, full text arrives only via
``read`` (the owner directive, TODO.md L3; the retired phase-24
full-text seeding is gone — locked A1/A6).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_summary_seed_context.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the story's gate
is the deterministic contract, not the real model's behavior: the mock
LLM's tail-echo trigger (``END_OF_NOTES_TRIGGER``) quotes the last 160
chars of the seeded ``<documents>`` block, and the scripted
summary-read flow (``SUMMARY_SEED_READ_TRIGGER``) echoes the whole read
tool result — both are the house scripted-turn lenses on the LLM's
context (the mock is the only E2E lens), so what reached the prompt is
assertable byte-exactly.
KB fixture — one host temp dir (``tmp_path_factory``; the app runs on
the same host) registered as a local-directory source (the
``test_ls_tree_drilldown.py`` / ``test_read_truncation_cap.py``
registration + real-Sync pattern — registration through the
authenticated API, the real in-process ``POST /api/sync`` pipeline; no
git anywhere), with SEVEN markdown documents (every file markdown —
the locked A2 end-to-end proof: the mock ``SUMMARY_MODE`` digest must
land on every doc, markdown included) whose bodies are
token-controlled so the hybrid gate picks the intended suggested set
deterministically (``synced_kb`` pins it with the app's REAL
retrieval — a fixture-text regression that breaks the design fails at
setup with a clear message, not at the wire assertions):
* each doc = a neutral 24-token prelude (no question tokens — the
mock's first-24-token digest therefore shares NOTHING with either
question, so a doc's embedded summary chunk carries no strength
tokens and never outranks that doc's own content chunks) +
``seed vault rotation notes`` repeated *i* times (the strength
gradient) + six unique filler tokens + a unique tail SENTINEL on
the document's LAST line (outside the digest — the full-text
marker);
* the measured rank order (probe, stable across re-imports) is
``doc-a > doc-d > doc-b > doc-c > doc-e > doc-f > doc-g`` for the
tail question (×1 strength gradient — md5-collision-reordered) and a
clean ``doc-a > doc-b > doc-c > doc-d > doc-e > doc-f > doc-g`` for
the read question (×3 gradient — the collision noise cannot cross
the wider gaps); the suggested tier (LOCKED A3: top-5, NO floor) is
the first five of each, the related tier (rank 6+, ``related_max_docs``
= 2) is ``doc-f``/``doc-g`` in both.
Test → contract mapping (the task's cases a–e; the story is the owner
TODO item — one Playwright file per story, A16):
1. ``test_summaries_seed_the_prompt_not_the_full_text`` — cases (a) +
(b) + (d): the tail-echo question's answer quotes the last 160
chars of the seeded ``<documents>`` block, which end in the LAST
suggested document's SUMMARY — the mock's byte-stable
``SUMMARY_MODE`` digest tail + pointer line (case a: summaries
reached the prompt; case d: the markdown doc's block carries the
digest, not a content preview — the phase-30 digest shape, and the
``synced_kb`` pin proves every doc's stored summary IS the digest).
The inverse of the retired phase-24 pin: NO document's tail
sentinel is in the echoed context — the full content of no
suggested doc ever reached the model (case b). The grounded turn
cites the 5 suggested docs (case e's chip surface, no read yet) and
renders the de-emphasized related row (rank 6+); the durable
record carries suggested + related.
2. ``test_read_suggested_doc_adds_full_text_and_cites`` — cases (c) +
(e): the scripted flow ``read``s the rank-1 suggested doc — the
read SUCCEEDS (phase 118: the seeds are summaries, not full text —
the retired top-2 seed-read refusal is gone) and the mock's
verbatim echo of the read result lands the tail sentinel in the
answer (case c: the full text now arrives through the ``read``
tool, not the seed); the citation chips = suggested + read,
deduped — the read doc is among the suggested, so the chip row is
exactly the 5 suggested docs, and the related row renders rank 6+
(case e); the durable record carries suggested + related + read.
"""
from __future__ import annotations
import json
import os
import re
import subprocess
import sys
import time
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import httpx
import pytest
from playwright.sync_api import Locator, Page, expect
from sqlalchemy import select, text
from app.config import Settings as _Settings
from app.db import SessionLocal
from app.models import Document, QueryLog
from app.rag.retriever import TRUNCATION_MARKER, retrieve, select_related, select_suggested
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
from e2e.mock_llm import TOKEN_RE, embed_text
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module app binds its own port instead
# (a same-port second uvicorn dies on bind and would drive the wrong
# server). Env-overridable.
APP_PORT = int(os.environ.get("E2E_APP_PORT_SEEDCTX", "8138"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
SOURCE = "seedkb" # the local directory's basename = the source name
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
# --------------------------------------------------------------------------
# Fixture documents (deterministic, token-controlled — see the module
# docstring for the design and the measured rank order)
# --------------------------------------------------------------------------
#: 22 neutral tokens — together with the 2-token title they ARE the
#: mock's first-24-token digest, so no summary chunk shares a token
#: with either question (the summary chunks rank below every content
#: chunk; the digest is the byte-stable assertion surface).
PRELUDE = (
"fixture preamble block one two three four five six seven eight nine ten "
"eleven twelve thirteen fourteen fifteen sixteen seventeen eighteen nineteen"
)
assert len(TOKEN_RE.findall(PRELUDE.lower())) == 22
#: The strength gradient (each token a question token — ``seed``,
#: ``vault``, ``rotation``, ``notes``) — repeated *i* times in doc *i*.
PHRASE = "seed vault rotation notes"
#: (path, title, phrase repeats, filler token, tail sentinel) — the
#: MEASURED rank order (probe, stable across re-imports) is
#: doc-a > doc-d > doc-b > doc-c > doc-e > doc-f > doc-g: the suggested
#: tier is the first five, the related tier the last two.
DOCS: list[tuple[str, str, int, str, str]] = [
("doc-a.md", "Zeta Alfa", 7, "fillera", "SEEDA-TAIL-7f3a"),
("doc-b.md", "Zeta Bravo", 6, "fillerb", "SEEDB-TAIL-8c4d"),
("doc-c.md", "Zeta Charlie", 5, "fillerc", "SEEDC-TAIL-9d5e"),
("doc-d.md", "Zeta Delta", 4, "fillerd", "SEEDD-TAIL-0e6f"),
("doc-e.md", "Zeta Echo", 3, "fillere", "SEEDE-TAIL-1f7a"),
("doc-f.md", "Zeta Foxtrot", 2, "fillerf", "SEEDF-TAIL-2a8b"),
("doc-g.md", "Zeta Golf", 1, "fillerg", "SEEDG-TAIL-3b9c"),
]
#: The suggested tier per scripted question (the MEASURED rank order —
#: probe-verified, stable across re-imports): the tail question's ×1
#: strength gradient is md5-collision-reordered (d before b), the read
#: question's ×3 gradient is clean. Related is the rank-6+ remainder in
#: both (the gradient never puts a strength doc below the filler docs).
SUGGESTED_TAIL = ["doc-a.md", "doc-d.md", "doc-b.md", "doc-c.md", "doc-e.md"]
SUGGESTED_READ = ["doc-a.md", "doc-b.md", "doc-c.md", "doc-d.md", "doc-e.md"]
RELATED_PATHS = ["doc-f.md", "doc-g.md"]
LAST_SUGGESTED = "doc-e.md" # the tail-echo target (case a/d)
READ_TARGET = "doc-a.md" # rank 1 — a suggested doc (case c)
SENTINELS = [sentinel for _p, _t, _i, _f, sentinel in DOCS]
READ_SENTINEL = next(s for p, _t, _i, _f, s in DOCS if p == READ_TARGET)
def _doc_content(title: str, i: int, filler: str, sentinel: str) -> str:
return (
f"# {title}\n"
f"\n{PRELUDE}\n"
f"\n{' '.join([PHRASE] * i)}\n"
f"\n{' '.join([filler] * 6)}\n"
f"\n{sentinel}\n"
)
def _expected_summary(content: str, path: str) -> str:
"""The mock lite model's byte-stable digest + the code pointer line
(mirrors ``mock_llm.compose_answer``'s ``SUMMARY_MODE`` branch —
first 24 tokens of the document content — plus the summarizer's
deterministic ``Source:`` line; no model output is ever trusted)."""
digest = " ".join(TOKEN_RE.findall(content.lower())[:24])
return f"This document covers {digest}.\nSource: {SOURCE}/{path}"
#: The turn's questions (the mock's trigger phrases — see the module
#: docstring). The tail question carries ``END_OF_NOTES_TRIGGER``; its
#: strength tokens ground the turn (best cosine ≈ 0.55 ≥ the E2E 0.30
#: threshold). The read question carries ``SUMMARY_SEED_READ_TRIGGER``
#: with its own scripted call after the colon (the house scripted-turn
#: convention — ``_READ_CAP_CALL_RE`` / ``_DRILL_CALL_RE`` shape),
#: ``seed vault rotation notes`` × 3 for the same grounding.
TAIL_QUESTION = "Show the end of your notes about the seed vault rotation, please."
READ_QUESTION = (
f"Read the suggested document: read {SOURCE}/{READ_TARGET} — "
f"{PHRASE} {PHRASE} {PHRASE}"
)
assert "show the end of your notes" in TAIL_QUESTION.lower()
assert "read the suggested document" in READ_QUESTION.lower()
assert re.fullmatch(r"[a-z0-9_./-]+", f"{SOURCE}/{READ_TARGET}")
# --------------------------------------------------------------------------
# Fixtures
# --------------------------------------------------------------------------
@pytest.fixture(scope="module")
def seed_dirs(tmp_path_factory: pytest.TempPathFactory) -> Path:
"""The story's local-directory source: one host temp dir (the app
server runs on the same host, so the path is visible to it) holding
the seven token-controlled markdown documents. The directory's
basename is the source name (``kind=local``, phase 38)."""
root = tmp_path_factory.mktemp("bor_seed_ctx") / SOURCE
root.mkdir()
for path, title, i, filler, sentinel in DOCS:
(root / path).write_text(
_doc_content(title, i, filler, sentinel), encoding="utf-8"
)
assert (root / READ_TARGET).is_file()
return root
@pytest.fixture(scope="module")
def app_server(mock_llm: int, seed_dirs: Path) -> Iterator[str]:
"""The real app under test — per-module app (the conftest pattern,
cf. ``test_read_truncation_cap.py``): NO ``BOR_GIT_SOURCES`` (the
env fallback is git-only — the source here is a DB-registered local
directory), the mock LLM, the mock-calibrated threshold, and the
leak-guarded code defaults (the suggested/related tier settings
ride their code defaults — 5 / 2 — exactly like the production
``.env``-free defaults). The session app is never started in this
isolated run, so no port clash."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated gate (conftest pattern): the strength gradient
# keeps every scripted turn grounded (best cosine ≈ 0.55 / ≈ 0.80).
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
# Phase 67: instant retry waits + the code-default budget (the
# conftest leak-guard pattern).
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(_Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
# reads it from cwd) — override it with an EMPTY value (the env var
# beats the .env file): the registry must hold EXACTLY the local
# directory this suite registers (a leftover env git list would
# pollute the KB the scripted turns run against).
env["BOR_GIT_SOURCES"] = ""
# Leak guards (conftest pattern): an operator's local (gitignored)
# .env cannot leak corpus-specific settings into the app under test.
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
_Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = _Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = _Settings.model_fields["footer_text"].default
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern): the E2E suites
share one Postgres, so a leftover git_sources row or document would
pollute the retrieval the scripted turns run against (the strength
gradient's margins are pinned against EXACTLY these seven
documents)."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources, folder_summaries"
)
)
db.commit()
def _wait_sync_done_http(client: httpx.Client, timeout_s: float = 180.0) -> dict[str, Any]:
"""Poll the (cookie-authenticated) status endpoint until the run
reaches a terminal state (the test_ls_tree_drilldown pattern, over
plain httpx — this fixture has no browser page yet)."""
deadline = time.monotonic() + timeout_s
body: dict[str, Any] = {}
while time.monotonic() < deadline:
r = client.get("/api/sync/status")
assert r.status_code == 200, r.text
body = r.json()
if body["state"] in ("success", "failed"):
return body
time.sleep(0.5)
raise AssertionError(f"sync did not reach a terminal state: {body}")
def _assert_tiers(question: str, suggested_paths: list[str]) -> None:
"""Pin the strength-gradient design with the app's REAL hybrid
retrieval over the mock's embeddings (deterministic): the suggested
tier is exactly the five strength docs in the MEASURED rank order
for THIS question (LOCKED A3 — top-5, NO floor) and the related
tier is the rank-6+ remainder (``related_max_docs`` = 2). A
fixture-text regression that breaks the design fails here, at
setup, with a clear message."""
with SessionLocal() as db:
chunks = retrieve(db, question, embed_text(question))
suggested = [f"{d.source}/{d.path}" for d in select_suggested(chunks)]
related = [
f"{d.source}/{d.path}"
for d in select_related(
chunks,
{d.id for d in select_suggested(chunks)},
_Settings.model_fields["related_max_docs"].default,
)
]
expected = [f"{SOURCE}/{p}" for p in suggested_paths]
assert suggested == expected, (
f"suggested tier drifted for {question!r}: {suggested} "
f"(expected {expected})"
)
assert related == [f"{SOURCE}/{p}" for p in RELATED_PATHS], (
f"related tier drifted for {question!r}: {related}"
)
@pytest.fixture(scope="module")
def synced_kb(app_server: str, seed_dirs: Path) -> None:
"""The story's precondition: the one-source KB synced under the
deterministic mock. Registers the temp directory through the
authenticated API (the ``test_local_directory_sources.py``
pattern), runs the REAL in-process sync (``POST /api/sync`` — walk
→ chunk → embed → summaries → overview → folder summaries →
version bump), pins the stored content byte-identical to the
fixture strings, pins LOCKED A2 end-to-end (every doc — markdown
included — stores the mock's byte-stable digest + exactly one
embedded ``is_summary`` chunk), and pins the tier design for both
scripted questions."""
_truncate_all()
with httpx.Client(base_url=app_server, timeout=30.0) as client:
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, r.text
r = client.post(
"/api/git-sources", json={"kind": "local", "path": str(seed_dirs)}
)
assert r.status_code == 201, r.text
r = client.post("/api/sync")
assert r.status_code == 202, r.text
body = _wait_sync_done_http(client)
assert body["state"] == "success", body
detail = body["detail"]
assert detail["added"] == len(DOCS), detail
assert detail["pruned"] == 0, detail
# The import stored the fixture strings BYTE-IDENTICALLY and, for
# EVERY doc (markdown included — locked A2), the mock's byte-stable
# digest: the deterministic assertion surface of the whole suite.
with SessionLocal() as db:
for path, title, i, filler, sentinel in DOCS:
stored = db.scalar(
select(Document).where(
Document.source == SOURCE, Document.path == path
)
)
assert stored is not None, f"fixture doc {path} was not imported"
expected = _doc_content(title, i, filler, sentinel)
assert stored.content == expected, f"stored content drifted for {path}"
assert stored.summary == _expected_summary(expected, path), (
f"summary is not the mock digest for {path}: {stored.summary!r}"
)
schunks = [c for c in stored.chunks if c.is_summary]
assert len(schunks) == 1 and schunks[0].position == -1, (
f"expected exactly one is_summary chunk for {path}"
)
assert schunks[0].embedding is not None, (
f"the is_summary chunk of {path} is not embedded"
)
_assert_tiers(TAIL_QUESTION, SUGGESTED_TAIL)
_assert_tiers(READ_QUESTION, SUGGESTED_READ)
@pytest.fixture(autouse=True)
def _clean(db_ready: None) -> Iterator[None]:
"""Per-test query_log isolation (the KB itself is module-scoped —
the scripted turns never change it, so the registry and the KB
persist across the tests of this module)."""
with SessionLocal() as db:
db.execute(text("TRUNCATE query_log"))
db.commit()
yield
with SessionLocal() as db:
db.execute(text("TRUNCATE query_log"))
db.commit()
# --------------------------------------------------------------------------
# Page helpers (the test_ls_tree_drilldown / test_read_truncation_cap
# house patterns)
# --------------------------------------------------------------------------
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
#: (a response clone read in the background) — wire-level assertions
#: for the ``tool`` / ``done`` frames, independent of the UI rendering.
SSE_HOOK = """
() => {
if (window.__sseInstalled) return;
window.__sseInstalled = true;
window.__sseFrames = [];
const origFetch = window.fetch;
window.fetch = async function (...args) {
const res = await origFetch.apply(this, args);
try {
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
if (url.includes('/api/chat')) {
res.clone().text().then((bodyText) => {
for (const block of bodyText.split('\\n\\n')) {
const line = block.trim();
if (line.startsWith('data: ')) {
window.__sseFrames.push(line.slice(6));
}
}
});
}
} catch (e) { /* non-clonable responses: ignored */ }
return res;
};
}
"""
def _install_page_hooks(page: Page) -> None:
page.evaluate(SSE_HOOK)
def _frames(page: Page) -> list[dict]:
"""The SSE frames captured since the last submit (``_submit``
clears the buffer), once the hook's background read settles."""
deadline = time.monotonic() + 30.0
while True:
raw = page.evaluate("() => window.__sseFrames || []")
parsed = [json.loads(line) for line in raw if line]
if any(f.get("type") == "done" for f in parsed):
return parsed
if time.monotonic() > deadline:
raise AssertionError(
f"SSE hook captured no `done` frame (frames so far: "
f"{len(parsed)}) — hook install failed?"
)
time.sleep(0.05)
def _tool_frames(frames: list[dict]) -> list[dict]:
return [f for f in frames if f.get("type") == "tool"]
def _submit(page: Page, question: str) -> None:
page.evaluate("window.__sseFrames = []")
page.fill("#message-input", question)
page.click("#send-btn")
# The user bubble lands synchronously with the submit handler.
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
def _wait_settled(page: Page) -> None:
"""The turn is complete: answer text in the bubble, button recovered
(the phase-48 settle wait, the test_agent_document_tools helper)."""
expect(page.locator(".msg.brain .bubble").last).not_to_have_text("", timeout=30_000)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator("#send-label")).to_have_text("Send", timeout=30_000)
def _last_brain(page: Page) -> Locator:
return page.locator(".msg.brain").last
def _last_query_log() -> QueryLog:
with SessionLocal() as db:
rows = db.scalars(select(QueryLog)).all()
assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}"
return rows[0]
def _assert_suggested_chips_and_related_row(page: Page, suggested_paths: list[str]) -> None:
"""The citation surface of a grounded turn (LOCKED A4): the chip row
is the suggested set (+ agent reads, deduped — asserted per test)
and the de-emphasized ``related-docs`` row carries the rank-6+
remainder (phase-113 UI, never a citation chip)."""
chips = page.locator(".msg.brain .source-chip")
expect(chips).to_have_count(len(suggested_paths))
for path in suggested_paths:
expect(
chips.filter(has_text=path), message=f"chip for {path}"
).to_have_count(1)
row = page.locator(".msg.brain .related-docs")
expect(row).to_have_count(1)
expect(row.first).to_have_attribute("aria-label", "Nearby docs, in case")
links = page.locator(".msg.brain .related-docs .related-doc")
expect(links).to_have_count(len(RELATED_PATHS))
for i, path in enumerate(RELATED_PATHS):
expect(links.nth(i)).to_contain_text(f"{SOURCE}/{path}")
# The related links are never citation chips.
expect(page.locator(".msg.brain .related-docs .source-chip")).to_have_count(0)
# --------------------------------------------------------------------------
# 1. Cases (a) + (b) + (d): summaries seed the prompt — no full text
# --------------------------------------------------------------------------
def test_summaries_seed_the_prompt_not_the_full_text(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
"""The tail-echo question quotes the last 160 chars of the seeded
``<documents>`` block: they end in the LAST suggested doc's SUMMARY
(the mock's byte-stable digest tail + pointer — cases a + d: the
markdown doc's block carries the digest, not a content preview),
and NO document's tail sentinel is in the echoed context (case b:
the full text of no suggested doc reached the model — the inverse
of the retired phase-24 pin)."""
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, TAIL_QUESTION)
_wait_settled(page)
# No tools on this turn — the answer is the mock's direct tail echo
# of the seeded context (the summary-seed lens).
frames = _frames(page)
assert _tool_frames(frames) == [], _tool_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The grounded turn cites the 5 suggested docs (LOCKED A4 — no read
# yet: the chip row IS the suggested set, in rank order).
assert [
(s["source"], s["path"]) for s in done["sources"]
] == [(SOURCE, p) for p in SUGGESTED_TAIL], done["sources"]
bubble = _last_brain(page).locator(".bubble")
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER, timeout=30_000)
# Case (a) + (d): the echoed tail ends in the LAST suggested doc's
# summary — the markdown doc-e's byte-stable digest tail + pointer
# line (the digest is the mock's SUMMARY_MODE shape, pinned in
# ``synced_kb`` — a content preview would carry the prelude/phrase
# text instead, and the pointer line only ever exists on a stored
# summary). The bubble renders the answer as markdown, which
# collapses the summary's newline — so pin each LINE separately
# (the digest line's tail sits inside the echoed 160 chars; the
# pointer line is single-line too).
last_doc = next(d for d in DOCS if d[0] == LAST_SUGGESTED)
last_summary = _expected_summary(_doc_content(*last_doc[1:]), LAST_SUGGESTED)
digest_line = last_summary.split("\n", 1)[0]
expect(bubble).to_contain_text("Source: seedkb/doc-e.md")
expect(bubble).to_contain_text(digest_line[-100:])
# The digest is NOT the raw content: the echoed tail cannot carry
# any document's tail sentinel — the inverse of the retired
# phase-24 full-text pin (case b).
for sentinel in SENTINELS:
expect(bubble).not_to_contain_text(sentinel)
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
# Case (e)'s chip surface (no read yet): chips = the 5 suggested
# docs; the related row renders rank 6+ (the de-emphasized row).
_assert_suggested_chips_and_related_row(page, SUGGESTED_TAIL)
# Durable record: grounded; suggested + related (LOCKED A3 — the log
# records retrieval, not citations).
row = _last_query_log()
assert row.question == TAIL_QUESTION
assert row.deflected is False
assert row.sources == ", ".join(f"{SOURCE}/{p}" for p in SUGGESTED_TAIL + RELATED_PATHS)
# --------------------------------------------------------------------------
# 2. Cases (c) + (e): ``read`` adds the full text; chips = suggested +
# read (deduped); related row = rank 6+
# --------------------------------------------------------------------------
def test_read_suggested_doc_adds_full_text_and_cites(
page: Page, app_url: str, synced_kb: None, db_ready: None
) -> None:
"""The scripted flow ``read``s the rank-1 suggested doc: the read
SUCCEEDS (phase 118 — the seeds are summaries, not full text; the
retired top-2 seed-read refusal is gone) and the mock's verbatim
echo of the read result lands the document's tail sentinel in the
answer (case c: the full text reached the model through the
``read`` tool, not the seed). The citation chips = suggested +
read, deduped — the read doc is among the suggested, so the chip
row is exactly the 5 suggested docs; the related row renders rank
6+ (case e)."""
page.set_default_timeout(30_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, READ_QUESTION)
_wait_settled(page)
# Exactly ONE executed tool call: the scripted read of the
# rank-1 suggested doc.
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "read", "argument": f"{SOURCE}/{READ_TARGET}"}
], _tool_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# LOCKED A4: done.sources = suggested + read, deduped — the read doc
# is already among the suggested, so the citation surface is
# exactly the 5 suggested docs (in rank order).
assert [
(s["source"], s["path"]) for s in done["sources"]
] == [(SOURCE, p) for p in SUGGESTED_READ], done["sources"]
# done.related = the rank-6+ remainder (deduped against the cited).
assert [
(s["source"], s["path"]) for s in done["related"]
] == [(SOURCE, p) for p in RELATED_PATHS], done["related"]
# The live Reading line carries the combined identity.
line = _last_brain(page).locator(".tool-call")
expect(line).to_have_count(1)
expect(line).to_contain_text(f"Reading {SOURCE}/{READ_TARGET}")
# Case (c): the answer is the mock's verbatim echo of the READ
# RESULT — its header and the document's tail sentinel (the last
# line, outside the digest): the full text reached the model
# through the read, not the seed.
bubble = _last_brain(page).locator(".bubble")
expect(bubble).to_contain_text(f"Document {SOURCE}/{READ_TARGET}:", timeout=30_000)
expect(bubble).to_contain_text(READ_SENTINEL)
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
# The seed itself carried no full text: none of the OTHER docs'
# sentinels are in the answer either.
for path, _t, _i, _f, sentinel in DOCS:
if path != READ_TARGET:
expect(bubble).not_to_contain_text(sentinel)
# Case (e): the UI chip row = suggested + read (deduped — the read
# doc is among the suggested, so chips = the 5 suggested docs), and
# the related row renders rank 6+.
_assert_suggested_chips_and_related_row(page, SUGGESTED_READ)
# Durable record: grounded; suggested + related + read (deduped,
# LOCKED A3).
row = _last_query_log()
assert row.question == READ_QUESTION
assert row.deflected is False
assert row.sources == ", ".join(f"{SOURCE}/{p}" for p in SUGGESTED_READ + RELATED_PATHS)
+28 -17
View File
@@ -71,8 +71,11 @@ computed-style assertion):
gray ok-ink on gray ok-bg, text intact.
7. ``test_tool_call_lines_gray`` — screenshot 1: a mock-LLM turn that
executes the tools renders the "Listing documents" / "Reading
<source/path>" lines — gray accent-ink text, the gray accent-line
left border, and the gray brand-soft path chip, all text intact.
<source/path>" lines — gray accent-ink text, all text intact.
(Phase 117 deboxed the line and dechipped the path on the owner's
visual-glitch report — the accent rides the TEXT: the pins now
assert the old accent-line border and the brand-soft chip
background are GONE, not gray.)
8. ``test_reset_removes_tag_byte_identical`` — the no-op contract: the
gray tag is present in the live document pre-reset; Reset to
defaults removes it from the LIVE document, a fresh load serves NO
@@ -790,10 +793,12 @@ def test_local_badge_gray_labeled(
#: document (a chunk carrying the mock's own bag-of-words embedding) that
#: grounds the turn, and one CATALOG-ONLY document (indexed, no chunks) the
#: mock's single-read flow reads. The catalog-only document sorts FIRST
#: ("Checklist" < "ThemeNotes") — the mock reads the first catalog line, and
#: it must NOT be the in-context retrieval document: the agent's read tool
#: refuses documents already in the prompt (ALREADY_IN_CONTEXT), and a
#: refused single-read flow would re-loop ls/read to the round cap.
#: ("Checklist" < "ThemeNotes") — the mock reads the first catalog line.
#: (Phase 118, A6: the read target's seed status no longer matters — a
#: first read of ANY document succeeds; the ALREADY_IN_CONTEXT refusal
#: fires only for a document already READ in the same turn. The
#: catalog-only design stands because the flow reads the first catalog
#: line, which must be a real, readable document.)
READ_SOURCE = "Checklist"
READ_PATH = "read-me.md"
READ_SP = f"{READ_SOURCE}/{READ_PATH}"
@@ -929,11 +934,15 @@ def test_tool_call_lines_gray(page: Page, app_url: str, db_ready: None) -> None:
expect(page.locator("#send-label")).to_have_text("Send", timeout=60_000)
# Screenshot 1: the yellow "Listing documents" / "Reading" lines —
# now gray accent-ink text with the gray accent-line left border,
# the lines' text intact. Phase 94: the drill-down ls adds a THIRD
# line between them — the drill ls scoped to the first source of
# the top level (registry order: Checklist — the read target's
# source).
# now gray accent-ink TEXT, the lines' text intact. Phase 94: the
# drill-down ls adds a THIRD line between them — the drill ls scoped
# to the first source of the top level (registry order: Checklist —
# the read target's source). Phase 117 (owner visual-glitch report)
# deboxed the line and dechipped the path: the accent rides the
# text color — there is NO left border and NO chip background to
# be gray, so the pins assert their ABSENCE (a colored border or
# chip background returning would fail both the width/alpha
# assertion and the theme completeness contract).
lines = page.locator(".msg.brain .tool-call")
expect(lines).to_have_count(3)
expect(lines.nth(0)).to_contain_text("Listing documents")
@@ -942,14 +951,16 @@ def test_tool_call_lines_gray(page: Page, app_url: str, db_ready: None) -> None:
expect(lines.nth(2)).to_contain_text("Reading")
expect(lines.nth(2)).to_contain_text(READ_SP)
_assert_gray(lines.nth(0), "color", GRAY["accent_ink"], label="ls line text")
_assert_gray(
lines.nth(0), "borderLeftColor", GRAY["accent_line"], label="ls line border"
)
assert (
lines.nth(0).evaluate("el => getComputedStyle(el).borderLeftWidth") == "0px"
), "phase 117 debox: the tool line must carry no left border"
_assert_gray(lines.nth(2), "color", GRAY["accent_ink"], label="read line text")
# The path chip on the Reading line: gray brand-soft background +
# gray ink (the screenshot's code chip — still gray under the ramp).
# The path `code` on the Reading line: phase 117 dechipped it —
# transparent background, gray ink (still gray under the ramp).
code = lines.nth(2).locator("code")
_assert_gray(code, "backgroundColor", GRAY["brand_soft"], label="read chip bg")
assert (
code.evaluate("el => getComputedStyle(el).backgroundColor") == "rgba(0, 0, 0, 0)"
), "phase 117 dechip: the path must have no chip background"
_assert_gray(code, "color", GRAY["ink"], label="read chip text")
-316
View File
@@ -1,316 +0,0 @@
"""Phase 24 E2E (Playwright): a matched document reaches the LLM whole.
Story: ``.agents/user_stories/whole-document-context.md``
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov
The mock LLM's tail-echo trigger (``END_OF_NOTES_TRIGGER``, see
``tests/e2e/mock_llm.py``) makes the model quote the last 160 chars of
the document context. A sentinel placed on the *last line* of a document
therefore appears in the rendered answer **iff the entire document was in
the prompt** — which is what makes the no-truncation contract (A7 revised,
owner permission 2026-08-24: matched parent documents are never cut)
provable end-to-end.
The oversized documents are seeded directly via SQLAlchemy (a
``documents`` row + 2–3 ``chunks`` rows whose embeddings are the mock's
own deterministic bag-of-words vectors, so the question's live mock
embedding genuinely overlaps — no fixture files added:
``tests/fixtures/docs/`` stays at its 13 files (phase 47), other suites
pin ``summary.added == 13``).
"""
from __future__ import annotations
import asyncio
import hashlib
from collections.abc import Callable, Sequence
from datetime import UTC, datetime
from pathlib import Path
from threading import Thread
from typing import Any
from playwright.sync_api import Page, expect
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.config import Settings
from app.db import SessionLocal
from app.models import Chunk, Document, QueryLog
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from app.rag.retriever import TRUNCATION_MARKER
from e2e.auth_helpers import login
from tests.e2e.mock_llm import embed_text
REPO = Path(__file__).resolve().parents[2]
FIXTURES = REPO / "tests" / "fixtures" / "docs"
#: The pre-phase-24 ``BOR_MAX_CONTEXT_CHARS`` default — the budget this
#: suite proves is gone from the document path.
OLD_CONTEXT_CAP = 24_000
QUESTION = "Show the end of your notes about the gitlab install playbook, please."
SMALL_QUESTION = "How is my Kubernetes cluster set up?"
MOCK_ANSWER_MARKER = "Deterministic mock answer for E2E"
# --- Content builders (deterministic, token-controlled) -------------------
def _repeated(line: str, min_chars: int) -> str:
""""line" (newline-terminated) repeated until at least min_chars chars."""
unit = line + "\n"
return unit * max(1, -(-min_chars // len(unit)))
def _doc_slices(content: str, n: int) -> list[str]:
"""Even slices of *content* (the last slice keeps the final line)."""
step = len(content) // n
return [content[i * step : (i + 1) * step] for i in range(n - 1)] + [
content[(n - 1) * step :]
]
def _gitlab_30k_doc(sentinel: str) -> str:
"""A ~30 000-char document (past the old 24k cap): a body of repeated
"gitlab install playbook" lines — the same tokens the question carries,
so hybrid retrieval genuinely hits — whose LAST line is a unique
sentinel only a tail echo can surface."""
body = _repeated(
"gitlab install playbook: run the gitlab install playbook on the homelab host.",
OLD_CONTEXT_CAP + 6_000,
)
return body + sentinel + "\n"
def _pair_doc(strong_line: str, filler_line: str, sentinel: str) -> tuple[str, list[str]]:
"""A ~16 000-char document: a ~2 000-char first chunk carrying the
question's key tokens, a ~14 000-char low-overlap remainder, and a
unique sentinel as the last line. Returns (content, chunk_texts)."""
chunk0 = _repeated(strong_line, 2_000)
chunk1 = _repeated(filler_line, 14_000)
return chunk0 + chunk1 + sentinel + "\n", [chunk0, chunk1]
# --- DB seeding (TRUNCATE-then-seed, cf. test_chat_rag.py) -----------------
def _seed_doc(
db: Session,
source: str,
path: str,
title: str,
content: str,
chunk_texts: Sequence[str],
) -> None:
"""One ``documents`` row + one ``chunks`` row per chunk text.
Each chunk's embedding is the mock's own ``embed_text`` vector, so the
app's live mock embedding of the question genuinely overlaps.
"""
doc = Document(
source=source,
path=path,
full_path=f"/tmp/{path}",
title=title,
content=content,
content_hash=hashlib.sha256(content.encode("utf-8")).hexdigest(),
indexed_at=datetime.now(UTC),
)
db.add(doc)
db.flush()
db.add_all(
Chunk(document_id=doc.id, position=i, content=chunk, embedding=embed_text(chunk))
for i, chunk in enumerate(chunk_texts)
)
def _reset_db(seed: Callable[[Session], None] | None = None) -> None:
"""Truncate the KB (and query log), then optionally run *seed*."""
with SessionLocal() as db:
db.execute(text("TRUNCATE chunks, documents, query_log"))
db.commit()
if seed is not None:
seed(db)
db.commit()
# --- Importer + thread helpers (test_chat_rag.py pattern) ------------------
async def _import_fixtures(mock_port: int) -> ImportSummary:
kwargs: dict[str, Any] = {
"_env_file": None,
"llm_base_url": f"http://127.0.0.1:{mock_port}/v1",
}
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
return await import_sources([FIXTURES], LLMClient(settings))
def _run_in_thread(coro: Any) -> Any:
"""Run a coroutine on a worker thread.
Playwright's sync API keeps an asyncio loop running on the test thread,
so ``asyncio.run`` cannot be called directly from a test body.
"""
box: dict[str, Any] = {}
def runner() -> None:
try:
box["value"] = asyncio.run(coro)
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
box["error"] = e
t = Thread(target=runner)
t.start()
t.join()
if "error" in box:
raise box["error"]
return box["value"]
def _ask(page: Page, app_url: str, question: str) -> Any:
"""Submit *question* and wait for the streamed brain bubble."""
page.set_default_timeout(30_000)
login(page, app_url, next="/")
page.fill("#message-input", question)
page.click("#send-btn")
bubble = page.locator(".msg.brain .bubble")
bubble.first.wait_for(state="visible", timeout=30_000)
return bubble.first
def _last_query_log() -> QueryLog:
with SessionLocal() as db:
rows = db.scalars(select(QueryLog)).all()
assert len(rows) == 1, f"expected exactly one query_log row, got {len(rows)}"
return rows[0]
# --- Story tests -------------------------------------------------------------
def test_whole_document_over_old_cap_reaches_llm(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""A 30k document (past the old 24k cap) reaches the LLM whole: its
tail sentinel — the last 160 chars of the context — is echoed back."""
sentinel = "WHOLE-DOC-TAIL-GITLAB-30K"
content = _gitlab_30k_doc(sentinel)
assert len(content) > OLD_CONTEXT_CAP # this is the point of the test
def seed(db: Session) -> None:
_seed_doc(
db,
"Homelab",
"gitlab-30k.md",
"GitLab Install Playbook (30k)",
content,
_doc_slices(content, 3),
)
_reset_db(seed)
bubble = _ask(page, app_url, QUESTION)
# The tail sentinel exists only on the document's last line — its
# presence proves the entire 30k document was in the LLM prompt.
expect(bubble).to_contain_text(sentinel, timeout=30_000)
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
# Grounded: the document's source chip renders under the bubble.
chip = page.locator(".msg.brain .source-chip", has_text="gitlab-30k.md")
expect(chip).to_have_count(1)
expect(chip.first).to_contain_text("Homelab/gitlab-30k.md")
# Button recovers (never stale) and the turn was grounded.
expect(page.locator("#send-btn")).to_be_enabled()
expect(page.locator("#send-label")).to_have_text("Send")
row = _last_query_log()
assert row.question == QUESTION
assert row.deflected is False
assert "Homelab/gitlab-30k.md" in row.sources
def test_second_document_of_over_cap_pair_reaches_llm(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""Two ~16k documents (32k combined — under the old budget the
lower-ranked one was truncated in place): the SECOND document, the
last block inside <documents>, reaches the LLM whole — its sentinel is
the one the tail echo surfaces."""
sentinel_a = "WHOLE-DOC-TAIL-PAIR-A"
sentinel_b = "WHOLE-DOC-TAIL-PAIR-B"
# Doc A's first chunk carries the question's key tokens → it ranks
# first in both candidate lists → it comes first in <documents>.
content_a, chunks_a = _pair_doc(
"gitlab install playbook: run the gitlab install playbook end to end.",
"the server room keeps a steady temperature and the racks are labelled.",
sentinel_a,
)
# Doc B's first chunk has only a weaker overlap ("playbook", "notes")
# → it ranks second → it is the LAST block inside <documents>.
content_b, chunks_b = _pair_doc(
"playbook notes: the playbook notes track what changed and where.",
"the rack elevation drawing shows cable trays and pdu positions.",
sentinel_b,
)
assert len(content_a) + len(content_b) > OLD_CONTEXT_CAP # 32k > 24k
def seed(db: Session) -> None:
_seed_doc(
db, "Homelab", "gitlab-install-playbook.md",
"GitLab Install Playbook", content_a, chunks_a,
)
_seed_doc(
db, "Homelab", "playbook-notes.md",
"Playbook Notes", content_b, chunks_b,
)
_reset_db(seed)
bubble = _ask(page, app_url, QUESTION)
# The tail echo quotes doc B's sentinel (the last block's tail) — doc B
# was in the prompt whole, past the old budget. Doc A's sentinel sits
# mid-prompt, so it must NOT be in the quoted tail: that is what pins
# the rank order (A first, B last inside <documents>).
expect(bubble).to_contain_text(sentinel_b, timeout=30_000)
expect(bubble).not_to_contain_text(sentinel_a)
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
# Both documents are cited (top-2), in rank order.
chips = page.locator(".msg.brain .source-chip")
expect(chips).to_have_count(2)
expect(page.locator(".msg.brain .source-chip",
has_text="gitlab-install-playbook.md")).to_have_count(1)
expect(page.locator(".msg.brain .source-chip",
has_text="playbook-notes.md")).to_have_count(1)
row = _last_query_log()
assert row.question == QUESTION
assert row.deflected is False
assert "Homelab/gitlab-install-playbook.md" in row.sources
assert "Homelab/playbook-notes.md" in row.sources
def test_small_document_path_unchanged(
page: Page, app_url: str, mock_llm: int, db_ready: None
) -> None:
"""Regression: the standard (small) fixtures still take the grounded
path, byte-identical to before — no marker, kubernetes.md cited."""
_reset_db(None)
summary = _run_in_thread(_import_fixtures(mock_llm))
assert summary.added == 13 # A9 formats (phase 47 added quadlet+j2)
bubble = _ask(page, app_url, SMALL_QUESTION)
expect(bubble).to_contain_text(SMALL_QUESTION, timeout=30_000)
expect(bubble).to_contain_text(MOCK_ANSWER_MARKER)
expect(bubble).not_to_contain_text(TRUNCATION_MARKER)
chip = page.locator(".msg.brain .source-chip", has_text="kubernetes.md")
expect(chip).to_have_count(1)
row = _last_query_log()
assert row.deflected is False
assert "docs/homelab/kubernetes.md" in row.sources