phase: 118_summary_seed_context
Build and Push Containers / build-and-push-app (push) Successful in 2m2s
Build and Push Containers / build-and-push-db (push) Successful in 14s

**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.

- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
This commit is contained in:
2026-09-16 06:57:49 -04:00
parent 21aad84a6d
commit 9820c361b0
80 changed files with 4690 additions and 1302 deletions
+169 -66
View File
@@ -221,50 +221,71 @@ def test_agent_tools_names_and_parameters() -> None:
"'homelab/active'). Omit it to list every source."
)
read = by_name["read"]["function"]
# Tool-calling fast loop (2026-09-04, controlled fixture gate):
# the do-not-read rule is FRONT-LOADED — the controlled gate's
# telemetry showed the `lite` model obeying the user's "open it /
# read it" and reading seed-context documents the <documents>
# section already carries (every refusal of a 12-call run was
# ALREADY_IN_CONTEXT); the rule now leads the description instead
# of sitting mid-paragraph, and the tool is framed as "only for
# documents NOT already in <documents>". Phase 95 (task 01): the
# read-truncation sentence is inserted before the one-call-at-a-
# time discipline clause (the discipline rule stays last, as in the
# other two tools) — a capped read carries the TRUNCATED notice and
# the `grep` follow-up (the pinned copy).
# Phase 118 (task 04, A6): the read description is rewritten for
# the summary-seed mode — the <documents> section shows the
# top-ranked documents' SUMMARIES (their full texts are NOT in
# the prompt yet); read adds one of them (or any other document)
# to the context; only an already-read document is refused. The
# combined-identity handoff to the `ls` output / the <documents>
# blocks stays; the truncation-notice paragraph (phase 95) and
# the one-call-at-a-time discipline clause (task 05) survive the
# rewrite byte-identical.
assert read["description"] == (
"Do not call this tool for a document already shown in "
"the <documents> section, even when the user asks you to "
"open or read it — its full text is already in your "
"prompt; answer directly from it. Use it only to add a "
"document NOT already in <documents> to your context, "
"by its combined `source/path` string. Very large "
"documents are truncated: you receive the first part "
"plus a TRUNCATED notice naming how many more characters "
"exist — the notice is authoritative, the document did "
"NOT end where it stopped. Follow it and use `grep` "
"(pattern) to locate the rest — it searches the whole "
"document. Call one tool at a time — wait for this "
"result before your next call."
"The <documents> section shows the SUMMARIES of the "
"top-ranked documents — their full texts are NOT in "
"your prompt yet. Use this tool to add one of them (or "
"any other document) to your context, by its combined "
"`source/path` string, exactly as shown in the `ls` "
"output or the <documents> blocks. Do not re-read a "
"document you have already read — its full text is "
"already in your prompt. Very large documents are "
"truncated: you receive the first part plus a TRUNCATED "
"notice naming how many more characters exist — the "
"notice is authoritative, the document did NOT end "
"where it stopped. Follow it and use `grep` (pattern) "
"to locate the rest — it searches the whole document. "
"Call one tool at a time — wait for this result before "
"your next call."
)
# The byte-preserved contracts inside the rewrite, pinned as
# substrings: the combined source/path identity, the
# truncation-notice paragraph, the one-call-at-a-time
# discipline sentence.
assert (
"by its combined `source/path` string, exactly as shown in "
"the `ls` output or the <documents> blocks"
) in read["description"]
assert (
"Very large documents are truncated: you receive the first "
"part plus a TRUNCATED notice naming how many more "
"characters exist — the notice is authoritative, the "
"document did NOT end where it stopped. Follow it and use "
"`grep` (pattern) to locate the rest — it searches the "
"whole document."
) in read["description"]
assert (
"Call one tool at a time — wait for this result before "
"your next call."
) in read["description"]
read_params = read["parameters"]
assert read_params["type"] == "object"
assert read_params["required"] == ["path"]
assert set(read_params["properties"]) == {"path"}
assert read_params["properties"]["path"]["type"] == "string"
# The combined source/path string is the canonical document identity
# (phase 70) — the description pins it with a worked example. Phase
# 72 (task 02): the bare-path contract is stated up front; task 05
# (live gate iteration 1): the do-not-re-read clause (the dedupe
# refusal's prevention at the prompt).
# (phase 70) — the description pins it with a worked example; the
# bare-path contract is stated up front (phase 72, task 02). Phase
# 118 (task 04, A6): the trailing clause is the do-not-RE-READ
# teaching — the seeds are summary blocks, not full text, so a
# first read of a suggested document succeeds and only an
# already-read document is refused.
assert read_params["properties"]["path"]["description"] == (
"The document to add to your context, as the combined "
"`source/path` string exactly as shown in the `ls` output (e.g. "
"'homelab/active/container_caddy/caddy.md'). A bare document "
"path (without the source name) will not resolve. Only pass a "
"document NOT already shown in the <documents> section — it is "
"already in your context; do not re-read it."
"path (without the source name) will not resolve. Do not "
"re-read a document you have already read — its full "
"text is already in your prompt."
)
grep = by_name["grep"]["function"]
# Task 05 (live gate iterations 2-6, refined in the 2026-09-03
@@ -1455,15 +1476,17 @@ def test_read_bare_filename_without_slash_keeps_no_db_refusal(
assert llm.requests[1][1] == AGENT_TOOLS
def test_read_bare_path_of_seed_doc_gets_suggestion_then_dedupe(
def test_read_bare_path_of_suggested_doc_gets_suggestion_then_succeeds(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Dedupe precedence: the in-context dedupe fires on the SPLIT pair
of the argument — the bare path of an in-context document
(``read('app/rag/importer.py')`` with ``sample/app/rag/importer.py``
seeded) is NOT that pair, so it is not a dedupe: it gets the
suggestion line naming the combined identity, and the model's next,
correctly-formed call is then deduped as ALREADY_IN_CONTEXT."""
"""Dedupe precedence (phase 118): the in-context dedupe fires on
the SPLIT pair of a document already READ — the bare path of a
SUGGESTED (seeded) document (``read('app/rag/importer.py')`` with
``sample/app/rag/importer.py`` seeded) is NOT that pair, so it is
not a dedupe: it gets the suggestion line naming the combined
identity, and the model's next, correctly-formed call ADDS the
suggested document's full text (A6: the seeds are summary blocks,
not full text — a first read of a suggested document succeeds)."""
seed = [_doc("sample", "app/rag/importer.py", "Importer", "IMPORTER")]
def _find(db: Any, source: str, path: str) -> Document | None:
@@ -1482,8 +1505,9 @@ def test_read_bare_path_of_seed_doc_gets_suggestion_then_dedupe(
],
[
# Round 2: the corrected call (the suggested combined
# identity) — the seed document is already in context, so it
# dedupes.
# identity) — the suggested document's full text is NOT in
# the prompt (only its summary is), so this read succeeds
# and adds the full content.
ToolCallPiece(
id="call_2",
name="read",
@@ -1493,12 +1517,18 @@ def test_read_bare_path_of_seed_doc_gets_suggestion_then_dedupe(
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
assert holder.read_docs == [] and holder.tool_calls == 0 # both refused
# Round 1 refused (the teaching), round 2 executed (A6).
assert holder.read_docs == [seed[0]]
assert holder.tool_calls == 1
assert llm.requests[1][0][3]["content"] == (
"No document at 'app/rag/importer.py' — "
"did you mean 'sample/app/rag/importer.py'?"
)
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
assert llm.requests[2][0][5]["content"] == (
"Document sample/app/rag/importer.py:\n"
"date: 2024-06-15\n"
"IMPORTER"
)
@pytest.mark.parametrize(
@@ -1529,16 +1559,23 @@ def test_read_missing_arguments_refused(
assert llm.requests[1][1] == AGENT_TOOLS
def test_reading_a_seed_doc_is_already_in_context(monkeypatch: pytest.MonkeyPatch) -> None:
"""The combined identity of a seeded document: its split pair is in
the known set → ALREADY_IN_CONTEXT with no DB lookup (the dedupe
check precedes the resolve)."""
def test_reading_a_suggested_seed_doc_adds_its_full_text(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 118 (A6): a FIRST read of a suggested (seeded) document
SUCCEEDS — the seed is a summary block in the prompt, not the full
text, so it falls out of the dedupe set (``holder.read_docs``
only): the read goes through the existing path unchanged — the
full-content result (header + date line), appended to
``holder.read_docs``, counted in ``holder.tool_calls``."""
seed = [_doc("Homelab", "kubernetes.md", "Kubernetes", "K8S-CONTENT")]
def _boom(*_a: Any, **_k: Any) -> None:
raise AssertionError("find_document must not be called for a seeded doc")
monkeypatch.setattr(agent, "find_document", _boom)
monkeypatch.setattr(
agent,
"find_document",
lambda db, source, path: seed[0]
if (source, path) == ("Homelab", "kubernetes.md")
else None,
)
holder = AgentHolder()
llm = ScriptedLLM(
[
@@ -1551,11 +1588,58 @@ def test_reading_a_seed_doc_is_already_in_context(monkeypatch: pytest.MonkeyPatc
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
assert holder.read_docs == [] and holder.tool_calls == 0
assert llm.requests[1][0][3]["content"] == agent.ALREADY_IN_CONTEXT
assert holder.read_docs == [seed[0]]
assert holder.tool_calls == 1
assert llm.requests[1][0][3]["content"] == (
"Document Homelab/kubernetes.md:\ndate: 2024-06-15\nK8S-CONTENT"
)
assert llm.requests[1][1] == AGENT_TOOLS
def test_re_reading_a_suggested_doc_is_deduped(monkeypatch: pytest.MonkeyPatch) -> None:
"""Phase 118 (A6): once the suggested document's full text has been
read into context, a SECOND read of the same combined identity is
refused with the byte-identical ``ALREADY_IN_CONTEXT`` line — the
dedupe set is ``holder.read_docs`` only, so the refusal fires with
NO DB lookup (the dedupe check precedes the resolve) and the
counters stay untouched."""
seed = [_doc("Homelab", "kubernetes.md", "Kubernetes", "K8S-CONTENT")]
lookups: list[tuple[str, str]] = []
def _find(db: Any, source: str, path: str) -> Document | None:
lookups.append((source, path))
return seed[0] if (source, path) == ("Homelab", "kubernetes.md") else None
monkeypatch.setattr(agent, "find_document", _find)
holder = AgentHolder()
llm = ScriptedLLM(
[
ToolCallPiece(
id="call_1",
name="read",
arguments={"path": "Homelab/kubernetes.md"},
)
],
[
ToolCallPiece(
id="call_2",
name="read",
arguments={"path": "Homelab/kubernetes.md"},
)
],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
assert holder.read_docs == [seed[0]] # appended exactly once
assert holder.tool_calls == 1 # the re-read counts nothing
assert lookups == [("Homelab", "kubernetes.md")] # the re-read deduped pre-lookup
assert llm.requests[1][0][3]["content"] == (
"Document Homelab/kubernetes.md:\ndate: 2024-06-15\nK8S-CONTENT"
)
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
# Rejected → the tools are still offered on the next request (the
# round cap is the only bound).
assert llm.requests[1][1] == AGENT_TOOLS
assert llm.requests[2][1] == AGENT_TOOLS
def test_reading_an_already_read_doc_is_deduped(monkeypatch: pytest.MonkeyPatch) -> None:
@@ -1718,28 +1802,47 @@ def test_run_agent_short_read_yields_no_tool_result_piece(
assert isinstance(out[1], StreamPiece)
def test_read_truncation_does_not_touch_refusal_paths(
def test_read_truncation_paths_with_suggested_seed(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 95: the read refusal paths are untouched by the cap — a SEED
document that is over the cap is still refused with
``ALREADY_IN_CONTEXT`` (not truncated, nothing recorded, nothing
counted), and an unknown path is still the no-document refusal (no
"""Phase 118 (A6) re-target of the phase-95 pins ("seed" →
"suggested"): a SUGGESTED (seeded) document that is over the cap is
read through the ordinary cap path — cut at the cap + the shared
marker + the pinned notice, the truncation recorded on the holder,
and the call still successful (the seed is a summary, not full
text) — and an unknown path is still the no-document refusal (no
content is read, so no truncation either)."""
big = "B" * 5000 # far over the tiny cap below
seed = _doc("S", "seed.md", "Seed", big)
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
monkeypatch.setattr(agent, "all_documents", lambda db: [])
# (a) Reading the (over-cap) seed doc → ALREADY_IN_CONTEXT (refusal).
monkeypatch.setattr(
agent,
"find_document",
lambda db, source, path: seed
if (source, path) == ("S", "seed.md")
else None,
)
monkeypatch.setattr(agent, "all_documents", lambda db: [seed])
# (a) Reading the (over-cap) SUGGESTED doc → the capped read: cut at
# the cap, marker + pinned notice, holder entry, still counted.
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/seed.md"})],
[StreamPiece("content", "ans")],
)
asyncio.run(_run(llm, holder, _settings(read_max_chars=100), seed_docs=[seed]))
assert llm.requests[1][0][3]["content"] == agent.ALREADY_IN_CONTEXT
assert holder.read_truncations == []
assert holder.tool_calls == 0 and holder.read_docs == []
assert llm.requests[1][0][3]["content"] == (
"Document S/seed.md:\n"
"date: 2024-06-15\n"
+ big[:100]
+ "\n"
+ TRUNCATION_MARKER
+ "\n"
+ READ_TRUNCATION_NOTICE.format(shown=100, total=5000)
)
# (argument, cap, total) — the raw argument, the cap kept, the
# true length.
assert holder.read_truncations == [("S/seed.md", 100, 5000)]
assert holder.tool_calls == 1 and holder.read_docs == [seed]
# (b) An unknown path → the no-document refusal (argument echoed),
# even though a big doc could have truncated — no content is read.
holder2 = AgentHolder()
+264 -120
View File
@@ -8,6 +8,7 @@ session, and the LLM all faked, so the whole deflection contract
"""
from __future__ import annotations
import hashlib
import json
import uuid
from collections.abc import Iterator
@@ -23,6 +24,7 @@ from app.main import app as fastapi_app
from app.models import Document, KbOverview, QueryLog
from app.rag.agent import AGENT_TOOLS
from app.rag.llm import StreamPiece
from app.rag.prompts import build_deflect_prompt
from app.rag.retriever import RetrievedChunk, weak_hit_titles
from app.rag.suggestions import MAX_SUGGESTIONS, derive_suggestions
from tests.conftest import ADMIN_PASSWORD
@@ -46,7 +48,9 @@ def _settings(threshold: float = 0.30, floor: float | None = None) -> Settings:
)
def _doc(title: str, content: str) -> Document:
def _doc(
title: str, content: str, summary: str | None = None
) -> Document:
return Document(
id=uuid.uuid4(),
source="Homelab",
@@ -54,6 +58,10 @@ def _doc(title: str, content: str) -> Document:
full_path="/tmp/doc.md",
title=title,
content=content,
# Phase 30/118: the stored lite-model summary — the HIGH block's
# BODY (task 03). ``None`` exercises the A5 preview fallback
# (the first ``suggestion_preview_chars`` content chars).
summary=summary,
content_hash="0" * 64,
# Phase 106, D5: the HIGH block formats the row's created_at
# UTC date part — the detached fixture carries it (the NOT NULL
@@ -179,11 +187,12 @@ def test_gate_lexical_only_chunk_does_not_inflate_cosine() -> None:
plan = chat_api.plan_turn(chunks, _settings(threshold=0.30))
assert plan.top_score == pytest.approx(0.55)
assert plan.deflected is False # 0.55 >= 0.30 anyway
# Phase 113 (the usefulness bar): Beta's doc ranks first by fused
# score, but a lexical-only doc (cosine 0.0 by construction) cannot
# clear the bar — it lands in the RELATED tier, never the cited one.
assert plan.docs[0].title == "Alpha"
assert plan.related_docs[0].title == "Beta"
# Phase 118 (A3): the suggestion tier has NO floor — the lexical-
# only doc (cosine 0.0 by construction) is SUGGESTED when it ranks.
# Rank order is the fused score, so Beta (0.90) leads Alpha (0.50);
# with two docs nothing is left for the related tier (rank 6+).
assert [d.title for d in plan.suggested_docs] == ["Beta", "Alpha"]
assert plan.related_docs == []
# ---------- lexical support floor (A8 revised 2026-09-14) ----------
@@ -373,178 +382,225 @@ def test_gate_zero_chunks_deflects_with_fallback_chips() -> None:
assert 2 <= len(plan.suggestions) <= MAX_SUGGESTIONS
# ---------- usefulness bar tiering (phase 113, LOCKED A2/A4) ----------
# ---------- summary-seed tiering (phase 118, LOCKED A3/A4/A6) ----------
def _bar_settings(
def _tier_settings(
threshold: float = 0.62,
lex_floor: float = 0.35,
source_floor: float = 0.35,
suggested_cap: int = 5,
related_cap: int = 2,
top_n: int = 2,
) -> Settings:
"""Explicit code defaults (production calibration) — the env's mock-
calibrated floor (tests/conftest.py) is overridden per test."""
calibrated values (tests/conftest.py) are overridden per test.
``source_usefulness_floor`` / ``top_n_docs`` are deliberately left
at their code defaults: phase 118 retired their seeding role (A6) —
``plan_turn`` never consults them (pinned in
``test_plan_turn_does_not_consult_retired_seeding_settings``)."""
return Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
relevance_threshold=threshold,
lexical_support_floor=lex_floor,
source_usefulness_floor=source_floor,
suggested_docs=suggested_cap,
related_max_docs=related_cap,
top_n_docs=top_n,
)
def test_plan_turn_high_tiers_strong_plus_weak() -> None:
"""Grounded turn: the bar-clearing doc is cited (and in the prompt),
the weak 2nd doc loses its citation slot and lands in related_docs —
the recurring incident's fix at the plan level."""
def _seven_docs_with_summaries() -> list[Document]:
"""The 7-doc fixture (rank 1–7 by fused score): every document has a
stored SUMMARY (distinct sentinel) and a distinct FULL-CONTENT
sentinel that must never reach the prompt (A6: the ``read`` tool is
the only full-text path)."""
return [
_doc(
f"Doc {i}",
f"FULL_CONTENT_SENTINEL_{i}_SHOULD_NEVER_REACH_THE_PROMPT",
summary=f"SUMMARY_TEXT_{i}",
)
for i in range(7)
]
def test_plan_turn_high_seeds_top5_suggested_related_is_rank6plus() -> None:
"""The phase-118 core pin (LOCKED A3/A6): the HIGH prompt seeds
exactly the top-5 suggested documents' SUMMARY text and NONE of
their full content; the related tier is rank 6+ (docs 6–7, capped
by ``related_max_docs``)."""
docs_in = _seven_docs_with_summaries()
chunks = [
_chunk(d, 0.9 - 0.1 * i, cosine=0.8 - 0.05 * i) for i, d in enumerate(docs_in)
]
plan = chat_api.plan_turn(chunks, _tier_settings())
assert plan.deflected is False
assert [d.title for d in plan.suggested_docs] == [f"Doc {i}" for i in range(5)]
assert [d.title for d in plan.related_docs] == ["Doc 5", "Doc 6"]
# The prompt seeds the five summaries …
for i in range(5):
assert f"SUMMARY_TEXT_{i}" in plan.system_prompt
# … and NONE of the seven documents' full content (suggested OR
# related) reaches the LLM (A6).
for i in range(7):
assert f"FULL_CONTENT_SENTINEL_{i}" not in plan.system_prompt
def test_plan_turn_high_single_strong_doc_yields_one_suggested() -> None:
"""The suggested cap is a CEILING, not a quota: one doc ⇒ one
suggested doc, an empty related tier (nothing beyond rank 1)."""
strong = _doc("Kubernetes Homelab Cluster", "STRONG_DOC_CONTENT")
plan = chat_api.plan_turn([_chunk(strong, 0.90, cosine=0.80)], _tier_settings())
assert plan.deflected is False
assert [d.title for d in plan.suggested_docs] == ["Kubernetes Homelab Cluster"]
assert plan.related_docs == []
# No stored summary (the fixture default) → the A5 preview fallback
# carries the short content whole (under the 400-char cap).
assert "STRONG_DOC_CONTENT" in plan.system_prompt
def test_plan_turn_high_suggests_strong_and_weak_no_floor() -> None:
"""The recurring incident under phase 118 (A3): the weak 2nd doc no
longer loses a citation slot to a bar — the floor never filters, so
BOTH docs are suggested (rank order) and ride the citation surface
(A4); the HIGH prompt seeds both summaries (the A5 fallback carries
the short fixture content whole)."""
strong = _doc("Kubernetes Homelab Cluster", "STRONG_DOC_CONTENT")
weak = _doc("Backup Strategy", "WEAK_DOC_CONTENT")
chunks = [_chunk(strong, 0.90, cosine=0.80), _chunk(weak, 0.80, cosine=0.20)]
plan = chat_api.plan_turn(chunks, _bar_settings())
plan = chat_api.plan_turn(chunks, _tier_settings())
assert plan.deflected is False
assert [d.title for d in plan.docs] == ["Kubernetes Homelab Cluster"]
assert [d.title for d in plan.related_docs] == ["Backup Strategy"]
# The HIGH prompt carries the cited doc's content only.
assert "STRONG_DOC_CONTENT" in plan.system_prompt
assert "WEAK_DOC_CONTENT" not in plan.system_prompt
def test_plan_turn_high_single_strong_doc_yields_one_cited() -> None:
"""top_n_docs is a CEILING, not a quota: one strong doc ⇒ one cited doc,
an empty related tier (LOCKED A2)."""
strong = _doc("Kubernetes Homelab Cluster", "STRONG_DOC_CONTENT")
plan = chat_api.plan_turn([_chunk(strong, 0.90, cosine=0.80)], _bar_settings())
assert plan.deflected is False
assert [d.title for d in plan.docs] == ["Kubernetes Homelab Cluster"]
assert [d.title for d in plan.suggested_docs] == [
"Kubernetes Homelab Cluster",
"Backup Strategy",
]
assert plan.related_docs == []
assert "STRONG_DOC_CONTENT" in plan.system_prompt
assert "WEAK_DOC_CONTENT" in plan.system_prompt
def test_plan_turn_low_weak_hits_fall_to_related() -> None:
"""Deflected turn: nothing clears the bar ⇒ the cited tier is empty
and the weak hits fall to related_docs (the done frame's home for
their visibility). The LOW prompt is unchanged (titles only)."""
def test_plan_turn_low_weak_hits_are_suggested_record() -> None:
"""Deflected turn: the weak hits are SUGGESTED too (no floor, A3) —
the TurnPlan carries suggested + related for the durable record —
while the LOW prompt itself stays byte-identical (weak-hit titles
only, never content)."""
a = _doc("Alpha", "ALPHA_DOC_NEVER_SENT")
b = _doc("Beta", "BETA_DOC_NEVER_SENT")
chunks = [_chunk(a, 0.30, cosine=0.20), _chunk(b, 0.20, cosine=0.15)]
plan = chat_api.plan_turn(chunks, _bar_settings())
plan = chat_api.plan_turn(chunks, _tier_settings())
assert plan.deflected is True
assert plan.docs == [] # no citation slot below the bar
assert [d.title for d in plan.related_docs] == ["Alpha", "Beta"] # rank order
assert [d.title for d in plan.suggested_docs] == ["Alpha", "Beta"] # rank order
assert plan.related_docs == [] # nothing beyond rank 2 for 2 docs
assert "ALPHA_DOC_NEVER_SENT" not in plan.system_prompt
assert "Beta" in plan.system_prompt # weak-hit titles still carried
assert plan.suggestions # chips unchanged
def test_plan_turn_related_cap_zero_kills_the_related_tier() -> None:
"""related_max_docs=0 is the kill switch: weak docs are scored but
neither cited nor related (the pre-phase-113 visibility, minus the
false citation — a deflected turn cites nothing)."""
a = _doc("Alpha", "AAA")
b = _doc("Beta", "BBB")
chunks = [_chunk(a, 0.30, cosine=0.20), _chunk(b, 0.20, cosine=0.15)]
plan = chat_api.plan_turn(chunks, _bar_settings(related_cap=0))
assert plan.deflected is True
assert plan.docs == []
assert plan.related_docs == []
def test_plan_turn_floor_zero_keeps_legacy_cited_docs() -> None:
"""source_usefulness_floor=0 disables the bar: plan.docs is the legacy
rank-ordered top-N (any cosine, incl. 0.0 lexical-only) and the
related tier is empty."""
a = _doc("Alpha", "AAA")
b = _doc("Beta", "BBB")
chunks = [
_chunk(a, 0.90, cosine=0.0, fts_hit=True), # lexical-only, rank 1
_chunk(b, 0.80, cosine=0.10),
]
plan = chat_api.plan_turn(
chunks, _bar_settings(source_floor=0.0, lex_floor=0.05)
)
assert plan.deflected is False # 0.10 + the fts hit clears the 0.05 lex floor
assert [d.title for d in plan.docs] == ["Alpha", "Beta"] # legacy order
assert plan.related_docs == []
def test_plan_turn_lexically_grounded_below_source_floor_has_no_cited_docs() -> None:
"""The degenerate operator config (citation bar STRICTER than the
grounding bar): a turn grounded by a corroborated-lexical hit whose
cosine sits between the two floors has an EMPTY cited tier — the HIGH
prompt carries no document content (the tools remain the escape
hatch). The bar is a citation filter, not a gate input."""
doc = _doc("Static DNS", "DNS_DOC_CONTENT")
plan = chat_api.plan_turn(
[_chunk(doc, 0.50, cosine=0.35, fts_hit=True)],
_bar_settings(threshold=0.62, lex_floor=0.30, source_floor=0.50),
)
assert plan.deflected is False # 0.35 >= lex floor 0.30, fts fired
assert plan.docs == [] # 0.35 < source floor 0.50 — no citation slot
assert "DNS_DOC_CONTENT" not in plan.system_prompt
# The doc still SCORED — it rides the related tier (the "nearby docs"
# row), it is not invisible.
assert [d.title for d in plan.related_docs] == ["Static DNS"]
def test_plan_turn_related_tier_capped_in_rank_order() -> None:
"""Grounded turn, four bar-clearing docs, ceiling 2: cited = the top-2
in rank order; related = the next two (the ceiling overflow, any
cosine), capped at related_max_docs."""
docs_in = [
_doc(f"Doc {i}", f"DOC_CONTENT_{i}") for i in range(4)
]
"""related_max_docs=0 is the kill switch: rank-6+ docs are scored
and suggested-adjacent but neither suggested nor related — the
done frame's row stays empty."""
docs_in = _seven_docs_with_summaries()
chunks = [
_chunk(d, 0.9 - 0.1 * i, cosine=0.8 - 0.05 * i) for i, d in enumerate(docs_in)
]
plan = chat_api.plan_turn(chunks, _bar_settings(top_n=2, related_cap=2))
plan = chat_api.plan_turn(chunks, _tier_settings(related_cap=0))
assert plan.deflected is False
assert [d.title for d in plan.docs] == ["Doc 0", "Doc 1"]
assert [d.title for d in plan.related_docs] == ["Doc 2", "Doc 3"]
assert len(plan.suggested_docs) == 5 # the suggestion tier is untouched
assert plan.related_docs == []
# The cap restores the rank-6+ row (rank order, capped).
wide = chat_api.plan_turn(chunks, _tier_settings(related_cap=3))
assert [d.title for d in wide.related_docs] == ["Doc 5", "Doc 6"]
def test_plan_turn_suggested_docs_setting_caps_the_suggested_tier() -> None:
"""``suggested_docs`` (``BOR_SUGGESTED_DOCS``) is honored as the
suggestion cap: 3 here ⇒ the top-3 rank-ordered docs are suggested
and rank 4+ falls to the related tier (capped at ``related_max_docs``)."""
docs_in = _seven_docs_with_summaries()
chunks = [
_chunk(d, 0.9 - 0.1 * i, cosine=0.8 - 0.05 * i) for i, d in enumerate(docs_in)
]
plan = chat_api.plan_turn(chunks, _tier_settings(suggested_cap=3))
assert [d.title for d in plan.suggested_docs] == ["Doc 0", "Doc 1", "Doc 2"]
assert [d.title for d in plan.related_docs] == ["Doc 3", "Doc 4"] # cap 2
def test_plan_turn_does_not_consult_retired_seeding_settings() -> None:
"""Phase 118 (A6): ``top_n_docs`` and ``source_usefulness_floor``
lost their seeding role — ``plan_turn`` never consults them. A
degenerate config (the maximum legal bar — 0.62, above every chunk's
cosine of 0.50 — and a top-N of 1) changes nothing: the suggested
tier is still the no-floor top-5 in rank order and the related tier
is still rank 6+.
(Behavioral pin — the settings themselves stay, env back-compat.)"""
docs_in = _seven_docs_with_summaries()
chunks = [
_chunk(docs_in[0], 0.9, cosine=0.50, fts_hit=True),
] + [
_chunk(d, 0.9 - 0.1 * i, cosine=0.50)
for i, d in enumerate(docs_in[1:], start=1)
]
settings = Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
relevance_threshold=0.62,
lexical_support_floor=0.35,
top_n_docs=1, # retired: the old full-text seeding ceiling
# retired: the maximum legal bar (== threshold) — above every
# cosine here (0.50), so the OLD tiering would cite nothing.
source_usefulness_floor=0.62,
related_max_docs=2,
)
plan = chat_api.plan_turn(chunks, settings)
assert plan.deflected is False # 0.50 >= the 0.35 lex floor, fts fired
assert [d.title for d in plan.suggested_docs] == [f"Doc {i}" for i in range(5)]
assert [d.title for d in plan.related_docs] == ["Doc 5", "Doc 6"]
# ---------- summary hits (phase 30: summary → full source document) ----------
def test_summary_hit_on_selected_top_doc_counts() -> None:
"""HIGH branch: the top document was hit via its summary chunk ⇒ 1.
Context assembly is unchanged (A7 revised): the *source* document's
full content lands in the prompt, not the summary text alone.
"""
def test_summary_hit_on_suggested_doc_counts() -> None:
"""HIGH branch: a suggested (rank-1) document hit via its summary
chunk ⇒ 1. Phase 118 (A6): the *summary* is what the LLM sees in
the prompt (no stored summary here → the A5 preview fallback carries
the short fixture content whole)."""
a = _doc("Alpha", "ALPHA_FULL_SOURCE_CONTENT")
b = _doc("Beta", "BETA_FULL_SOURCE_CONTENT")
chunks = [
_chunk(a, 0.90, is_summary=True), # top doc reached through its summary
_chunk(a, 0.90, is_summary=True), # suggested doc reached through its summary
_chunk(b, 0.50),
]
plan = chat_api.plan_turn(chunks, _settings(threshold=0.30))
assert plan.deflected is False
assert plan.summary_hits == 1
# The full source document is what the LLM sees (phase 24 contract).
# The A5 preview fallback (short content, under the 400-char cap)
# carries the content whole — the block body, never more.
assert "ALPHA_FULL_SOURCE_CONTENT" in plan.system_prompt
def test_summary_hit_outside_top_n_selection_not_counted() -> None:
"""A summary chunk on a document outside the top-N (default 2) selection
does not count — only hits that landed in the selected context do."""
a = _doc("Alpha", "ALPHA_CONTENT")
b = _doc("Beta", "BETA_CONTENT")
c = _doc("Gamma", "GAMMA_CONTENT")
def test_summary_hits_counts_suggested_parent_only() -> None:
"""Phase 118 redefinition (redefined from the phase-113 cited set):
a summary chunk counts ONLY when its parent document is in the
SUGGESTED set — a rank-1 (suggested) parent counts, a rank-6
(related-tier-only) parent does not."""
docs_in = [_doc(f"Doc {i}", f"CONTENT_{i}") for i in range(7)]
chunks = [
_chunk(a, 0.90),
_chunk(b, 0.80),
_chunk(c, 0.70, is_summary=True), # 3rd-ranked doc — not selected
_chunk(docs_in[0], 0.90, is_summary=True), # suggested parent — counts
_chunk(docs_in[1], 0.80),
_chunk(docs_in[2], 0.70),
_chunk(docs_in[3], 0.60),
_chunk(docs_in[4], 0.50),
_chunk(docs_in[5], 0.40, is_summary=True), # related-only parent — does not
_chunk(docs_in[6], 0.30),
]
plan = chat_api.plan_turn(chunks, _settings(threshold=0.30))
assert plan.deflected is False
assert [d.title for d in plan.docs] == ["Alpha", "Beta"]
assert plan.summary_hits == 0
assert [d.title for d in plan.suggested_docs] == [f"Doc {i}" for i in range(5)]
assert [d.title for d in plan.related_docs] == ["Doc 5", "Doc 6"]
assert plan.summary_hits == 1
def test_low_branch_counts_summary_hit_on_selected_doc() -> None:
def test_low_branch_counts_summary_hit_on_suggested_doc() -> None:
"""LOW (deflected) branch records ``summary_hits`` too: the weak hit's
parent is still the selected (weak-hit) document."""
parent is still the suggested (weak-hit) document (no floor, A3)."""
a = _doc("Gamma", "GAMMA_DOC_CONTENT")
b = _doc("Delta", "DELTA_DOC_CONTENT")
chunks = [
@@ -683,7 +739,43 @@ def test_high_path_unaffected() -> None:
assert "Reply in plain text only" not in plan.system_prompt
assert "ALPHA_DOC_CONTENT" in plan.system_prompt
assert "BETA_DOC_CONTENT" in plan.system_prompt
assert [d.title for d in plan.docs] == ["Kubernetes Homelab Cluster", "Backup Strategy"]
# Phase 118 (A3): both docs are suggested (no floor) — the durable-
# record input — with nothing left for the related tier (rank 6+).
assert [d.title for d in plan.suggested_docs] == [
"Kubernetes Homelab Cluster",
"Backup Strategy",
]
assert plan.related_docs == []
# ---------- deflected branch: byte-identical (A8 gate untouched) ----------
def test_low_prompt_byte_identical_to_pre_task_sha_pin() -> None:
"""Phase 118 (A8 gate untouched): the deflected prompt is
BYTE-IDENTICAL to pre-task on the same chunks — it is exactly
``build_deflect_prompt(weak_hit_titles(…))`` (the tiering feeds the
TurnPlan's durable-record fields, never the LOW prompt). The sha256
pin makes any future LOW-body drift loud; the suggestions/chips and
the deflected flag are unchanged."""
a = _doc("Kubernetes Homelab Cluster", "ALPHA_DOC_CONTENT")
b = _doc("Backup Strategy", "BETA_DOC_CONTENT")
chunks = [_chunk(b, 0.10), _chunk(a, 0.20)]
plan = chat_api.plan_turn(chunks, _settings())
assert plan.deflected is True
expected = build_deflect_prompt(weak_hit_titles(chunks))
assert plan.system_prompt == expected # byte-identical to the pre-task build
assert (
hashlib.sha256(plan.system_prompt.encode("utf-8")).hexdigest()
== "603395e013c97be8a13837eda533c0b7bd5da4f7a0806b0ae8ad7d6c52420913"
)
assert plan.suggestions # the "Maybe try" chips are unchanged
# Both tiers still ride the plan (the durable-record inputs, A3).
assert [d.title for d in plan.suggested_docs] == [
"Kubernetes Homelab Cluster",
"Backup Strategy",
]
assert plan.related_docs == []
# ---------- weak_hit_titles (fake retriever mapping) ----------
@@ -1059,3 +1151,55 @@ def test_endpoint_no_kb_row_prompt_unchanged(
assert "<knowledge_base>" not in system["content"]
log_lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
assert log_lines and "kb_chars=0" in log_lines[-1]
def test_endpoint_log_line_records_suggested_after_summary_hits(
client: TestClient,
gate_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Phase 118 (PLAN §9 extension): the per-turn log line gains
``suggested=N`` immediately AFTER ``summary_hits=N`` — the field
order of every existing field is untouched (the phase-114
``retries=N scaffold_stripped=N`` tail stays last). The value is the
seeded suggestion tier's size (``len(plan.suggested_docs)``)."""
_session, _llm = gate_env
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_SENT")
chunks = [_chunk(doc, 0.30), _chunk(doc, 0.20, is_summary=True)]
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever(chunks))
with caplog.at_level("INFO", logger="app.chat"):
_ask(client, "How is my Kubernetes cluster set up?")
log_lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
assert log_lines
line = log_lines[-1]
# The new slot: suggested=N right after summary_hits=N (one suggested
# doc — the single fixture doc — and one summary hit on it).
assert "summary_hits=1 suggested=1" in line
# The full field order (the phase-114 tail stays last).
order = (
"question=",
"embed_ms=",
"top_score=",
"fts_hits=",
"summary_hits=",
"suggested=",
"tuning=",
"kb_chars=",
"history_msgs=",
"threshold=",
"deflected=",
"sources=",
"thinking_chars=",
"tool_calls=",
"total_ms=",
"retries=",
"scaffold_stripped=",
)
idx = -1
for field in order:
pos = line.find(field)
assert pos > idx, f"{field} out of order in the per-turn log line"
idx = pos
+53
View File
@@ -282,6 +282,59 @@ def test_related_max_docs_rejects_negative(
_settings()
def test_suggested_docs_default_and_env_override(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 118 (LOCKED A3): the start-here suggestion tier cap —
default 5 (the owner directive, TODO L3), env-tunable."""
monkeypatch.delenv("BOR_SUGGESTED_DOCS", raising=False)
assert _settings().suggested_docs == 5
monkeypatch.setenv("BOR_SUGGESTED_DOCS", "3")
assert _settings().suggested_docs == 3
def test_suggested_docs_rejects_zero_and_negative(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""``0`` (no starting points) and a negative cap are typos (the
``agent_max_rounds`` pattern); ``1`` is the minimum legal value."""
monkeypatch.setenv("BOR_SUGGESTED_DOCS", "0")
with pytest.raises(ValidationError, match="suggested_docs"):
_settings()
monkeypatch.setenv("BOR_SUGGESTED_DOCS", "-2")
with pytest.raises(ValidationError, match="suggested_docs"):
_settings()
monkeypatch.setenv("BOR_SUGGESTED_DOCS", "1")
assert _settings().suggested_docs == 1
def test_suggestion_preview_chars_default_and_env_override(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Phase 118, task 03 (LOCKED A5): the NULL-summary preview cap —
default 400, env-tunable via ``BOR_SUGGESTION_PREVIEW_CHARS``."""
monkeypatch.delenv("BOR_SUGGESTION_PREVIEW_CHARS", raising=False)
assert _settings().suggestion_preview_chars == 400
monkeypatch.setenv("BOR_SUGGESTION_PREVIEW_CHARS", "800")
assert _settings().suggestion_preview_chars == 800
def test_suggestion_preview_chars_rejects_zero_and_negative(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""``0``/negative would preview an empty/absent prefix — typos that
fail loudly (the ``agent_max_rounds`` pattern); ``1`` is the minimum
legal value."""
monkeypatch.setenv("BOR_SUGGESTION_PREVIEW_CHARS", "0")
with pytest.raises(ValidationError, match="suggestion_preview_chars"):
_settings()
monkeypatch.setenv("BOR_SUGGESTION_PREVIEW_CHARS", "-5")
with pytest.raises(ValidationError, match="suggestion_preview_chars"):
_settings()
monkeypatch.setenv("BOR_SUGGESTION_PREVIEW_CHARS", "1")
assert _settings().suggestion_preview_chars == 1
def test_read_max_chars_default_and_env_override(
monkeypatch: pytest.MonkeyPatch,
) -> None:
+272 -22
View File
@@ -4,8 +4,10 @@ The walk tests are pure filesystem (``tmp_path``); the delta and summary
tests run against the local compose Postgres (preferred — a real vector
table), skipping with clear instructions when the stack is not up.
Summaries (phase 30): non-markdown files get a ``lite``-model summary via
the fake's deterministic ``chat`` (``"Summary of <first token>"``); the
Summaries (phase 30; phase 118, A2: every file, markdown included):
every file gets a ``lite``-model summary via the fake's deterministic
``chat`` (``"Summary of <first token>"``); an unchanged doc whose summary
is NULL is backfilled on the next run (``summary_backfilled``); the
sentinel word ``SUMMARY-BLOWUP`` makes ``chat`` raise :class:`LLMError`
for the fail-soft path.
"""
@@ -13,10 +15,12 @@ from __future__ import annotations
import asyncio
import logging
from datetime import UTC, datetime
from pathlib import Path
import pytest
from sqlalchemy import func, select
from sqlalchemy.orm import Session
import app.rag.importer as importer
from app.config import Settings
@@ -29,7 +33,7 @@ from app.rag.importer import (
iter_importable_files,
match_extension,
)
from app.rag.llm import EmbeddingError
from app.rag.llm import EmbeddingError, LLMError
from tests.fakes import FakeEmbedder
#: The original seven A9 formats as dotted suffixes (pre-phase-47 default
@@ -47,6 +51,15 @@ class _PoisonEmbedder(FakeEmbedder):
return await super().embed(texts)
class _FailingChatEmbedder(FakeEmbedder):
"""A ``lite`` model that always fails (drives the summary fail-soft
path — including the phase-118 backfill)."""
async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str:
self.chat_calls.append(list(messages))
raise LLMError("simulated lite-model failure (test sentinel)")
class _CapEmbedder(FakeEmbedder):
"""Simulates the endpoint's ~1024-token input cap at ~1.1 chars/token:
any single text over 1000 chars is rejected (URL-dense worst case)."""
@@ -266,16 +279,19 @@ def test_added_then_unchanged_then_updated_then_pruned(db, tmp_path: Path) -> No
try:
s1 = asyncio.run(import_sources([root], llm, session=db))
assert (s1.files, s1.added, s1.unchanged, s1.updated, s1.pruned) == (2, 2, 0, 0, 0)
# a.md has two sections (2 chunks), b.md one (1 chunk).
# a.md has two sections (2 chunks), b.md one (1 chunk) — content
# chunks only; the ``is_summary`` chunks live in ``summaries``.
assert s1.chunks == 3
# Embeddings are stored with the configured dimension.
assert s1.summaries == 2 # phase 118 (A2): markdown is summarized too
# Embeddings are stored with the configured dimension: 3 content
# chunks + 2 ``is_summary`` chunks (one per doc, phase 118 A2).
n = db.scalar(
select(func.count())
.select_from(Chunk)
.join(Document, Document.id == Chunk.document_id)
.where(Document.source == root.name)
)
assert n == 3
assert n == 5
for c in db.scalars(
select(Chunk)
.join(Document, Document.id == Chunk.document_id)
@@ -300,14 +316,15 @@ def test_added_then_unchanged_then_updated_then_pruned(db, tmp_path: Path) -> No
assert db.scalar(
select(Document).where(Document.source == root.name, Document.path == "a.md")
) is None
# Chunks of the pruned document are gone (FK cascade).
# Chunks of the pruned document are gone (FK cascade); b.md's
# content chunk + its ``is_summary`` chunk survive (phase 118 A2).
n_after = db.scalar(
select(func.count())
.select_from(Chunk)
.join(Document, Document.id == Chunk.document_id)
.where(Document.source == root.name)
)
assert n_after == 1
assert n_after == 2
finally:
_cleanup_source(db, root.name)
@@ -438,8 +455,11 @@ def test_chunk_positions_and_titles(db, tmp_path: Path) -> None:
select(Document).where(Document.source == root.name, Document.path == "multi.md")
)
assert doc is not None
positions = sorted(c.position for c in doc.chunks)
assert positions == list(range(len(doc.chunks))) and len(doc.chunks) >= 2
# 0-based CONTENT positions (the ``is_summary`` chunk sits at −1,
# phase 30/118).
content = [c for c in doc.chunks if not c.is_summary]
positions = sorted(c.position for c in content)
assert positions == list(range(len(content))) and len(content) >= 2
finally:
_cleanup_source(db, root.name)
@@ -535,7 +555,7 @@ def test_quadlet_and_j2_files_get_stem_titles_and_per_format_counts(
_cleanup_source(db, root.name)
# ---------- phase 30: lite-model summaries for non-markdown files ----------
# ---------- phase 30: lite-model summaries (phase 118: every file) ----------
def test_non_markdown_file_gets_stored_and_indexed_summary(db, tmp_path: Path) -> None:
@@ -571,9 +591,11 @@ def test_non_markdown_file_gets_stored_and_indexed_summary(db, tmp_path: Path) -
_cleanup_source(db, root.name)
def test_markdown_file_never_gets_summary(db, tmp_path: Path) -> None:
"""Markdown is already natural language: no summary, no ``is_summary``
chunk, and the ``lite`` model is never called."""
def test_markdown_file_gets_stored_summary(db, tmp_path: Path) -> None:
"""Phase 118 (A2): markdown is summarized too (the phase-30 exclusion
is retired) — ``documents.summary`` is set and one ``is_summary``
chunk (position −1, embedded) is indexed alongside the content
chunks."""
root = tmp_path / "mdsrc"
root.mkdir()
(root / "note.md").write_text("# Note\n\nmarkdown body\n")
@@ -581,14 +603,240 @@ def test_markdown_file_never_gets_summary(db, tmp_path: Path) -> None:
try:
summary = asyncio.run(import_sources([root], llm, session=db))
assert summary.added == 1
assert summary.summaries == 0 and summary.summary_errors == 0
assert llm.chat_calls == [] # the model was never asked
assert summary.summaries == 1 and summary.summary_errors == 0
assert llm.chat_calls # the lite model WAS asked (phase 118)
doc = db.scalar(
select(Document).where(Document.source == root.name, Document.path == "note.md")
)
assert doc is not None
assert doc.summary is None
assert doc.chunks and all(not c.is_summary for c in doc.chunks)
assert doc.summary is not None
# Deterministic fake reply + the code-appended pointer line.
assert doc.summary.startswith("Summary of")
assert doc.summary.endswith(f"Source: {root.name}/note.md")
schunks = [c for c in doc.chunks if c.is_summary]
assert len(schunks) == 1
assert schunks[0].position == -1
assert schunks[0].content == doc.summary
assert schunks[0].embedding is not None and len(schunks[0].embedding) == 768
# Content chunks stay 0-based and are never flagged as summaries.
content = [c for c in doc.chunks if not c.is_summary]
assert sorted(c.position for c in content) == list(range(len(content)))
finally:
_cleanup_source(db, root.name)
# ---------- phase 118 (A2): NULL-summary backfill on the unchanged path ----------
def _clear_stored_summary(db: Session, doc: Document) -> None:
"""Simulate a NULL-summary row (a pre-phase-30 row, or a cleared
summary): the content stays, only the summary + its chunk go away."""
doc.summary = None
for c in [c for c in doc.chunks if c.is_summary]:
doc.chunks.remove(c)
db.commit()
def test_unchanged_doc_with_null_summary_is_backfilled(db, tmp_path: Path) -> None:
"""Phase 118 (A2): an unchanged doc whose summary is NULL gets a
summary-only backfill on the next sync: summary stored + one embedded
``is_summary`` chunk, counted ``summary_backfilled`` — never
``summaries``, never added/updated/pruned, no content re-embed."""
root = tmp_path / "bfill"
root.mkdir()
(root / "note.md").write_text("# Note\n\nmarkdown body\n")
llm = FakeEmbedder()
try:
first = asyncio.run(import_sources([root], llm, session=db))
assert (first.added, first.summaries) == (1, 1)
doc = db.scalar(
select(Document).where(Document.source == root.name, Document.path == "note.md")
)
assert doc is not None and doc.summary is not None
_clear_stored_summary(db, doc) # the NULL-summary row the backfill targets
embed_before = len(llm.calls)
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.updated, second.pruned) == (0, 0, 0)
assert second.unchanged == 1
assert second.summary_backfilled == 1
assert second.summaries == 0 and second.summary_errors == 0
# No content re-embed: exactly one new embed batch, the summary
# text only.
assert len(llm.calls) == embed_before + 1
db.expire_all()
doc = db.scalar(
select(Document).where(Document.source == root.name, Document.path == "note.md")
)
assert doc is not None
assert doc.summary is not None
assert llm.calls[-1] == [doc.summary] # only the backfilled summary
schunks = [c for c in doc.chunks if c.is_summary]
assert len(schunks) == 1
assert schunks[0].position == -1
assert schunks[0].content == doc.summary
assert schunks[0].embedding is not None and len(schunks[0].embedding) == 768
# The content chunk is untouched.
content = [c for c in doc.chunks if not c.is_summary]
assert len(content) == 1 and content[0].embedding is not None
finally:
_cleanup_source(db, root.name)
def test_unchanged_doc_with_stored_summary_never_resummarizes(db, tmp_path: Path) -> None:
"""Phase 118 (A2): an unchanged doc that ALREADY has a summary (the
third sync of the lifecycle) makes no summary LLM call at all and
gains no chunks — owner-edited (non-NULL) summaries are never
touched."""
root = tmp_path / "noref"
root.mkdir()
(root / "note.md").write_text("# Note\n\nmarkdown body\n")
llm = FakeEmbedder()
try:
asyncio.run(import_sources([root], llm, session=db))
chat_before = len(llm.chat_calls)
embed_before = len(llm.calls)
chunk_before = db.scalar(
select(func.count())
.select_from(Chunk)
.join(Document, Document.id == Chunk.document_id)
.where(Document.source == root.name)
)
second = asyncio.run(import_sources([root], llm, session=db))
assert second.unchanged == 1
assert second.summary_backfilled == 0 and second.summaries == 0
assert second.summary_errors == 0
assert len(llm.chat_calls) == chat_before # the model was never asked
assert len(llm.calls) == embed_before # no embedding of any kind
chunk_after = db.scalar(
select(func.count())
.select_from(Chunk)
.join(Document, Document.id == Chunk.document_id)
.where(Document.source == root.name)
)
assert chunk_after == chunk_before # no new chunk of any kind
finally:
_cleanup_source(db, root.name)
def test_unchanged_doc_with_empty_string_summary_is_never_backfilled(
db, tmp_path: Path
) -> None:
"""Phase 118 (A2, strict ``is None``): an empty-string summary is
owner-set (phase 57) — the backfill skips it, the ``lite`` model is
never called, and the value stays byte-identical."""
root = tmp_path / "emptysum"
root.mkdir()
(root / "note.md").write_text("# Note\n\nmarkdown body\n")
llm = FakeEmbedder()
try:
asyncio.run(import_sources([root], llm, session=db))
doc = db.scalar(
select(Document).where(Document.source == root.name, Document.path == "note.md")
)
assert doc is not None
doc.summary = "" # the owner-set empty string (never NULL)
db.commit()
chat_before = len(llm.chat_calls)
second = asyncio.run(import_sources([root], llm, session=db))
assert second.unchanged == 1
assert second.summary_backfilled == 0 and second.summaries == 0
assert second.summary_errors == 0
assert len(llm.chat_calls) == chat_before # the model was never asked
db.expire_all()
doc = db.scalar(
select(Document).where(Document.source == root.name, Document.path == "note.md")
)
assert doc is not None
assert doc.summary == "" # byte-identical — never overwritten
finally:
_cleanup_source(db, root.name)
def test_backfill_runs_on_manually_dated_doc_without_touching_the_date(
db, tmp_path: Path
) -> None:
"""Phase 118 (A2, assumption 7): ``created_at_manual`` protects the
DATE only (phase 106, D1) — a manually-dated, NULL-summary doc still
gets its backfilled summary, and the stored date stays byte-untouched
even though a refresh was due."""
root = tmp_path / "manualdate"
root.mkdir()
(root / "note.md").write_text("# Note\n\nmarkdown body\n")
llm = FakeEmbedder()
try:
asyncio.run(import_sources([root], llm, session=db))
doc = db.scalar(
select(Document).where(Document.source == root.name, Document.path == "note.md")
)
assert doc is not None
manual = datetime(2020, 5, 4, 12, 0, 0, tzinfo=UTC)
doc.created_at = manual
doc.created_at_manual = True
_clear_stored_summary(db, doc) # the NULL-summary row the backfill targets
second = asyncio.run(import_sources([root], llm, session=db))
assert second.unchanged == 1
assert second.summary_backfilled == 1 and second.summary_errors == 0
# A date refresh WAS due (the mtime differs from the 2020
# correction) but the manual flag withheld it — the backfill
# never touches the date either.
assert second.dates_updated == 0
db.expire_all()
doc = db.scalar(
select(Document).where(Document.source == root.name, Document.path == "note.md")
)
assert doc is not None
assert doc.summary is not None # the backfill landed
assert doc.created_at == manual # byte-untouched
assert doc.created_at_manual is True
finally:
_cleanup_source(db, root.name)
def test_backfill_failure_is_fail_soft_and_date_still_refreshes(
db, tmp_path: Path
) -> None:
"""Phase 118 (A2): a backfill whose ``lite`` call fails rolls back
its own session work only — ``summary_errors=1``, the doc row
untouched — while the UNCHANGED path's date refresh still runs
afterwards."""
root = tmp_path / "bfillfail"
root.mkdir()
(root / "note.md").write_text("# Note\n\nmarkdown body\n")
llm = FakeEmbedder()
try:
asyncio.run(import_sources([root], llm, session=db))
doc = db.scalar(
select(Document).where(Document.source == root.name, Document.path == "note.md")
)
assert doc is not None
_clear_stored_summary(db, doc) # the NULL-summary row the backfill targets
# Force a date drift so the refresh is DUE on this run.
doc.created_at = datetime(2020, 1, 1, tzinfo=UTC)
db.commit()
second = asyncio.run(import_sources([root], _FailingChatEmbedder(), session=db))
assert second.unchanged == 1
assert (second.added, second.updated, second.pruned) == (0, 0, 0)
assert second.summary_errors == 1
assert second.summary_backfilled == 0 and second.summaries == 0
db.expire_all()
doc = db.scalar(
select(Document).where(Document.source == root.name, Document.path == "note.md")
)
assert doc is not None
assert doc.summary is None # the failed backfill left the row untouched
assert not any(c.is_summary for c in doc.chunks)
# …but the date refresh ran (the failure only rolled back the
# summary's own session work).
assert second.dates_updated == 1
assert doc.created_at != datetime(2020, 1, 1, tzinfo=UTC)
finally:
_cleanup_source(db, root.name)
@@ -720,11 +968,13 @@ def test_import_summary_log_line_includes_summary_counters(
caplog: pytest.LogCaptureFixture,
) -> None:
"""PLAN §9 summary line: the phase-30 counters sit between
``embed_batches`` and ``formats``; the phase-106 date-refresh
counter sits between ``summary_errors`` and ``formats``."""
``embed_batches`` and ``formats``; the phase-118 backfill counter
sits between ``summary_errors`` and ``dates_updated``; the
phase-106 date-refresh counter sits before ``formats``."""
s = ImportSummary()
s.files, s.added, s.chunks, s.embed_batches = 3, 3, 5, 4
s.summaries, s.summary_errors = 2, 1
s.summary_backfilled = 1
s.dates_updated = 0
s.formats = {"md": 1, "yaml": 2}
with caplog.at_level(logging.INFO, logger="app.importer"):
@@ -732,8 +982,8 @@ def test_import_summary_log_line_includes_summary_counters(
line = caplog.records[-1].getMessage()
assert line == (
"import: summary files=3 added=3 updated=0 unchanged=0 pruned=0 errors=0 "
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 dates_updated=0 "
"formats=yaml:2,md:1"
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
"dates_updated=0 formats=yaml:2,md:1"
)
+59 -3
View File
@@ -16,6 +16,7 @@ import hashlib
from app.rag.prompts import (
PERSONA,
SUGGEST_INTRO,
TOOLS_SECTION,
_base,
build_deflect_prompt,
@@ -67,9 +68,16 @@ def test_high_and_low_bases_byte_locked() -> None:
# ---------- TOOLS_SECTION (the HIGH prompt's locked ``<tools>`` copy) ----------
#: Pre-phase-112 anchors for ``TOOLS_SECTION``.
TOOLS_SECTION_SHA256 = "b834cbe368055e65da82ae3e37a91e6c658c713954703fc79b849a6ebdf4aa53"
TOOLS_SECTION_LEN = 2273
#: Anchors for ``TOOLS_SECTION`` — pre-phase-112 values, RE-CUT for
#: phase 118 (task 04, A6, owner directive 2026-09-15): the ``read``
#: clause was rewritten for the summary-seed mode (the ``<documents>``
#: section holds SUMMARIES — ``read`` adds the full text; only an
#: already-read document is refused). Only that clause moved — the
#: prefix (the ``ls``-clause opening) and the suffix (the
#: discipline-rules ending) survived byte-identical, so they are the
#: same anchors as pre-phase-118.
TOOLS_SECTION_SHA256 = "87ee80faf0170da6ab1de518177313def6460785613fa074312a2a5f9f071750"
TOOLS_SECTION_LEN = 2465
TOOLS_SECTION_PREFIX = (
"<tools>\n"
"You may extend your context with three tools. `ls` lists the "
@@ -120,3 +128,51 @@ def test_deflect_body_byte_locked() -> None:
assert prompt.index("DEFLECT_MODE") < prompt.index(
"Reply in plain text only"
) # the marker precedes the plain-text line
#: The full LOW prompt build on the canonical fixture titles — sha-pinned
#: (phase 118, task 03): the deflection path is UNTOUCHED by the
#: summary-seed re-revision (LOCKED A8) — the same inputs must produce
#: the pre-phase bytes, so this anchor is a pre-phase-118 value.
LOW_PROMPT_SHA256 = "726eddb4eb3bcc26c840011f6f8635d6af55aa09d4d064bbb9e256caa9665837"
LOW_PROMPT_LEN = 968
def test_low_prompt_build_byte_identical_to_pre_phase() -> None:
"""Phase 118 contract: the LOW prompt output is byte-identical to
pre-phase for identical inputs — the summary seeding (and the new
``SUGGEST_INTRO`` line) never leaks into the deflection path."""
prompt = build_deflect_prompt(["T1", "T2"])
assert len(prompt) == LOW_PROMPT_LEN
assert _sha256(prompt) == LOW_PROMPT_SHA256
assert "SUGGEST_INTRO" not in prompt and "<documents>" not in prompt
# ---------- SUGGEST_INTRO (phase 118, task 03 — the start-here framing) ----------
#: Phase-118 anchors for ``SUGGEST_INTRO`` — the ``<documents>``
#: section's intro line (the owner's "start here if these summaries seem
#: right to you" framing, TODO L3). The E2E mock's ``_document_block``
#: parser is regex-based over the block markup (which stays byte-stable
#: around the intro), so this constant is a prompt-copy lock, pinned the
#: way ``TOOLS_SECTION`` is: sha256 + prefix + total length.
SUGGEST_INTRO_SHA256 = "7b14d2dedc6ebc4e440d32dd1edb979a7461c94034b9541c9f37b9042f394d3a"
SUGGEST_INTRO_LEN = 323
SUGGEST_INTRO_PREFIX = (
"The blocks below are the summaries of the top-ranked documents for "
"your question — start here if one seems right to you: "
)
def test_suggest_intro_byte_locked() -> None:
"""The start-here framing is LOCKED copy (phase 118): sha256 + exact
prefix + total length; the three contracts it must carry (summaries
are the starting points; ``read`` adds the full text, which is NOT
in the prompt until read; cite by path) are pinned as substrings."""
assert len(SUGGEST_INTRO) == SUGGEST_INTRO_LEN
assert _sha256(SUGGEST_INTRO) == SUGGEST_INTRO_SHA256
assert SUGGEST_INTRO.startswith(SUGGEST_INTRO_PREFIX)
assert "call `read`" in SUGGEST_INTRO
assert "combined `source/path`" in SUGGEST_INTRO
assert "its full text is not in the prompt until you read it" in SUGGEST_INTRO
assert "Cite the document(s) you used, by path." in SUGGEST_INTRO
+303 -24
View File
@@ -17,9 +17,13 @@ stated up front — the combined ``source/path`` identity for
the drill-down tree contract — one level per call, sources at the
top, folders + files below, ``grep`` as the without-listing locator —
while the ``read``/``grep`` clauses and the discipline rules are
byte-identical): the teaching refusals in :mod:`app.rag.agent`
re-state the same contract; the ``<tools>`` marker keying (HIGH
only) is unchanged.
byte-identical; phase 118, task 04: the ``read`` clause rewritten
for the summary-seed mode — the ``<documents>`` section holds
SUMMARIES, ``read`` adds the full text, and only an already-read
document is refused — while the ``ls``/``grep`` clauses and the
discipline rules stay byte-identical): the teaching refusals in
:mod:`app.rag.agent` re-state the same contract; the ``<tools>``
marker keying (HIGH only) is unchanged.
"""
from __future__ import annotations
@@ -32,6 +36,7 @@ from app.config import Settings
from app.models import Document
from app.rag.prompts import (
PERSONA,
SUGGEST_INTRO,
TOOLS_SECTION,
_base,
build_deflect_prompt,
@@ -55,7 +60,12 @@ KB_INTRO = "The basic categories of everything in this knowledge base (generated
_FIXTURE_CREATED_AT = datetime(2024, 6, 15, 12, 0, 0, tzinfo=UTC)
def _doc(path: str, content: str, title: str) -> Document:
def _doc(
path: str,
content: str,
title: str,
summary: str | None = None,
) -> Document:
return Document(
id=uuid.uuid4(),
source="Homelab",
@@ -65,6 +75,7 @@ def _doc(path: str, content: str, title: str) -> Document:
content=content,
content_hash="0" * 64,
created_at=_FIXTURE_CREATED_AT,
summary=summary,
)
@@ -93,23 +104,64 @@ def test_persona_owner_edits_are_preserved() -> None:
assert "HONESTY GATE" in PERSONA # the gate itself is intact
def test_high_prompt_carries_relevance_marker_and_full_documents() -> None:
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
def test_high_prompt_carries_relevance_marker_and_summary_block() -> None:
"""Phase 118 (LOCKED A6): the grounded turn seeds the document's
SUMMARY — the full content never reaches the prompt (the ``read``
tool is the only full-text path)."""
doc = _doc(
"kubernetes.md",
"Talos Linux on three nodes. FULL_CONTENT_SENTINEL_987654",
"Kubernetes Homelab Cluster",
summary="A Talos Linux cluster on three nodes.",
)
prompt = build_high_prompt([doc])
assert "<relevance>HIGH</relevance>" in prompt
assert "DEFLECT_MODE" not in prompt
assert "<documents>" in prompt and "</documents>" in prompt
assert 'path="kubernetes.md"' in prompt
assert "Talos Linux on three nodes." in prompt
assert "A Talos Linux cluster on three nodes." in prompt # the summary
assert "FULL_CONTENT_SENTINEL_987654" not in prompt # never the content
assert "Talos Linux on three nodes." not in prompt # nor the full sentence
assert "HONESTY GATE" in prompt # persona intact
def test_high_prompt_lists_multiple_documents_in_order() -> None:
a = _doc("a.md", "CONTENT_A", "Title A")
b = _doc("b.md", "CONTENT_B", "Title B")
a = _doc("a.md", "CONTENT_A", "Title A", summary="Summary A.")
b = _doc("b.md", "CONTENT_B", "Title B", summary="Summary B.")
prompt = build_high_prompt([a, b])
assert prompt.index("CONTENT_A") < prompt.index("CONTENT_B")
assert prompt.index("Summary A.") < prompt.index("Summary B.")
assert 'title="Title B"' in prompt
assert "CONTENT_A" not in prompt and "CONTENT_B" not in prompt
def test_high_prompt_seeds_summaries_of_all_suggested_docs() -> None:
"""The phase-118 completion pin: a grounded prompt built from five
summary-bearing documents contains ALL FIVE summaries + the intro,
and ZERO full-content characters (the full texts are not in the
prompt at all)."""
docs = [
_doc(
f"doc{i}.md",
f"FULL_CONTENT_SENTINEL_{i} " + "x" * 100,
f"Title {i}",
summary=f"SUMMARY_{i} of the document.",
)
for i in range(5)
]
prompt = build_high_prompt(docs)
assert SUGGEST_INTRO in prompt
for i in range(5):
assert f"SUMMARY_{i} of the document.\n" in prompt
assert f"FULL_CONTENT_SENTINEL_{i}" not in prompt
assert f'path="doc{i}.md"' in prompt
# One block per document, in the given order.
assert prompt.count("<document ") == 5
positions = [prompt.index(f"SUMMARY_{i}") for i in range(5)]
assert positions == sorted(positions)
# The intro leads the section: <documents> → intro → first block.
i_open = prompt.index("<documents>")
i_block = prompt.index("<document ")
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n\n"
def test_high_prompt_without_documents_stays_honest() -> None:
@@ -143,7 +195,9 @@ def test_zero_note_prompt_is_byte_identical_to_pre_steering() -> None:
"""Phase 15 contract: with no steering notes the prompt is exactly what
it was before the <tuning> section existed. (Phase 37: the HIGH prompt
additionally carries the ``<tools>`` section after the mode body — the
fixtures account for it; the LOW prompt is untouched.)"""
fixtures account for it; phase 118: the ``<documents>`` section leads
with the ``SUGGEST_INTRO`` line — the fixture accounts for it; the
LOW prompt is untouched.)"""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
block = (
'<document source="Homelab" path="kubernetes.md" '
@@ -152,7 +206,14 @@ def test_zero_note_prompt_is_byte_identical_to_pre_steering() -> None:
"</document>"
)
assert build_high_prompt([doc]) == (
_base("HIGH") + "\n<documents>\n" + block + "\n</documents>" + "\n" + TOOLS_SECTION
_base("HIGH")
+ "\n<documents>\n"
+ SUGGEST_INTRO
+ "\n\n"
+ block
+ "\n</documents>"
+ "\n"
+ TOOLS_SECTION
)
# Phase 71: the LOW prompt carries the owner-permitted plain-text
# line after the DEFLECT_MODE sentence (the marker-keying contract
@@ -257,6 +318,86 @@ def test_tools_section_phase95_read_truncation_clause() -> None:
assert "Very large documents are capped" in build_high_prompt([doc])
def test_tools_section_phase118_summary_seed_read_clause() -> None:
"""Phase 118 (task 04, A6): the ``read`` clause is rewritten for
the summary-seed mode — the ``<documents>`` section holds
SUMMARIES (a suggested document's full text is not in the prompt
until ``read`` adds it); do not re-read an already-read document
(answer from the text already in the prompt); if the user asks to
open or read a suggested document, ``read`` it. The phase-72 "do
not call ``read`` for a ``<documents>`` document" copy is retired.
The ``ls`` clause (phase 94 drill-down contract), the ``grep``
clause, and the discipline rules stay byte-identical — the
prompt lock's prefix/suffix anchors survive (see
``test_prompt_lock``)."""
# The new summary-seed copy (pinned byte-for-byte).
assert (
"The <documents> section holds SUMMARIES — the full text of a "
"suggested document is not in your prompt until you `read` it. "
"Do not re-read a document you have already read — its full "
"text is already in your prompt; answer directly from it. If "
"the user asks you to open or read a suggested document, "
"`read` it — that is the point of the section."
) in TOOLS_SECTION
# The read identity handoff now also names the <documents> summary
# blocks (the combined identity is shown there), keeping the
# "including the source name" contract.
assert (
"exactly as shown in the `ls` output — including the source "
"name — or in the <documents> summary blocks — adding its "
"full content to your context"
) in TOOLS_SECTION
# The retired pre-phase-118 copy is gone.
assert "Do not call `read` for a document already shown in" not in TOOLS_SECTION
assert "even when the user asks you to open or read it" not in TOOLS_SECTION
# The ls clause (phase 94 drill-down contract) — byte-identical.
assert (
"`ls` lists the knowledge base as a tree, one level at a "
"time: with no `path` it lists every synced source with its "
"document count and a summary of its contents; with a source "
"name (e.g. 'homelab') it lists that source's top-level "
"folders and files; with a `source/folder` path it drills one "
"level deeper. A listing shows only that level's subfolders "
"and its own files — never the whole knowledge base in one "
"call — and each folder line's summary says what the folder "
"contains before you drill into it. File lines are `source: X "
"| path: Y | title: Z`; to find one specific document without "
"listing, use `grep`."
) in TOOLS_SECTION
# The grep clause — byte-identical.
assert (
"`grep` locates an exact string (case-insensitive) in the "
"indexed documents and returns up to 20 matching "
"`source/path:line: text` lines — a locator, not a "
"context-adder: read the winner with `read`. A grep pattern "
"is a plain substring, NEVER a regex — '.*' and '\\.' are "
"literal text there; if such a pattern returns no matches, "
"retry with the plain text you expect to see. For a normal "
"search pass only `pattern` — its optional `path` argument "
"limits the search to one document you already know, by the "
"same combined `source/path` string; never a source name — a "
"bare document path (without the source name) will not "
"resolve there either."
) in TOOLS_SECTION
# The discipline rules — byte-identical.
assert (
"Make exactly one tool call per reply — a reply carrying two "
"tool calls runs only the first, the second is discarded — "
"and wait for the result before the next call. Never repeat a "
"call that was refused or already succeeded — the refusal "
"already told you the correct form. Answer as soon as you "
"have what you need."
) in TOOLS_SECTION
# The new read clause rides the built HIGH prompt and never the
# LOW (deflected) prompt.
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes", summary="K8S")
high = build_high_prompt([doc])
assert "The <documents> section holds SUMMARIES" in high
assert "`read` it — that is the point of the section." in high
low = build_deflect_prompt(["T1"])
assert "The <documents> section holds SUMMARIES" not in low
def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
"""Phase 72/94: the contract clauses ride the HIGH prompt with the
rest of the section and never leak into the LOW/deflection prompt
@@ -273,20 +414,37 @@ def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
assert "including the source name" not in low
def test_documents_section_has_no_leading_intro() -> None:
"""Phase 72, task 05 (gate iterations 2-3, reverted): the
``<documents>`` section must NOT lead with an in-context reminder
or name the ``<document>`` blocks — the live telemetry showed that
copy primed the model to latch the seed documents' paths as
``ls`` scopes (the incident turn regressed to a cap-reached loop
on run 2 and re-trapped on run 5), and the reminder never flipped
the seed-doc ``read``s (15/15 across gate runs 1-5). The section
is exactly the document blocks again."""
def test_documents_section_leads_with_the_suggest_intro() -> None:
"""Phase 118, task 03: the ``<documents>`` section leads with the
start-here :data:`SUGGEST_INTRO` line BEFORE the first block (the
phase-15 ``_STEERING_INTRO`` / phase-31 ``_KB_INTRO`` precedent) —
only when at least one block is present. This is the summary-as-
starting-point framing, not the reverted phase-72 do-not-read
reminder (that copy taught the seed texts as already-read context;
with A6's summary seeding the blocks are starting points the model
may ``read`` through to full text)."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
high = build_high_prompt([doc])
i_open = high.index("<documents>")
i_block = high.index('<document source="Homelab"')
assert high[i_open : i_block] == "<documents>\n" # no intro line
assert high[i_open : i_block] == f"<documents>\n{SUGGEST_INTRO}\n\n"
# The intro occurs exactly once and sits inside the section.
assert high.count(SUGGEST_INTRO) == 1
assert high.index(SUGGEST_INTRO) > high.index("<documents>")
assert high.index(SUGGEST_INTRO) < high.index("</documents>")
def test_documents_section_without_blocks_has_no_intro() -> None:
"""The intro rides only on present blocks: an empty ``<documents>``
section is exactly the fallback line again (no intro, no blocks)."""
high = build_high_prompt([])
assert SUGGEST_INTRO not in high
i_open = high.index("<documents>")
i_close = high.index("</documents>")
assert (
high[i_open : i_close]
== "<documents>\n(no documents matched — do not invent specifics)\n"
)
def test_tools_section_old_names_and_budget_copy_gone() -> None:
@@ -314,6 +472,117 @@ def test_high_prompt_still_ends_with_tools_section() -> None:
assert old not in prompt
# ---------- phase 118 (task 03): the NULL-summary preview fallback (A5) ----------
def _content_only_doc(content: str) -> Document:
return _doc("kubernetes.md", content, "Kubernetes Homelab Cluster")
def test_null_summary_falls_back_to_preview_plus_marker() -> None:
"""A doc whose ``summary`` is None (a fail-soft import miss) seeds the
first ``suggestion_preview_chars`` (default 400) content characters +
the shared ``[…truncated…]`` marker on its own line — never the full
content, never an LLM call."""
content = "A" * 400 + "B" * 600 # 1000 chars
prompt = build_high_prompt([_content_only_doc(content)])
body_start = prompt.index('date="2024-06-15">\n') + len('date="2024-06-15">\n')
body_end = prompt.index("\n</document>", body_start)
assert prompt[body_start:body_end] == content[:400] + "\n" + TRUNCATION_MARKER
assert "B" * 40 not in prompt # beyond the 400-char cut
assert prompt.count(TRUNCATION_MARKER) == 1 # only the block's marker
def test_whitespace_summary_uses_the_same_preview_fallback() -> None:
for blank in ("", " ", "\n \t "):
doc = _doc("kubernetes.md", "A" * 500 + "B" * 500, "T", summary=blank)
prompt = build_high_prompt([doc])
assert "A" * 400 in prompt
assert "B" * 40 not in prompt
assert TRUNCATION_MARKER in prompt
def test_short_content_preview_is_the_whole_content_unmarked() -> None:
"""Content at or under the cap rides whole — nothing was cut, so no
marker (the marker signals truncation, not the fallback)."""
for content in ("short body", "C" * 399, "D" * 400):
prompt = build_high_prompt([_content_only_doc(content)])
assert content + "\n</document>" in prompt
assert TRUNCATION_MARKER not in prompt
def test_summary_stripped_and_never_truncated_by_the_preview_cap() -> None:
"""The summary body is the stripped ``doc.summary`` — even when it is
longer than the preview cap (the cap bounds only the FALLBACK; a
stored summary is trusted context, LOCKED A5)."""
summary = "S" * 900
content = "CONTENT_SENTINEL " + "x" * 50
prompt = build_high_prompt([_doc("kubernetes.md", content, "T", summary=f" {summary}\n")])
assert summary in prompt
assert content not in prompt
assert TRUNCATION_MARKER not in prompt
def test_preview_cap_setting_is_honored_on_the_fallback_path(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A non-default ``suggestion_preview_chars`` (``BOR_SUGGESTION_PREVIEW_CHARS``
— the env mapping is pinned in :mod:`tests.unit.test_config`) cuts the
preview at the setting's cap on the fallback path."""
from app.rag import prompts as prompts_mod
monkeypatch.setattr(
prompts_mod,
"get_settings",
lambda: Settings(_env_file=None, suggestion_preview_chars=10), # pyright: ignore[reportCallIssue]
)
content = "A" * 100 + "B" * 100
prompt = build_high_prompt([_content_only_doc(content)])
assert "A" * 10 in prompt
assert "A" * 11 not in prompt
assert TRUNCATION_MARKER in prompt
def test_preview_fallback_never_reads_settings_for_summarized_docs(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""House pattern (``build_steering_section`` / ``build_kb_section``):
the fallback cap is read from settings on the fallback path ONLY — a
prompt built from summary-bearing docs makes no settings call for
it (a dead ``get_settings`` must not break such a build)."""
from app.rag import prompts as prompts_mod
doc = _doc("kubernetes.md", "CONTENT_SENTINEL", "T", summary="the summary")
def _dead() -> None: # pragma: no cover - must never be called
raise AssertionError("get_settings() called for a summarized doc")
monkeypatch.setattr(prompts_mod, "get_settings", _dead)
prompt = build_high_prompt([doc])
assert "the summary\n</document>" in prompt
assert "CONTENT_SENTINEL" not in prompt
# ---------- phase 118: the LOW prompt stays byte-identical ----------
def test_low_prompt_byte_identical_to_pre_task() -> None:
"""LOCKED A8 surface: the deflection prompt is untouched by the
summary seeding — byte-identical build on the same inputs (the full
sha pin lives in :mod:`tests.unit.test_prompt_lock`)."""
for titles, notes, kb in (
(["T1", "T2"], None, None),
(["T1"], ["be concise"], None),
([], None, OVERVIEW),
(["T1", "T2"], ["be concise"], OVERVIEW),
):
prompt = build_deflect_prompt(titles, notes=notes, kb_overview=kb)
assert "DEFLECT_MODE" in prompt
assert "<documents>" not in prompt
assert SUGGEST_INTRO not in prompt
assert "<tools>" not in prompt
def test_relevance_placeholder_rejected_for_garbage() -> None:
with pytest.raises(ValueError, match="HIGH or LOW"):
_base("MEDIUM")
@@ -401,7 +670,9 @@ def test_no_overview_prompt_is_byte_identical_to_pre_phase() -> None:
every prompt is exactly what it was before the ``<knowledge_base>``
section existed — with or without steering notes. (Phase 37: the HIGH
prompt additionally carries the ``<tools>`` section after the mode
body — the fixtures account for it; the LOW prompt is untouched.)"""
body; phase 118: the ``<documents>`` section leads with the
``SUGGEST_INTRO`` line — the fixtures account for both; the LOW
prompt is untouched.)"""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
block = (
'<document source="Homelab" path="kubernetes.md" '
@@ -409,7 +680,15 @@ def test_no_overview_prompt_is_byte_identical_to_pre_phase() -> None:
"Talos Linux on three nodes.\n"
"</document>"
)
docs_block = "\n<documents>\n" + block + "\n</documents>" + "\n" + TOOLS_SECTION
docs_block = (
"\n<documents>\n"
+ SUGGEST_INTRO
+ "\n\n"
+ block
+ "\n</documents>"
+ "\n"
+ TOOLS_SECTION
)
high_plain = _base("HIGH") + docs_block
high_steered = _base("HIGH") + "\n" + build_steering_section(["be concise"]) + docs_block
# Phase 71: the owner-permitted plain-text line is part of the
+28
View File
@@ -58,6 +58,7 @@ def _doc(
title: str = "T",
content: str = "CONTENT",
created_at: datetime = CREATED_AT,
summary: str | None = None,
) -> Document:
return Document(
id=uuid.uuid4(),
@@ -68,6 +69,7 @@ def _doc(
content=content,
content_hash="0" * 64,
created_at=created_at,
summary=summary,
)
@@ -171,6 +173,32 @@ def test_high_block_date_always_present_for_every_document() -> None:
assert prompt.count("<document ") == prompt.count(' date="')
def test_high_block_date_survives_the_summary_body_change() -> None:
"""Phase 118 (task 03): the block BODY became the document's summary
(never the full content) — the D5 identity attributes, including
``date`` after ``title``, survive byte-identical around the new
body (the E2E mock's block parser keys off exactly these)."""
doc = _doc(
source="S",
path="P",
title="T",
content="FULL_CONTENT_SENTINEL_42",
created_at=CREATED_AT,
summary="The stored summary.",
)
prompt = build_high_prompt([doc])
block = (
f'<document source="S" path="P" title="T" date="{DATE}">\n'
"The stored summary.\n"
"</document>"
)
assert block in prompt
# Attribute order pinned: date directly after title, body after.
assert f'title="T" date="{DATE}">' in prompt
# The full content stays out (A6) — only the summary rides the block.
assert "FULL_CONTENT_SENTINEL_42" not in prompt
# --------------------------------------------------------------------
# The deflection prompt — byte-identical to the pre-phase text (A8)
# --------------------------------------------------------------------
+229
View File
@@ -11,12 +11,16 @@ from types import SimpleNamespace
import pytest
from app.config import Settings
from app.models import Document
from app.rag import retriever
from app.rag.retriever import (
TRUNCATION_MARKER,
RetrievedChunk,
select_documents,
select_documents_tiered,
select_related,
select_suggested,
)
@@ -258,6 +262,231 @@ def test_select_documents_wrapper_is_legacy_tiering() -> None:
)[0]
# ---------------------------------------------------------------------------
# Phase 118 — select_suggested: the top-N "start here" tier, NO floor (A3)
# ---------------------------------------------------------------------------
def test_suggested_rank_order_by_first_seen_chunk() -> None:
"""The SAME stable walk as ``select_documents_tiered``: a document's
rank is fixed by its FIRST seen chunk in score-descending order — a
doc whose best chunk appears later in the input list still ranks
where that chunk falls."""
a = _doc("a.md", "A" * 50)
b = _doc("b.md", "B" * 50)
c = _doc("c.md", "C" * 50)
chunks = [
_chunk(a, 0.4, position=0), # a's weak chunk comes first
_chunk(b, 0.8),
_chunk(a, 0.9, position=2), # a's best chunk comes last
_chunk(c, 0.5),
]
out = select_suggested(chunks, n=5)
assert [d.path for d in out] == ["a.md", "b.md", "c.md"]
# The rows carry the full content byte-identical (A6: the content is
# what ``read`` serves later — never truncated).
assert out[0].content == "A" * 50
assert TRUNCATION_MARKER not in out[0].content
def test_suggested_dedupes_multiple_chunks_to_one_row() -> None:
"""Multiple hit chunks of one document collapse to a single row."""
a = _doc("a.md", "A" * 50)
b = _doc("b.md", "B" * 50)
chunks = [
_chunk(a, 0.2),
_chunk(b, 0.7),
_chunk(a, 0.9, position=2),
_chunk(a, 0.5),
]
out = select_suggested(chunks, n=5)
assert [d.path for d in out] == ["a.md", "b.md"] # one row per document
assert out[0] is a
def test_suggested_caps_at_n_in_rank_order() -> None:
docs_in = [_doc(f"d{i}.md", "X" * 20) for i in range(7)]
chunks = [_chunk(d, 0.9 - 0.1 * i) for i, d in enumerate(docs_in)]
out = select_suggested(chunks, n=3)
assert [d.path for d in out] == ["d0.md", "d1.md", "d2.md"]
def test_suggested_default_cap_is_the_settings_value(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""``n`` omitted → ``BOR_SUGGESTED_DOCS`` caps the walk — default 5
(LOCKED A3, the top-5 "start here" directive, TODO L3) — and the cap
is the setting's LIVE value, not a frozen constant."""
docs_in = [_doc(f"d{i}.md", "X" * 20) for i in range(7)]
chunks = [_chunk(d, 0.9 - 0.1 * i) for i, d in enumerate(docs_in)]
settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
assert settings.suggested_docs == 5 # the production default
monkeypatch.setattr(retriever, "get_settings", lambda: settings)
assert [d.path for d in select_suggested(chunks)] == [
f"d{i}.md" for i in range(5)
]
small = Settings(_env_file=None, suggested_docs=2) # pyright: ignore[reportCallIssue]
monkeypatch.setattr(retriever, "get_settings", lambda: small)
assert [d.path for d in select_suggested(chunks)] == ["d0.md", "d1.md"]
def test_suggested_never_filters_on_cosine_floor() -> None:
"""NO floor (LOCKED A3): a lexical-only hit (cosine 0.0 by
construction) is a suggestion when it ranks — the contrast pin
against ``select_documents_tiered``'s floored cited tier on the SAME
input, which demotes it to the related tier."""
a = _doc("a.md", "A" * 50)
b = _doc("b.md", "B" * 50)
lexical_only = RetrievedChunk(
chunk_id=uuid.uuid4(),
position=0,
content="X" * 10,
score=0.9, # top fused rank (the FTS hit)
document=a,
cosine=0.0, # no vector rank — lexical-only
fts_hit=True,
)
chunks = [lexical_only, _cos_chunk(b, 0.8, 0.5)]
# Suggested: the floor never filters — a leads, b follows in rank order.
assert [d.path for d in select_suggested(chunks, n=5)] == ["a.md", "b.md"]
# Contrast: the same input through the phase-113 cited tier — the
# 0.35 usefulness bar demotes the lexical-only doc to related.
cited, related = select_documents_tiered(chunks, n=2, floor=0.35, related_cap=2)
assert [d.path for d in cited] == ["b.md"]
assert [d.path for d in related] == ["a.md"]
def test_suggested_tie_break_inherited_from_fused_order() -> None:
"""Equal fused scores keep the input (fused) order — the stable
score-only walk inherits ``fuse()``'s (-score, -cosine, path,
position) tie-break; the selector never re-sorts it away."""
a = _doc("a.md", "A" * 50)
b = _doc("b.md", "B" * 50)
chunks = [_cos_chunk(a, 0.7, 0.5), _cos_chunk(b, 0.7, 0.4)] # a wins on cosine
assert [d.path for d in select_suggested(chunks, n=5)] == ["a.md", "b.md"]
# A FULL tie (score AND cosine): the input position — ``fuse()``'s
# path/position tie-break already applied — decides. ``m.md`` sorts
# AFTER ``b2.md`` alphabetically, so any re-sort by path would flip
# the order; the fused input order must win.
m = _doc("m.md", "M" * 50)
b2 = _doc("b2.md", "B" * 50)
chunks = [_cos_chunk(m, 0.7, 0.4), _cos_chunk(b2, 0.7, 0.4)]
assert [d.path for d in select_suggested(chunks, n=5)] == ["m.md", "b2.md"]
def test_suggested_empty_chunks_yield_no_documents() -> None:
assert select_suggested([], n=5) == []
# ---------------------------------------------------------------------------
# Phase 118, task 05 — select_related: the rank-6+ tier after the suggested set
# ---------------------------------------------------------------------------
def test_related_walk_order_after_excluded_set() -> None:
"""The SAME stable score-descending walk as ``select_suggested``:
a document's rank is fixed by its FIRST seen chunk; documents in
*excluded_ids* (the suggested set) are skipped and the rest come
back in rank order."""
a = _doc("a.md", "A" * 50)
b = _doc("b.md", "B" * 50)
c = _doc("c.md", "C" * 50)
d = _doc("d.md", "D" * 50)
chunks = [
_chunk(a, 0.4, position=0), # a's weak chunk comes first
_chunk(b, 0.8),
_chunk(a, 0.9, position=2), # a's best chunk last — a ranks first
_chunk(c, 0.5),
_chunk(d, 0.3),
]
out = select_related(chunks, {a.id, b.id}, cap=2)
assert [x.path for x in out] == ["c.md", "d.md"]
def test_related_skips_excluded_documents() -> None:
"""Every document in *excluded_ids* is skipped, even when it would
rank inside the cap — the suggested set never rides the related row."""
a = _doc("a.md", "A" * 50)
b = _doc("b.md", "B" * 50)
c = _doc("c.md", "C" * 50)
chunks = [_chunk(a, 0.9), _chunk(b, 0.8), _chunk(c, 0.5)]
out = select_related(chunks, {a.id, b.id}, cap=5)
assert out == [c]
# Every doc excluded → empty, even with room left in the cap.
assert select_related(chunks, {a.id, b.id, c.id}, cap=5) == []
def test_related_caps_at_cap_in_rank_order() -> None:
"""The phase-118 turn wiring on 9 docs: suggested = the top 5,
related = rank 6–7 (capped at 2), disjoint from the suggested set."""
docs_in = [_doc(f"d{i}.md", "X" * 20) for i in range(9)]
chunks = [_chunk(d, 0.9 - 0.1 * i) for i, d in enumerate(docs_in)]
suggested = select_suggested(chunks, n=5)
out = select_related(chunks, {d.id for d in suggested}, cap=2)
assert [x.path for x in out] == ["d5.md", "d6.md"] # rank 6–7, capped
suggested_paths = {d.path for d in suggested}
assert suggested_paths.isdisjoint({x.path for x in out})
def test_related_cap_zero_yields_empty() -> None:
"""cap=0 is the kill switch (related_max_docs=0): no related docs,
the pre-phase-113 visibility."""
a = _doc("a.md", "A" * 50)
b = _doc("b.md", "B" * 50)
c = _doc("c.md", "C" * 50)
chunks = [_chunk(a, 0.9), _chunk(b, 0.8), _chunk(c, 0.5)]
assert select_related(chunks, {a.id}, cap=0) == []
def test_related_never_filters_on_cosine_floor() -> None:
"""NO floor: a lexical-only (cosine 0.0) doc is related when it
ranks after the excluded set — the related tier is visibility, not
citation (phase 118 applies no cosine floor to it)."""
a = _doc("a.md", "A" * 50)
b = _doc("b.md", "B" * 50)
lexical_only = RetrievedChunk(
chunk_id=uuid.uuid4(),
position=0,
content="X" * 10,
score=0.9, # top fused rank (the FTS hit)
document=a,
cosine=0.0, # no vector rank — lexical-only
fts_hit=True,
)
chunks = [lexical_only, _cos_chunk(b, 0.8, 0.5)]
out = select_related(chunks, set(), cap=5)
assert [x.path for x in out] == ["a.md", "b.md"]
def test_related_dedupes_multiple_chunks_to_one_row() -> None:
"""Multiple hit chunks of one document collapse to a single row
(first-seen-chunk rank, dedupe by document.id — the shared walk)."""
a = _doc("a.md", "A" * 50)
b = _doc("b.md", "B" * 50)
chunks = [_chunk(a, 0.2), _chunk(b, 0.7), _chunk(a, 0.9, position=2)]
out = select_related(chunks, set(), cap=5)
assert [x.path for x in out] == ["a.md", "b.md"] # one row per document
assert out[0] is a
def test_related_tie_break_inherited_from_fused_order() -> None:
"""Equal fused scores keep the input (fused) order — the stable
score-only walk inherits ``fuse()``'s (-score, -cosine, path,
position) tie-break; the selector never re-sorts it away."""
a = _doc("a.md", "A" * 50)
b = _doc("b.md", "B" * 50)
chunks = [_cos_chunk(a, 0.7, 0.5), _cos_chunk(b, 0.7, 0.4)] # a wins on cosine
assert [x.path for x in select_related(chunks, set(), cap=5)] == ["a.md", "b.md"]
m = _doc("m.md", "M" * 50)
b2 = _doc("b2.md", "B" * 50)
chunks = [_cos_chunk(m, 0.7, 0.4), _cos_chunk(b2, 0.7, 0.4)] # full tie
assert [x.path for x in select_related(chunks, set(), cap=5)] == ["m.md", "b2.md"]
def test_related_empty_chunks_yield_no_documents() -> None:
assert select_related([], set(), cap=5) == []
# ---------------------------------------------------------------------------
# Hybrid retrieval (A7): RRF fusion + lexical tsquery
# ---------------------------------------------------------------------------
+22 -1
View File
@@ -331,7 +331,16 @@ def test_extra_keys_still_forbidden() -> None:
def test_minimal_message_still_validates() -> None:
"""Optional keys may be ABSENT exactly as pre-phase-83."""
msg = ChatMessage.model_validate({"who": "user", "text": "hi"})
assert (msg.sources, msg.deflected, msg.suggestions, msg.thinking, msg.tools, msg.stopped) == (
assert (
msg.sources,
msg.related,
msg.deflected,
msg.suggestions,
msg.thinking,
msg.tools,
msg.stopped,
) == (
None,
None,
None,
None,
@@ -358,6 +367,7 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
"who": "user",
"text": "How did I install k3s on the new node?",
"sources": None,
"related": None,
"deflected": None,
"suggestions": None,
"thinking": None,
@@ -371,6 +381,14 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
{"source": "Homelab", "path": "kubernetes.md", "title": "Kubernetes Cluster"},
{"source": "Deployments", "path": "k3s-install.md", "title": "k3s Install Notes"},
],
# Phase 113 related-doc tier — an ACCEPTED key (the phase-113
# omission of this field made extra="forbid" 422 every
# done-time auto-save carrying it, so grounded turns' brain
# messages never persisted — the A2 quiet failure swallowed
# the 422). Round-trips like sources.
"related": [
{"source": "Homelab", "path": "traefik.md", "title": "Traefik Notes"}
],
"deflected": False,
"suggestions": None,
"thinking": "The kubernetes doc covers the cluster layout…",
@@ -398,6 +416,7 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
"who": "user",
"text": "And what ports does Traefik expose?",
"sources": None,
"related": None,
"deflected": None,
"suggestions": None,
"thinking": None,
@@ -408,6 +427,7 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
"who": "brain",
"text": "Traefik exposes 80/443 on every node.",
"sources": None,
"related": None,
"deflected": None,
"suggestions": ["What is the Traefik dashboard password?"],
"thinking": None,
@@ -433,6 +453,7 @@ def test_realistic_payload_round_trips_through_update_model() -> None:
"who": "brain",
"text": "answer",
"sources": [_source_ref()],
"related": None,
"deflected": None,
"suggestions": ["follow-up?"],
"thinking": "scratchpad",
+121 -94
View File
@@ -1,13 +1,15 @@
"""Unit: the phase-113 source-chip-quality contract (TODO L5 + L2c —
"the 2nd chip is often noise the answer never used").
Phase 113 demotes sub-floor hits out of the citation surface: the
done frame carries the cited tier in ``sources`` (rendered by
``appendSources`` as ``.source-chip`` pills, UNCHANGED) and the
related tier in ``related`` (rendered by the NEW ``appendRelated`` as
the de-emphasized labeled row — ``.related-doc`` links, never
``.source-chip``). A deflected turn carries ``sources: []`` → zero
chips; its weak hits live in the related row only.
Phase 118 re-tiers the same frame (LOCKED A3/A4): ``sources`` carries
the suggested tier (top-5, NO floor) + the agent-read docs (deduped)
— rendered by ``appendSources`` as ``.source-chip`` pills, UNCHANGED —
and ``related`` carries rank 6+ after the suggested set (rendered by
``appendRelated`` as the de-emphasized labeled row — ``.related-doc``
links, never ``.source-chip``). A deflected turn carries
``sources: []`` → zero chips; its weak hits are the suggested tier
(the durable record), and with ≤5 retrieved docs the related row is
empty.
This module pins the STATIC SOURCES the UI contract stands on, in the
house source-pin pattern (the test_chip_sizing_question_cap.py
@@ -26,24 +28,23 @@ house source-pin pattern (the test_chip_sizing_question_cap.py
1. **both docs weak** ("What is the capital of Mongolia?" →
``Trooper_Nagraz.pl`` + ``Trooper_Begzei.pl``, both unrelated) —
cited tier empty, the weak hits ride the related tier, capped at
``related_max_docs``; the FTS hit without vector corroboration
stays LOW (the A8-revised "Mongolia" case);
the weak hits are SUGGESTED (no floor, A3; the durable record),
nothing left for the related tier; the FTS hit without vector
corroboration stays LOW (the A8-revised "Mongolia" case);
2. **one strong + one weak** (the phase-gate question answered from
``brain-of-reese/.agents/validate.sh``; the 2nd chip
``ServMon/README.md`` unused) — exactly ONE cited ref, the weak
doc in ``related``;
``brain-of-reese/.agents/validate.sh``; ``ServMon/README.md``
alongside) — BOTH suggested (no floor) ⇒ both cited refs (A4),
no related tier;
3. **the Nagraz case** (``Trooper_Nagraz.pl`` strong,
``Trooper_Byzin.pl`` weak — same shape, different fixtures);
4. **the meta/history question** (no doc clears the bar, the agent
reads nothing — chips ``app/api/suggestions.py`` +
``108_history_wire_check/00_phase.md``, neither used) — pinned on
the DONE FRAME (endpoint-level, fake retriever/LLM/session): the
frame is row-only — ``sources: []`` (the UI's chip list — zero
chips) + the weak hits in ``related``;
5. **the agent-read exemption** (LOCKED A2): a below-floor doc the
agent ``read`` via the tool joins ``sources`` (cited, last) and is
excluded from ``related``.
4. **the meta/history question** (the agent reads nothing —
``app/api/suggestions.py`` + ``108_history_wire_check/00_phase.md``
alongside) — pinned on the DONE FRAME (endpoint-level, fake
retriever/LLM/session): ``sources: []`` (zero chips) and an EMPTY
related row (both weak docs are suggested, ≤5 docs retrieved);
5. **the agent-read exemption** (LOCKED A4): a rank-6+ (related-
tier) doc the agent ``read`` via the tool joins ``sources``
(cited, last) and is excluded from ``related``.
The browser behavior (chip counts on a single-source question, zero
chips on a deflected turn) is E2E-gated by
@@ -411,14 +412,18 @@ def _shape_settings() -> Settings:
"""The PRODUCTION calibration (the code defaults, explicit) — the
four shapes were observed LIVE under this threshold/floor pair.
``_env_file=None`` keeps the mock-calibrated values from
``tests/conftest.py`` (and any local ``.env``) out of the pin."""
``tests/conftest.py`` (and any local ``.env``) out of the pin.
``source_usefulness_floor`` / ``top_n_docs`` are legacy phase-113
settings — phase 118 retired their seeding role (A6; ``plan_turn``
never consults them), they are carried here for completeness."""
return Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
relevance_threshold=0.62,
lexical_support_floor=0.35,
source_usefulness_floor=0.35, # LOCKED A2 default
related_max_docs=2, # LOCKED A4 default
top_n_docs=2, # the ceiling — never a quota (LOCKED A2)
source_usefulness_floor=0.35, # retired by phase 118 (A6) — not consulted
related_max_docs=2, # the rank-6+ row cap (LOCKED A4)
top_n_docs=2, # retired by phase 118 (A6) — not consulted
suggested_docs=5, # the "start here" cap (LOCKED A3)
)
@@ -461,12 +466,12 @@ def _chunk(
def test_shape_1_mongolia_both_docs_weak_cite_nothing() -> None:
"""Observed shape 1 (TODO L110–113): "What is the capital of
Mongolia?" → chips ``Trooper_Nagraz.pl`` + ``Trooper_Begzei.pl``,
BOTH unrelated. Both below the bar: the cited tier is EMPTY (zero
citation chips) and the weak hits ride the related tier — in rank
order, capped at ``related_max_docs`` (the 3rd weak doc drops out).
The FTS hit without vector corroboration (0.20 < the 0.35 lexical
floor) stays LOW — the A8-revised "Mongolia" case; the weak content
never reaches the LLM."""
BOTH unrelated. Phase 118 (A3): the floor never filters — the weak
hits are the SUGGESTED tier (the durable record's input), and with
three retrieved docs nothing is left for the related tier (rank
6+). The FTS hit without vector corroboration (0.20 < the 0.35
lexical floor) stays LOW — the A8-revised "Mongolia" case; the weak
content never reaches the LLM (titles only)."""
nagraz = _doc("scripts", "scripts/Trooper_Nagraz.pl", "Trooper_Nagraz.pl",
"NAGRAZ_PL_CONTENT")
begzei = _doc("scripts", "scripts/Trooper_Begzei.pl", "Trooper_Begzei.pl",
@@ -476,64 +481,76 @@ def test_shape_1_mongolia_both_docs_weak_cite_nothing() -> None:
chunks = [
_chunk(nagraz, 0.033, cosine=0.20, fts_hit=True), # rank 1, lexical hit
_chunk(begzei, 0.031, cosine=0.12),
_chunk(third, 0.030, cosine=0.10), # below the cap — related drops it
_chunk(third, 0.030, cosine=0.10),
]
plan = chat_api.plan_turn(chunks, _shape_settings())
assert plan.deflected is True # 0.20 < 0.62 AND 0.20 < the 0.35 lex floor
assert plan.docs == [] # NO citation slot below the bar
assert [d.title for d in plan.related_docs] == [
# No floor (A3): the weak hits are suggested, in rank order (≤5).
assert [d.title for d in plan.suggested_docs] == [
"Trooper_Nagraz.pl",
"Trooper_Begzei.pl",
] # rank order, capped at related_max_docs (2)
assert len(plan.related_docs) <= 2
"Trooper_Third.pl",
]
assert plan.related_docs == [] # no rank-6+ doc among 3 retrieved
# The LOW prompt is titles only — none of the weak content is sent.
assert "NAGRAZ_PL_CONTENT" not in plan.system_prompt
assert "Trooper_Nagraz.pl" in plan.system_prompt # weak-hit titles carried
assert plan.suggestions # the "Maybe try" chips are unchanged
def test_shape_2_validate_sh_strong_plus_unused_second_chip() -> None:
def test_shape_2_validate_sh_strong_plus_weak_second_suggested() -> None:
"""Observed shape 2 (TODO L114–116): the phase-gate question is
answered from ``brain-of-reese/.agents/validate.sh`` — the 2nd chip
``ServMon/README.md`` was NEVER used. The strong doc clears the bar
and takes the only cited slot (top_n_docs is a ceiling, not a
quota); the weak 2nd doc demotes to related — never a citation.
The HIGH prompt carries the cited content only."""
answered from ``brain-of-reese/.agents/validate.sh`` with
``ServMon/README.md`` retrieved alongside (weak cosine). Phase 118
(A3): the floor never filters — the weak 2nd doc is SUGGESTED too
(both docs seed the HIGH prompt as summaries; the A5 fallback
carries the short fixture content whole), and A4 makes both
citation refs on the done frame — the "unused 2nd chip" is the
phase-113 shape, retired by the owner directive."""
validate = _doc("brain-of-reese", ".agents/validate.sh", "validate.sh",
"VALIDATE_SH_CONTENT")
servmon = _doc("ServMon", "README.md", "ServMon README",
"SERVMON_README_CONTENT")
chunks = [
_chunk(validate, 0.90, cosine=0.70), # clears threshold AND bar
_chunk(validate, 0.90, cosine=0.70), # clears the threshold
_chunk(servmon, 0.80, cosine=0.20), # high fused rank, weak cosine
]
plan = chat_api.plan_turn(chunks, _shape_settings())
assert plan.deflected is False # 0.70 >= 0.62
assert [d.title for d in plan.docs] == ["validate.sh"] # exactly ONE cited
assert [d.title for d in plan.related_docs] == ["ServMon README"]
assert "VALIDATE_SH_CONTENT" in plan.system_prompt
assert "SERVMON_README_CONTENT" not in plan.system_prompt
assert [d.title for d in plan.suggested_docs] == [
"validate.sh",
"ServMon README",
] # both suggested (no floor), rank order
assert plan.related_docs == [] # nothing beyond rank 2 for 2 docs
assert "VALIDATE_SH_CONTENT" in plan.system_prompt # A5 preview fallback
assert "SERVMON_README_CONTENT" in plan.system_prompt # ditto
def test_shape_3_nagraz_answered_by_own_doc_byzin_uncited() -> None:
def test_shape_3_nagraz_answered_by_own_doc_byzin_suggested() -> None:
"""Observed shape 3 (TODO L117–119): the Trooper_Nagraz question is
answered from ``Trooper_Nagraz.pl`` — the 2nd chip
``Trooper_Byzin.pl`` uncited. The SAME shape as shape 2 with
different fixtures — the bar filters the 2nd chip; it is not a
coincidence of the validate.sh pair."""
answered from ``Trooper_Nagraz.pl`` with ``Trooper_Byzin.pl``
retrieved alongside (weak cosine). The SAME shape as shape 2 with
different fixtures — phase 118's no-floor tiering suggests BOTH
(the phase-113 "bar filters the 2nd chip" story is retired); the
HIGH prompt seeds both summaries (A5 fallback for the short
fixture content)."""
nagraz = _doc("scripts", "scripts/Trooper_Nagraz.pl", "Trooper_Nagraz.pl",
"NAGRAZ_PL_CONTENT")
byzin = _doc("scripts", "scripts/Trooper_Byzin.pl", "Trooper_Byzin.pl",
"BYZIN_PL_CONTENT")
chunks = [
_chunk(nagraz, 0.85, cosine=0.70),
_chunk(byzin, 0.75, cosine=0.15), # below the bar
_chunk(byzin, 0.75, cosine=0.15), # weak cosine — still suggested (A3)
]
plan = chat_api.plan_turn(chunks, _shape_settings())
assert plan.deflected is False
assert [d.title for d in plan.docs] == ["Trooper_Nagraz.pl"] # 1 cited
assert [d.title for d in plan.related_docs] == ["Trooper_Byzin.pl"] # 1 related
assert "BYZIN_PL_CONTENT" not in plan.system_prompt
assert [d.title for d in plan.suggested_docs] == [
"Trooper_Nagraz.pl",
"Trooper_Byzin.pl",
] # both suggested (no floor), rank order
assert plan.related_docs == []
assert "NAGRAZ_PL_CONTENT" in plan.system_prompt # A5 preview fallback
assert "BYZIN_PL_CONTENT" in plan.system_prompt # ditto
# ---------- task 03: done-frame wire (endpoint-level fakes, no stack) ----------
@@ -664,21 +681,22 @@ def _fake_retriever(chunks: list[RetrievedChunk]) -> Any:
return retrieve
def test_shape_4_meta_question_deflected_frame_is_row_only(
def test_shape_4_meta_question_deflected_frame_has_no_chips_or_row(
client: TestClient,
chip_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Observed shape 4 (TODO L120–123), pinned on the DONE FRAME: a
meta question about the conversation's own history → chips
meta question about the conversation's own history → weak hits
``app/api/suggestions.py`` + ``108_history_wire_check/00_phase.md``,
neither used. No doc clears the bar and the agent reads nothing —
the frame is ROW-ONLY: ``sources: []`` (the UI chips every source
entry — zero chips) with the weak hits in ``related`` (rank order,
≤ ``related_max_docs``) — the de-emphasized row's links (the row's
rendering itself is pinned by task 02's source tests + the E2E).
The weak retrieval stays durably recorded (LOCKED A3); the weak
content never reaches the LLM (LOW prompt, titles only)."""
neither used. Phase 118: the agent reads nothing, the weak hits
are the SUGGESTED tier (no floor, A3 — the durable record's input)
and, with only two retrieved docs, nothing reaches rank 6+ — the
frame carries ``sources: []`` (zero chips — a deflected answer
cites nothing) AND an empty ``related`` row (the row's rendering
itself is pinned by task 02's source tests + the E2E). The weak
retrieval stays durably recorded (LOCKED A3); the weak content
never reaches the LLM (LOW prompt, titles only)."""
session, llm = chip_env
suggestions = _doc("brain-of-reese", "app/api/suggestions.py",
"suggestions.py", "SUGGESTIONS_PY_CONTENT")
@@ -701,13 +719,9 @@ def test_shape_4_meta_question_deflected_frame_is_row_only(
assert done["type"] == "done"
assert done["deflected"] is True
assert done["sources"] == [] # zero citation chips on the wire
related = done["related"]
assert [(s["source"], s["path"]) for s in related] == [
("brain-of-reese", "app/api/suggestions.py"),
("brain-of-reese", ".agents/108_history_wire_check/00_phase.md"),
] # rank order
assert len(related) <= 2 # related_max_docs
assert all(s["title"] for s in related) # the row's links carry the identity
# Phase 118 (A3): both weak docs are suggested (≤5, no floor) —
# nothing reaches rank 6+, so the related row is empty.
assert done["related"] == []
assert done["suggestions"] # the "Maybe try" chips are unchanged
(system, _user) = llm.seen[0][0], llm.seen[0][1]
@@ -722,16 +736,17 @@ def test_shape_4_meta_question_deflected_frame_is_row_only(
assert "108_history_wire_check/00_phase.md" in row.sources
def test_done_frame_single_cited_ref_strong_plus_weak(
def test_done_frame_carries_suggested_refs_strong_plus_weak(
client: TestClient,
chip_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Shape 2 on the wire — the single-document question's input to
"exactly one citation chip" (the E2E asserts the rendered chip):
the bar-clearing doc is the ONLY ``sources`` ref; the weak 2nd doc
rides ``related``; the tiers are disjoint (the done frame's dedupe).
The durable record keeps the FULL retrieval (LOCKED A3)."""
"""Shape 2 on the wire under phase 118 (LOCKED A4): the citation
surface is the suggested tier + the agent's reads (deduped) —
with two retrieved docs and no read, BOTH docs are ``sources``
refs (no floor — A3); nothing reaches rank 6+, so ``related`` is
empty; the tiers stay disjoint (the done frame's dedupe). The
durable record keeps the FULL retrieval (LOCKED A3)."""
session, _llm = chip_env
validate = _doc("brain-of-reese", ".agents/validate.sh", "validate.sh",
"VALIDATE_SH_CONTENT")
@@ -753,10 +768,9 @@ def test_done_frame_single_cited_ref_strong_plus_weak(
assert done["deflected"] is False
assert [(s["source"], s["path"]) for s in done["sources"]] == [
("brain-of-reese", ".agents/validate.sh"),
] # EXACTLY one citation chip on the wire
assert [(s["source"], s["path"]) for s in done["related"]] == [
("ServMon", "README.md"),
("ServMon", "README.md"), # A4: suggested + read — both suggested (A3)
]
assert done["related"] == [] # nothing reaches rank 6+ for 2 docs
cited = {(s["source"], s["path"]) for s in done["sources"]}
related = {(s["source"], s["path"]) for s in done["related"]}
assert cited.isdisjoint(related)
@@ -769,20 +783,25 @@ def test_done_frame_single_cited_ref_strong_plus_weak(
assert "ServMon/README.md" in row.sources
def test_agent_read_below_floor_doc_joins_sources(
def test_agent_read_related_doc_is_cited_not_related(
client: TestClient,
chip_env: tuple[_FakeSession, _CannedLLM],
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The agent-read exemption (LOCKED A2): a doc UNDER the bar that
the agent ``read`` via the tool is cited by definition — the model
read it, so it was used. It joins ``sources`` (after the retrieved
cited docs, deduped) and is EXCLUDED from ``related`` (a used doc
must never read as "nearby"); the other below-floor doc stays in
the tier. The read content reached the model (the tool result in
the follow-up request)."""
"""The agent-read exemption (LOCKED A4, phase-118 tiering): a
rank-6+ doc — the related tier ("nearby docs") — that the agent
``read`` via the tool is cited by definition: the model read it, so
it was used. It joins ``sources`` (after the suggested docs — it
was not suggested, so the read appends it last, deduped) and is
EXCLUDED from ``related`` (a used doc must never read as "nearby");
the other rank-6+ doc stays in the tier. The read content reached
the model (the tool result in the follow-up request)."""
session, _default_llm = chip_env
strong = _doc("docs", "strong.md", "Strong", "STRONG_DOC_CONTENT")
fillers = [
_doc("docs", f"filler{i}.md", f"Filler {i}", f"FILLER_{i}_CONTENT")
for i in range(1, 5) # ranks 2–5 — fill the suggested tier
]
weak_b = _doc("docs", "weak-b.md", "Weak B", "WEAK_B_READ_BY_AGENT")
weak_c = _doc("docs", "weak-c.md", "Weak C", "WEAK_C_CONTENT")
monkeypatch.setattr(
@@ -790,9 +809,13 @@ def test_agent_read_below_floor_doc_joins_sources(
"retrieve",
_fake_retriever(
[
_chunk(strong, 0.90, cosine=0.70), # clears the bar
_chunk(weak_b, 0.80, cosine=0.20), # below the bar — read by the agent
_chunk(weak_c, 0.70, cosine=0.10), # below the bar — nobody reads it
_chunk(strong, 0.90, cosine=0.70), # clears the threshold (rank 1)
_chunk(fillers[0], 0.85, cosine=0.30), # ranks 2–5: suggested
_chunk(fillers[1], 0.80, cosine=0.30),
_chunk(fillers[2], 0.75, cosine=0.30),
_chunk(fillers[3], 0.72, cosine=0.30),
_chunk(weak_b, 0.70, cosine=0.20), # rank 6 — related; read by the agent
_chunk(weak_c, 0.65, cosine=0.10), # rank 7 — related; nobody reads it
]
),
)
@@ -815,7 +838,10 @@ def test_agent_read_below_floor_doc_joins_sources(
assert "WEAK_B_READ_BY_AGENT" in tool_msgs[0]["content"]
sources = [(s["source"], s["path"]) for s in done["sources"]]
assert sources == [("docs", "strong.md"), ("docs", "weak-b.md")] # read ⇒ cited, last
# A4: suggested (5) + the read doc (last — it was not suggested).
assert sources[-1] == ("docs", "weak-b.md") # read ⇒ cited, last
assert len(sources) == 6
assert ("docs", "weak-c.md") not in sources # never suggested, never read
related = [(s["source"], s["path"]) for s in done["related"]]
assert related == [("docs", "weak-c.md")] # the read doc is not "nearby"
assert set(sources).isdisjoint(set(related))
@@ -823,3 +849,4 @@ def test_agent_read_below_floor_doc_joins_sources(
(row,) = session.added
assert isinstance(row, QueryLog)
assert "weak-b.md" in row.sources # the full retrieval is recorded (A3)
assert "weak-c.md" in row.sources # … rank 6+ included (suggested + related + read)