phase: 118_summary_seed_context
**Phase 118 final verification pass — complete.** All criteria verified; 4 pre-existing defects found and fixed.
- **Verified:** summary-seed wiring (`select_suggested` top-5 no-floor → summary blocks, no full text in HIGH prompt), all-doc markdown summaries + NULL backfill (`summary_backfilled`, no `sources_meta` bump), `read` adds full text with `read_docs`-only dedupe, `done.sources` = suggested+read / durable record = suggested+related+read + `suggested=N` log line (seen live in E2E), byte-locked PERSONA/LOW/TOOLS_SECTION, battery gate PASS recorded in `TOOL_CALLING_TESTING.md` §10 (turbo 2026-09-16: 1/2/4 GREEN, cond-3 reported 9/10 per A7, contract 21/21, caps 0).
- **Defects fixed (all pre-existing, none phase-118):** ① `ChatMessage` schema missing the phase-113 `related` key → `extra="forbid"` 422'd every done-time auto-save of grounded turns with a related tier, leaving `message_count=1` (root cause of `test_share_chat` 3F; browser-level instrumentation proved the PUT 422) — added the field + unit/integration pins; ② `test_theme_semantic_completion` pins stale vs phase-117 debox (border/chip removed) — re-targeted to assert border/chip *absence*; ③ `test_header_consistency` `<26`px pin red on 26.125px native date-input line — bound relaxed to `<34` (wrap-detection intent kept); ④ `test_navbar_refresh` bor.chat.v1 key set updated for `related`.
- **Test/lint/coverage:** `uv run pytest --cov=app --cov-report=term-missing` → **2506 passed, app/ 99%** (>90%); `uv run ruff check . && uv run pyright` → clean, 0 errors.
- **E2E:** new story suite in isolation → **2 passed**; full 103-suite matrix sweep (each isolated) → **all 103 green** after the fixes; `test_share_chat` 4 passed, `test_theme_semantic_completion` 8 passed, `test_header_consistency` 3 passed, `test_navbar_refresh` 7 passed.
- **Deviations:** none from LOCKED decisions. Note: orphaned diagnostic uvicorn processes briefly made E2E sessions exercise stale code — killed and re-verified; a sweep-regenerated tracked screenshot was restored. No commits made (harness commits).
- **Completion criteria:** all 7 ✅ (commit/phase-move is the harness's step).
- **Next pending phase:** none — `todo/` holds only this phase's overview pending the harness move.
This commit is contained in:
+169
-66
@@ -221,50 +221,71 @@ def test_agent_tools_names_and_parameters() -> None:
|
||||
"'homelab/active'). Omit it to list every source."
|
||||
)
|
||||
read = by_name["read"]["function"]
|
||||
# Tool-calling fast loop (2026-09-04, controlled fixture gate):
|
||||
# the do-not-read rule is FRONT-LOADED — the controlled gate's
|
||||
# telemetry showed the `lite` model obeying the user's "open it /
|
||||
# read it" and reading seed-context documents the <documents>
|
||||
# section already carries (every refusal of a 12-call run was
|
||||
# ALREADY_IN_CONTEXT); the rule now leads the description instead
|
||||
# of sitting mid-paragraph, and the tool is framed as "only for
|
||||
# documents NOT already in <documents>". Phase 95 (task 01): the
|
||||
# read-truncation sentence is inserted before the one-call-at-a-
|
||||
# time discipline clause (the discipline rule stays last, as in the
|
||||
# other two tools) — a capped read carries the TRUNCATED notice and
|
||||
# the `grep` follow-up (the pinned copy).
|
||||
# Phase 118 (task 04, A6): the read description is rewritten for
|
||||
# the summary-seed mode — the <documents> section shows the
|
||||
# top-ranked documents' SUMMARIES (their full texts are NOT in
|
||||
# the prompt yet); read adds one of them (or any other document)
|
||||
# to the context; only an already-read document is refused. The
|
||||
# combined-identity handoff to the `ls` output / the <documents>
|
||||
# blocks stays; the truncation-notice paragraph (phase 95) and
|
||||
# the one-call-at-a-time discipline clause (task 05) survive the
|
||||
# rewrite byte-identical.
|
||||
assert read["description"] == (
|
||||
"Do not call this tool for a document already shown in "
|
||||
"the <documents> section, even when the user asks you to "
|
||||
"open or read it — its full text is already in your "
|
||||
"prompt; answer directly from it. Use it only to add a "
|
||||
"document NOT already in <documents> to your context, "
|
||||
"by its combined `source/path` string. Very large "
|
||||
"documents are truncated: you receive the first part "
|
||||
"plus a TRUNCATED notice naming how many more characters "
|
||||
"exist — the notice is authoritative, the document did "
|
||||
"NOT end where it stopped. Follow it and use `grep` "
|
||||
"(pattern) to locate the rest — it searches the whole "
|
||||
"document. Call one tool at a time — wait for this "
|
||||
"result before your next call."
|
||||
"The <documents> section shows the SUMMARIES of the "
|
||||
"top-ranked documents — their full texts are NOT in "
|
||||
"your prompt yet. Use this tool to add one of them (or "
|
||||
"any other document) to your context, by its combined "
|
||||
"`source/path` string, exactly as shown in the `ls` "
|
||||
"output or the <documents> blocks. Do not re-read a "
|
||||
"document you have already read — its full text is "
|
||||
"already in your prompt. Very large documents are "
|
||||
"truncated: you receive the first part plus a TRUNCATED "
|
||||
"notice naming how many more characters exist — the "
|
||||
"notice is authoritative, the document did NOT end "
|
||||
"where it stopped. Follow it and use `grep` (pattern) "
|
||||
"to locate the rest — it searches the whole document. "
|
||||
"Call one tool at a time — wait for this result before "
|
||||
"your next call."
|
||||
)
|
||||
# The byte-preserved contracts inside the rewrite, pinned as
|
||||
# substrings: the combined source/path identity, the
|
||||
# truncation-notice paragraph, the one-call-at-a-time
|
||||
# discipline sentence.
|
||||
assert (
|
||||
"by its combined `source/path` string, exactly as shown in "
|
||||
"the `ls` output or the <documents> blocks"
|
||||
) in read["description"]
|
||||
assert (
|
||||
"Very large documents are truncated: you receive the first "
|
||||
"part plus a TRUNCATED notice naming how many more "
|
||||
"characters exist — the notice is authoritative, the "
|
||||
"document did NOT end where it stopped. Follow it and use "
|
||||
"`grep` (pattern) to locate the rest — it searches the "
|
||||
"whole document."
|
||||
) in read["description"]
|
||||
assert (
|
||||
"Call one tool at a time — wait for this result before "
|
||||
"your next call."
|
||||
) in read["description"]
|
||||
read_params = read["parameters"]
|
||||
assert read_params["type"] == "object"
|
||||
assert read_params["required"] == ["path"]
|
||||
assert set(read_params["properties"]) == {"path"}
|
||||
assert read_params["properties"]["path"]["type"] == "string"
|
||||
# The combined source/path string is the canonical document identity
|
||||
# (phase 70) — the description pins it with a worked example. Phase
|
||||
# 72 (task 02): the bare-path contract is stated up front; task 05
|
||||
# (live gate iteration 1): the do-not-re-read clause (the dedupe
|
||||
# refusal's prevention at the prompt).
|
||||
# (phase 70) — the description pins it with a worked example; the
|
||||
# bare-path contract is stated up front (phase 72, task 02). Phase
|
||||
# 118 (task 04, A6): the trailing clause is the do-not-RE-READ
|
||||
# teaching — the seeds are summary blocks, not full text, so a
|
||||
# first read of a suggested document succeeds and only an
|
||||
# already-read document is refused.
|
||||
assert read_params["properties"]["path"]["description"] == (
|
||||
"The document to add to your context, as the combined "
|
||||
"`source/path` string exactly as shown in the `ls` output (e.g. "
|
||||
"'homelab/active/container_caddy/caddy.md'). A bare document "
|
||||
"path (without the source name) will not resolve. Only pass a "
|
||||
"document NOT already shown in the <documents> section — it is "
|
||||
"already in your context; do not re-read it."
|
||||
"path (without the source name) will not resolve. Do not "
|
||||
"re-read a document you have already read — its full "
|
||||
"text is already in your prompt."
|
||||
)
|
||||
grep = by_name["grep"]["function"]
|
||||
# Task 05 (live gate iterations 2-6, refined in the 2026-09-03
|
||||
@@ -1455,15 +1476,17 @@ def test_read_bare_filename_without_slash_keeps_no_db_refusal(
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_read_bare_path_of_seed_doc_gets_suggestion_then_dedupe(
|
||||
def test_read_bare_path_of_suggested_doc_gets_suggestion_then_succeeds(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Dedupe precedence: the in-context dedupe fires on the SPLIT pair
|
||||
of the argument — the bare path of an in-context document
|
||||
(``read('app/rag/importer.py')`` with ``sample/app/rag/importer.py``
|
||||
seeded) is NOT that pair, so it is not a dedupe: it gets the
|
||||
suggestion line naming the combined identity, and the model's next,
|
||||
correctly-formed call is then deduped as ALREADY_IN_CONTEXT."""
|
||||
"""Dedupe precedence (phase 118): the in-context dedupe fires on
|
||||
the SPLIT pair of a document already READ — the bare path of a
|
||||
SUGGESTED (seeded) document (``read('app/rag/importer.py')`` with
|
||||
``sample/app/rag/importer.py`` seeded) is NOT that pair, so it is
|
||||
not a dedupe: it gets the suggestion line naming the combined
|
||||
identity, and the model's next, correctly-formed call ADDS the
|
||||
suggested document's full text (A6: the seeds are summary blocks,
|
||||
not full text — a first read of a suggested document succeeds)."""
|
||||
seed = [_doc("sample", "app/rag/importer.py", "Importer", "IMPORTER")]
|
||||
|
||||
def _find(db: Any, source: str, path: str) -> Document | None:
|
||||
@@ -1482,8 +1505,9 @@ def test_read_bare_path_of_seed_doc_gets_suggestion_then_dedupe(
|
||||
],
|
||||
[
|
||||
# Round 2: the corrected call (the suggested combined
|
||||
# identity) — the seed document is already in context, so it
|
||||
# dedupes.
|
||||
# identity) — the suggested document's full text is NOT in
|
||||
# the prompt (only its summary is), so this read succeeds
|
||||
# and adds the full content.
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read",
|
||||
@@ -1493,12 +1517,18 @@ def test_read_bare_path_of_seed_doc_gets_suggestion_then_dedupe(
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0 # both refused
|
||||
# Round 1 refused (the teaching), round 2 executed (A6).
|
||||
assert holder.read_docs == [seed[0]]
|
||||
assert holder.tool_calls == 1
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"No document at 'app/rag/importer.py' — "
|
||||
"did you mean 'sample/app/rag/importer.py'?"
|
||||
)
|
||||
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
assert llm.requests[2][0][5]["content"] == (
|
||||
"Document sample/app/rag/importer.py:\n"
|
||||
"date: 2024-06-15\n"
|
||||
"IMPORTER"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
@@ -1529,16 +1559,23 @@ def test_read_missing_arguments_refused(
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_reading_a_seed_doc_is_already_in_context(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The combined identity of a seeded document: its split pair is in
|
||||
the known set → ALREADY_IN_CONTEXT with no DB lookup (the dedupe
|
||||
check precedes the resolve)."""
|
||||
def test_reading_a_suggested_seed_doc_adds_its_full_text(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 118 (A6): a FIRST read of a suggested (seeded) document
|
||||
SUCCEEDS — the seed is a summary block in the prompt, not the full
|
||||
text, so it falls out of the dedupe set (``holder.read_docs``
|
||||
only): the read goes through the existing path unchanged — the
|
||||
full-content result (header + date line), appended to
|
||||
``holder.read_docs``, counted in ``holder.tool_calls``."""
|
||||
seed = [_doc("Homelab", "kubernetes.md", "Kubernetes", "K8S-CONTENT")]
|
||||
|
||||
def _boom(*_a: Any, **_k: Any) -> None:
|
||||
raise AssertionError("find_document must not be called for a seeded doc")
|
||||
|
||||
monkeypatch.setattr(agent, "find_document", _boom)
|
||||
monkeypatch.setattr(
|
||||
agent,
|
||||
"find_document",
|
||||
lambda db, source, path: seed[0]
|
||||
if (source, path) == ("Homelab", "kubernetes.md")
|
||||
else None,
|
||||
)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
@@ -1551,11 +1588,58 @@ def test_reading_a_seed_doc_is_already_in_context(monkeypatch: pytest.MonkeyPatc
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
|
||||
assert holder.read_docs == [] and holder.tool_calls == 0
|
||||
assert llm.requests[1][0][3]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
assert holder.read_docs == [seed[0]]
|
||||
assert holder.tool_calls == 1
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document Homelab/kubernetes.md:\ndate: 2024-06-15\nK8S-CONTENT"
|
||||
)
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_re_reading_a_suggested_doc_is_deduped(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Phase 118 (A6): once the suggested document's full text has been
|
||||
read into context, a SECOND read of the same combined identity is
|
||||
refused with the byte-identical ``ALREADY_IN_CONTEXT`` line — the
|
||||
dedupe set is ``holder.read_docs`` only, so the refusal fires with
|
||||
NO DB lookup (the dedupe check precedes the resolve) and the
|
||||
counters stay untouched."""
|
||||
seed = [_doc("Homelab", "kubernetes.md", "Kubernetes", "K8S-CONTENT")]
|
||||
lookups: list[tuple[str, str]] = []
|
||||
|
||||
def _find(db: Any, source: str, path: str) -> Document | None:
|
||||
lookups.append((source, path))
|
||||
return seed[0] if (source, path) == ("Homelab", "kubernetes.md") else None
|
||||
|
||||
monkeypatch.setattr(agent, "find_document", _find)
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_1",
|
||||
name="read",
|
||||
arguments={"path": "Homelab/kubernetes.md"},
|
||||
)
|
||||
],
|
||||
[
|
||||
ToolCallPiece(
|
||||
id="call_2",
|
||||
name="read",
|
||||
arguments={"path": "Homelab/kubernetes.md"},
|
||||
)
|
||||
],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(), seed_docs=seed))
|
||||
assert holder.read_docs == [seed[0]] # appended exactly once
|
||||
assert holder.tool_calls == 1 # the re-read counts nothing
|
||||
assert lookups == [("Homelab", "kubernetes.md")] # the re-read deduped pre-lookup
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document Homelab/kubernetes.md:\ndate: 2024-06-15\nK8S-CONTENT"
|
||||
)
|
||||
assert llm.requests[2][0][5]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
# Rejected → the tools are still offered on the next request (the
|
||||
# round cap is the only bound).
|
||||
assert llm.requests[1][1] == AGENT_TOOLS
|
||||
assert llm.requests[2][1] == AGENT_TOOLS
|
||||
|
||||
|
||||
def test_reading_an_already_read_doc_is_deduped(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
@@ -1718,28 +1802,47 @@ def test_run_agent_short_read_yields_no_tool_result_piece(
|
||||
assert isinstance(out[1], StreamPiece)
|
||||
|
||||
|
||||
def test_read_truncation_does_not_touch_refusal_paths(
|
||||
def test_read_truncation_paths_with_suggested_seed(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 95: the read refusal paths are untouched by the cap — a SEED
|
||||
document that is over the cap is still refused with
|
||||
``ALREADY_IN_CONTEXT`` (not truncated, nothing recorded, nothing
|
||||
counted), and an unknown path is still the no-document refusal (no
|
||||
"""Phase 118 (A6) re-target of the phase-95 pins ("seed" →
|
||||
"suggested"): a SUGGESTED (seeded) document that is over the cap is
|
||||
read through the ordinary cap path — cut at the cap + the shared
|
||||
marker + the pinned notice, the truncation recorded on the holder,
|
||||
and the call still successful (the seed is a summary, not full
|
||||
text) — and an unknown path is still the no-document refusal (no
|
||||
content is read, so no truncation either)."""
|
||||
big = "B" * 5000 # far over the tiny cap below
|
||||
seed = _doc("S", "seed.md", "Seed", big)
|
||||
monkeypatch.setattr(agent, "find_document", lambda db, source, path: None)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: [])
|
||||
# (a) Reading the (over-cap) seed doc → ALREADY_IN_CONTEXT (refusal).
|
||||
monkeypatch.setattr(
|
||||
agent,
|
||||
"find_document",
|
||||
lambda db, source, path: seed
|
||||
if (source, path) == ("S", "seed.md")
|
||||
else None,
|
||||
)
|
||||
monkeypatch.setattr(agent, "all_documents", lambda db: [seed])
|
||||
# (a) Reading the (over-cap) SUGGESTED doc → the capped read: cut at
|
||||
# the cap, marker + pinned notice, holder entry, still counted.
|
||||
holder = AgentHolder()
|
||||
llm = ScriptedLLM(
|
||||
[ToolCallPiece(id="call_1", name="read", arguments={"path": "S/seed.md"})],
|
||||
[StreamPiece("content", "ans")],
|
||||
)
|
||||
asyncio.run(_run(llm, holder, _settings(read_max_chars=100), seed_docs=[seed]))
|
||||
assert llm.requests[1][0][3]["content"] == agent.ALREADY_IN_CONTEXT
|
||||
assert holder.read_truncations == []
|
||||
assert holder.tool_calls == 0 and holder.read_docs == []
|
||||
assert llm.requests[1][0][3]["content"] == (
|
||||
"Document S/seed.md:\n"
|
||||
"date: 2024-06-15\n"
|
||||
+ big[:100]
|
||||
+ "\n"
|
||||
+ TRUNCATION_MARKER
|
||||
+ "\n"
|
||||
+ READ_TRUNCATION_NOTICE.format(shown=100, total=5000)
|
||||
)
|
||||
# (argument, cap, total) — the raw argument, the cap kept, the
|
||||
# true length.
|
||||
assert holder.read_truncations == [("S/seed.md", 100, 5000)]
|
||||
assert holder.tool_calls == 1 and holder.read_docs == [seed]
|
||||
# (b) An unknown path → the no-document refusal (argument echoed),
|
||||
# even though a big doc could have truncated — no content is read.
|
||||
holder2 = AgentHolder()
|
||||
|
||||
+264
-120
@@ -8,6 +8,7 @@ session, and the LLM all faked, so the whole deflection contract
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import uuid
|
||||
from collections.abc import Iterator
|
||||
@@ -23,6 +24,7 @@ from app.main import app as fastapi_app
|
||||
from app.models import Document, KbOverview, QueryLog
|
||||
from app.rag.agent import AGENT_TOOLS
|
||||
from app.rag.llm import StreamPiece
|
||||
from app.rag.prompts import build_deflect_prompt
|
||||
from app.rag.retriever import RetrievedChunk, weak_hit_titles
|
||||
from app.rag.suggestions import MAX_SUGGESTIONS, derive_suggestions
|
||||
from tests.conftest import ADMIN_PASSWORD
|
||||
@@ -46,7 +48,9 @@ def _settings(threshold: float = 0.30, floor: float | None = None) -> Settings:
|
||||
)
|
||||
|
||||
|
||||
def _doc(title: str, content: str) -> Document:
|
||||
def _doc(
|
||||
title: str, content: str, summary: str | None = None
|
||||
) -> Document:
|
||||
return Document(
|
||||
id=uuid.uuid4(),
|
||||
source="Homelab",
|
||||
@@ -54,6 +58,10 @@ def _doc(title: str, content: str) -> Document:
|
||||
full_path="/tmp/doc.md",
|
||||
title=title,
|
||||
content=content,
|
||||
# Phase 30/118: the stored lite-model summary — the HIGH block's
|
||||
# BODY (task 03). ``None`` exercises the A5 preview fallback
|
||||
# (the first ``suggestion_preview_chars`` content chars).
|
||||
summary=summary,
|
||||
content_hash="0" * 64,
|
||||
# Phase 106, D5: the HIGH block formats the row's created_at
|
||||
# UTC date part — the detached fixture carries it (the NOT NULL
|
||||
@@ -179,11 +187,12 @@ def test_gate_lexical_only_chunk_does_not_inflate_cosine() -> None:
|
||||
plan = chat_api.plan_turn(chunks, _settings(threshold=0.30))
|
||||
assert plan.top_score == pytest.approx(0.55)
|
||||
assert plan.deflected is False # 0.55 >= 0.30 anyway
|
||||
# Phase 113 (the usefulness bar): Beta's doc ranks first by fused
|
||||
# score, but a lexical-only doc (cosine 0.0 by construction) cannot
|
||||
# clear the bar — it lands in the RELATED tier, never the cited one.
|
||||
assert plan.docs[0].title == "Alpha"
|
||||
assert plan.related_docs[0].title == "Beta"
|
||||
# Phase 118 (A3): the suggestion tier has NO floor — the lexical-
|
||||
# only doc (cosine 0.0 by construction) is SUGGESTED when it ranks.
|
||||
# Rank order is the fused score, so Beta (0.90) leads Alpha (0.50);
|
||||
# with two docs nothing is left for the related tier (rank 6+).
|
||||
assert [d.title for d in plan.suggested_docs] == ["Beta", "Alpha"]
|
||||
assert plan.related_docs == []
|
||||
|
||||
|
||||
# ---------- lexical support floor (A8 revised 2026-09-14) ----------
|
||||
@@ -373,178 +382,225 @@ def test_gate_zero_chunks_deflects_with_fallback_chips() -> None:
|
||||
assert 2 <= len(plan.suggestions) <= MAX_SUGGESTIONS
|
||||
|
||||
|
||||
# ---------- usefulness bar tiering (phase 113, LOCKED A2/A4) ----------
|
||||
# ---------- summary-seed tiering (phase 118, LOCKED A3/A4/A6) ----------
|
||||
|
||||
|
||||
def _bar_settings(
|
||||
def _tier_settings(
|
||||
threshold: float = 0.62,
|
||||
lex_floor: float = 0.35,
|
||||
source_floor: float = 0.35,
|
||||
suggested_cap: int = 5,
|
||||
related_cap: int = 2,
|
||||
top_n: int = 2,
|
||||
) -> Settings:
|
||||
"""Explicit code defaults (production calibration) — the env's mock-
|
||||
calibrated floor (tests/conftest.py) is overridden per test."""
|
||||
calibrated values (tests/conftest.py) are overridden per test.
|
||||
``source_usefulness_floor`` / ``top_n_docs`` are deliberately left
|
||||
at their code defaults: phase 118 retired their seeding role (A6) —
|
||||
``plan_turn`` never consults them (pinned in
|
||||
``test_plan_turn_does_not_consult_retired_seeding_settings``)."""
|
||||
return Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=threshold,
|
||||
lexical_support_floor=lex_floor,
|
||||
source_usefulness_floor=source_floor,
|
||||
suggested_docs=suggested_cap,
|
||||
related_max_docs=related_cap,
|
||||
top_n_docs=top_n,
|
||||
)
|
||||
|
||||
|
||||
def test_plan_turn_high_tiers_strong_plus_weak() -> None:
|
||||
"""Grounded turn: the bar-clearing doc is cited (and in the prompt),
|
||||
the weak 2nd doc loses its citation slot and lands in related_docs —
|
||||
the recurring incident's fix at the plan level."""
|
||||
def _seven_docs_with_summaries() -> list[Document]:
|
||||
"""The 7-doc fixture (rank 1–7 by fused score): every document has a
|
||||
stored SUMMARY (distinct sentinel) and a distinct FULL-CONTENT
|
||||
sentinel that must never reach the prompt (A6: the ``read`` tool is
|
||||
the only full-text path)."""
|
||||
return [
|
||||
_doc(
|
||||
f"Doc {i}",
|
||||
f"FULL_CONTENT_SENTINEL_{i}_SHOULD_NEVER_REACH_THE_PROMPT",
|
||||
summary=f"SUMMARY_TEXT_{i}",
|
||||
)
|
||||
for i in range(7)
|
||||
]
|
||||
|
||||
|
||||
def test_plan_turn_high_seeds_top5_suggested_related_is_rank6plus() -> None:
|
||||
"""The phase-118 core pin (LOCKED A3/A6): the HIGH prompt seeds
|
||||
exactly the top-5 suggested documents' SUMMARY text and NONE of
|
||||
their full content; the related tier is rank 6+ (docs 6–7, capped
|
||||
by ``related_max_docs``)."""
|
||||
docs_in = _seven_docs_with_summaries()
|
||||
chunks = [
|
||||
_chunk(d, 0.9 - 0.1 * i, cosine=0.8 - 0.05 * i) for i, d in enumerate(docs_in)
|
||||
]
|
||||
plan = chat_api.plan_turn(chunks, _tier_settings())
|
||||
assert plan.deflected is False
|
||||
assert [d.title for d in plan.suggested_docs] == [f"Doc {i}" for i in range(5)]
|
||||
assert [d.title for d in plan.related_docs] == ["Doc 5", "Doc 6"]
|
||||
# The prompt seeds the five summaries …
|
||||
for i in range(5):
|
||||
assert f"SUMMARY_TEXT_{i}" in plan.system_prompt
|
||||
# … and NONE of the seven documents' full content (suggested OR
|
||||
# related) reaches the LLM (A6).
|
||||
for i in range(7):
|
||||
assert f"FULL_CONTENT_SENTINEL_{i}" not in plan.system_prompt
|
||||
|
||||
|
||||
def test_plan_turn_high_single_strong_doc_yields_one_suggested() -> None:
|
||||
"""The suggested cap is a CEILING, not a quota: one doc ⇒ one
|
||||
suggested doc, an empty related tier (nothing beyond rank 1)."""
|
||||
strong = _doc("Kubernetes Homelab Cluster", "STRONG_DOC_CONTENT")
|
||||
plan = chat_api.plan_turn([_chunk(strong, 0.90, cosine=0.80)], _tier_settings())
|
||||
assert plan.deflected is False
|
||||
assert [d.title for d in plan.suggested_docs] == ["Kubernetes Homelab Cluster"]
|
||||
assert plan.related_docs == []
|
||||
# No stored summary (the fixture default) → the A5 preview fallback
|
||||
# carries the short content whole (under the 400-char cap).
|
||||
assert "STRONG_DOC_CONTENT" in plan.system_prompt
|
||||
|
||||
|
||||
def test_plan_turn_high_suggests_strong_and_weak_no_floor() -> None:
|
||||
"""The recurring incident under phase 118 (A3): the weak 2nd doc no
|
||||
longer loses a citation slot to a bar — the floor never filters, so
|
||||
BOTH docs are suggested (rank order) and ride the citation surface
|
||||
(A4); the HIGH prompt seeds both summaries (the A5 fallback carries
|
||||
the short fixture content whole)."""
|
||||
strong = _doc("Kubernetes Homelab Cluster", "STRONG_DOC_CONTENT")
|
||||
weak = _doc("Backup Strategy", "WEAK_DOC_CONTENT")
|
||||
chunks = [_chunk(strong, 0.90, cosine=0.80), _chunk(weak, 0.80, cosine=0.20)]
|
||||
plan = chat_api.plan_turn(chunks, _bar_settings())
|
||||
plan = chat_api.plan_turn(chunks, _tier_settings())
|
||||
assert plan.deflected is False
|
||||
assert [d.title for d in plan.docs] == ["Kubernetes Homelab Cluster"]
|
||||
assert [d.title for d in plan.related_docs] == ["Backup Strategy"]
|
||||
# The HIGH prompt carries the cited doc's content only.
|
||||
assert "STRONG_DOC_CONTENT" in plan.system_prompt
|
||||
assert "WEAK_DOC_CONTENT" not in plan.system_prompt
|
||||
|
||||
|
||||
def test_plan_turn_high_single_strong_doc_yields_one_cited() -> None:
|
||||
"""top_n_docs is a CEILING, not a quota: one strong doc ⇒ one cited doc,
|
||||
an empty related tier (LOCKED A2)."""
|
||||
strong = _doc("Kubernetes Homelab Cluster", "STRONG_DOC_CONTENT")
|
||||
plan = chat_api.plan_turn([_chunk(strong, 0.90, cosine=0.80)], _bar_settings())
|
||||
assert plan.deflected is False
|
||||
assert [d.title for d in plan.docs] == ["Kubernetes Homelab Cluster"]
|
||||
assert [d.title for d in plan.suggested_docs] == [
|
||||
"Kubernetes Homelab Cluster",
|
||||
"Backup Strategy",
|
||||
]
|
||||
assert plan.related_docs == []
|
||||
assert "STRONG_DOC_CONTENT" in plan.system_prompt
|
||||
assert "WEAK_DOC_CONTENT" in plan.system_prompt
|
||||
|
||||
|
||||
def test_plan_turn_low_weak_hits_fall_to_related() -> None:
|
||||
"""Deflected turn: nothing clears the bar ⇒ the cited tier is empty
|
||||
and the weak hits fall to related_docs (the done frame's home for
|
||||
their visibility). The LOW prompt is unchanged (titles only)."""
|
||||
def test_plan_turn_low_weak_hits_are_suggested_record() -> None:
|
||||
"""Deflected turn: the weak hits are SUGGESTED too (no floor, A3) —
|
||||
the TurnPlan carries suggested + related for the durable record —
|
||||
while the LOW prompt itself stays byte-identical (weak-hit titles
|
||||
only, never content)."""
|
||||
a = _doc("Alpha", "ALPHA_DOC_NEVER_SENT")
|
||||
b = _doc("Beta", "BETA_DOC_NEVER_SENT")
|
||||
chunks = [_chunk(a, 0.30, cosine=0.20), _chunk(b, 0.20, cosine=0.15)]
|
||||
plan = chat_api.plan_turn(chunks, _bar_settings())
|
||||
plan = chat_api.plan_turn(chunks, _tier_settings())
|
||||
assert plan.deflected is True
|
||||
assert plan.docs == [] # no citation slot below the bar
|
||||
assert [d.title for d in plan.related_docs] == ["Alpha", "Beta"] # rank order
|
||||
assert [d.title for d in plan.suggested_docs] == ["Alpha", "Beta"] # rank order
|
||||
assert plan.related_docs == [] # nothing beyond rank 2 for 2 docs
|
||||
assert "ALPHA_DOC_NEVER_SENT" not in plan.system_prompt
|
||||
assert "Beta" in plan.system_prompt # weak-hit titles still carried
|
||||
assert plan.suggestions # chips unchanged
|
||||
|
||||
|
||||
def test_plan_turn_related_cap_zero_kills_the_related_tier() -> None:
|
||||
"""related_max_docs=0 is the kill switch: weak docs are scored but
|
||||
neither cited nor related (the pre-phase-113 visibility, minus the
|
||||
false citation — a deflected turn cites nothing)."""
|
||||
a = _doc("Alpha", "AAA")
|
||||
b = _doc("Beta", "BBB")
|
||||
chunks = [_chunk(a, 0.30, cosine=0.20), _chunk(b, 0.20, cosine=0.15)]
|
||||
plan = chat_api.plan_turn(chunks, _bar_settings(related_cap=0))
|
||||
assert plan.deflected is True
|
||||
assert plan.docs == []
|
||||
assert plan.related_docs == []
|
||||
|
||||
|
||||
def test_plan_turn_floor_zero_keeps_legacy_cited_docs() -> None:
|
||||
"""source_usefulness_floor=0 disables the bar: plan.docs is the legacy
|
||||
rank-ordered top-N (any cosine, incl. 0.0 lexical-only) and the
|
||||
related tier is empty."""
|
||||
a = _doc("Alpha", "AAA")
|
||||
b = _doc("Beta", "BBB")
|
||||
chunks = [
|
||||
_chunk(a, 0.90, cosine=0.0, fts_hit=True), # lexical-only, rank 1
|
||||
_chunk(b, 0.80, cosine=0.10),
|
||||
]
|
||||
plan = chat_api.plan_turn(
|
||||
chunks, _bar_settings(source_floor=0.0, lex_floor=0.05)
|
||||
)
|
||||
assert plan.deflected is False # 0.10 + the fts hit clears the 0.05 lex floor
|
||||
assert [d.title for d in plan.docs] == ["Alpha", "Beta"] # legacy order
|
||||
assert plan.related_docs == []
|
||||
|
||||
|
||||
def test_plan_turn_lexically_grounded_below_source_floor_has_no_cited_docs() -> None:
|
||||
"""The degenerate operator config (citation bar STRICTER than the
|
||||
grounding bar): a turn grounded by a corroborated-lexical hit whose
|
||||
cosine sits between the two floors has an EMPTY cited tier — the HIGH
|
||||
prompt carries no document content (the tools remain the escape
|
||||
hatch). The bar is a citation filter, not a gate input."""
|
||||
doc = _doc("Static DNS", "DNS_DOC_CONTENT")
|
||||
plan = chat_api.plan_turn(
|
||||
[_chunk(doc, 0.50, cosine=0.35, fts_hit=True)],
|
||||
_bar_settings(threshold=0.62, lex_floor=0.30, source_floor=0.50),
|
||||
)
|
||||
assert plan.deflected is False # 0.35 >= lex floor 0.30, fts fired
|
||||
assert plan.docs == [] # 0.35 < source floor 0.50 — no citation slot
|
||||
assert "DNS_DOC_CONTENT" not in plan.system_prompt
|
||||
# The doc still SCORED — it rides the related tier (the "nearby docs"
|
||||
# row), it is not invisible.
|
||||
assert [d.title for d in plan.related_docs] == ["Static DNS"]
|
||||
|
||||
|
||||
def test_plan_turn_related_tier_capped_in_rank_order() -> None:
|
||||
"""Grounded turn, four bar-clearing docs, ceiling 2: cited = the top-2
|
||||
in rank order; related = the next two (the ceiling overflow, any
|
||||
cosine), capped at related_max_docs."""
|
||||
docs_in = [
|
||||
_doc(f"Doc {i}", f"DOC_CONTENT_{i}") for i in range(4)
|
||||
]
|
||||
"""related_max_docs=0 is the kill switch: rank-6+ docs are scored
|
||||
and suggested-adjacent but neither suggested nor related — the
|
||||
done frame's row stays empty."""
|
||||
docs_in = _seven_docs_with_summaries()
|
||||
chunks = [
|
||||
_chunk(d, 0.9 - 0.1 * i, cosine=0.8 - 0.05 * i) for i, d in enumerate(docs_in)
|
||||
]
|
||||
plan = chat_api.plan_turn(chunks, _bar_settings(top_n=2, related_cap=2))
|
||||
plan = chat_api.plan_turn(chunks, _tier_settings(related_cap=0))
|
||||
assert plan.deflected is False
|
||||
assert [d.title for d in plan.docs] == ["Doc 0", "Doc 1"]
|
||||
assert [d.title for d in plan.related_docs] == ["Doc 2", "Doc 3"]
|
||||
assert len(plan.suggested_docs) == 5 # the suggestion tier is untouched
|
||||
assert plan.related_docs == []
|
||||
# The cap restores the rank-6+ row (rank order, capped).
|
||||
wide = chat_api.plan_turn(chunks, _tier_settings(related_cap=3))
|
||||
assert [d.title for d in wide.related_docs] == ["Doc 5", "Doc 6"]
|
||||
|
||||
|
||||
def test_plan_turn_suggested_docs_setting_caps_the_suggested_tier() -> None:
|
||||
"""``suggested_docs`` (``BOR_SUGGESTED_DOCS``) is honored as the
|
||||
suggestion cap: 3 here ⇒ the top-3 rank-ordered docs are suggested
|
||||
and rank 4+ falls to the related tier (capped at ``related_max_docs``)."""
|
||||
docs_in = _seven_docs_with_summaries()
|
||||
chunks = [
|
||||
_chunk(d, 0.9 - 0.1 * i, cosine=0.8 - 0.05 * i) for i, d in enumerate(docs_in)
|
||||
]
|
||||
plan = chat_api.plan_turn(chunks, _tier_settings(suggested_cap=3))
|
||||
assert [d.title for d in plan.suggested_docs] == ["Doc 0", "Doc 1", "Doc 2"]
|
||||
assert [d.title for d in plan.related_docs] == ["Doc 3", "Doc 4"] # cap 2
|
||||
|
||||
|
||||
def test_plan_turn_does_not_consult_retired_seeding_settings() -> None:
|
||||
"""Phase 118 (A6): ``top_n_docs`` and ``source_usefulness_floor``
|
||||
lost their seeding role — ``plan_turn`` never consults them. A
|
||||
degenerate config (the maximum legal bar — 0.62, above every chunk's
|
||||
cosine of 0.50 — and a top-N of 1) changes nothing: the suggested
|
||||
tier is still the no-floor top-5 in rank order and the related tier
|
||||
is still rank 6+.
|
||||
(Behavioral pin — the settings themselves stay, env back-compat.)"""
|
||||
docs_in = _seven_docs_with_summaries()
|
||||
chunks = [
|
||||
_chunk(docs_in[0], 0.9, cosine=0.50, fts_hit=True),
|
||||
] + [
|
||||
_chunk(d, 0.9 - 0.1 * i, cosine=0.50)
|
||||
for i, d in enumerate(docs_in[1:], start=1)
|
||||
]
|
||||
settings = Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=0.62,
|
||||
lexical_support_floor=0.35,
|
||||
top_n_docs=1, # retired: the old full-text seeding ceiling
|
||||
# retired: the maximum legal bar (== threshold) — above every
|
||||
# cosine here (0.50), so the OLD tiering would cite nothing.
|
||||
source_usefulness_floor=0.62,
|
||||
related_max_docs=2,
|
||||
)
|
||||
plan = chat_api.plan_turn(chunks, settings)
|
||||
assert plan.deflected is False # 0.50 >= the 0.35 lex floor, fts fired
|
||||
assert [d.title for d in plan.suggested_docs] == [f"Doc {i}" for i in range(5)]
|
||||
assert [d.title for d in plan.related_docs] == ["Doc 5", "Doc 6"]
|
||||
|
||||
|
||||
# ---------- summary hits (phase 30: summary → full source document) ----------
|
||||
|
||||
|
||||
def test_summary_hit_on_selected_top_doc_counts() -> None:
|
||||
"""HIGH branch: the top document was hit via its summary chunk ⇒ 1.
|
||||
|
||||
Context assembly is unchanged (A7 revised): the *source* document's
|
||||
full content lands in the prompt, not the summary text alone.
|
||||
"""
|
||||
def test_summary_hit_on_suggested_doc_counts() -> None:
|
||||
"""HIGH branch: a suggested (rank-1) document hit via its summary
|
||||
chunk ⇒ 1. Phase 118 (A6): the *summary* is what the LLM sees in
|
||||
the prompt (no stored summary here → the A5 preview fallback carries
|
||||
the short fixture content whole)."""
|
||||
a = _doc("Alpha", "ALPHA_FULL_SOURCE_CONTENT")
|
||||
b = _doc("Beta", "BETA_FULL_SOURCE_CONTENT")
|
||||
chunks = [
|
||||
_chunk(a, 0.90, is_summary=True), # top doc reached through its summary
|
||||
_chunk(a, 0.90, is_summary=True), # suggested doc reached through its summary
|
||||
_chunk(b, 0.50),
|
||||
]
|
||||
plan = chat_api.plan_turn(chunks, _settings(threshold=0.30))
|
||||
assert plan.deflected is False
|
||||
assert plan.summary_hits == 1
|
||||
# The full source document is what the LLM sees (phase 24 contract).
|
||||
# The A5 preview fallback (short content, under the 400-char cap)
|
||||
# carries the content whole — the block body, never more.
|
||||
assert "ALPHA_FULL_SOURCE_CONTENT" in plan.system_prompt
|
||||
|
||||
|
||||
def test_summary_hit_outside_top_n_selection_not_counted() -> None:
|
||||
"""A summary chunk on a document outside the top-N (default 2) selection
|
||||
does not count — only hits that landed in the selected context do."""
|
||||
a = _doc("Alpha", "ALPHA_CONTENT")
|
||||
b = _doc("Beta", "BETA_CONTENT")
|
||||
c = _doc("Gamma", "GAMMA_CONTENT")
|
||||
def test_summary_hits_counts_suggested_parent_only() -> None:
|
||||
"""Phase 118 redefinition (redefined from the phase-113 cited set):
|
||||
a summary chunk counts ONLY when its parent document is in the
|
||||
SUGGESTED set — a rank-1 (suggested) parent counts, a rank-6
|
||||
(related-tier-only) parent does not."""
|
||||
docs_in = [_doc(f"Doc {i}", f"CONTENT_{i}") for i in range(7)]
|
||||
chunks = [
|
||||
_chunk(a, 0.90),
|
||||
_chunk(b, 0.80),
|
||||
_chunk(c, 0.70, is_summary=True), # 3rd-ranked doc — not selected
|
||||
_chunk(docs_in[0], 0.90, is_summary=True), # suggested parent — counts
|
||||
_chunk(docs_in[1], 0.80),
|
||||
_chunk(docs_in[2], 0.70),
|
||||
_chunk(docs_in[3], 0.60),
|
||||
_chunk(docs_in[4], 0.50),
|
||||
_chunk(docs_in[5], 0.40, is_summary=True), # related-only parent — does not
|
||||
_chunk(docs_in[6], 0.30),
|
||||
]
|
||||
plan = chat_api.plan_turn(chunks, _settings(threshold=0.30))
|
||||
assert plan.deflected is False
|
||||
assert [d.title for d in plan.docs] == ["Alpha", "Beta"]
|
||||
assert plan.summary_hits == 0
|
||||
assert [d.title for d in plan.suggested_docs] == [f"Doc {i}" for i in range(5)]
|
||||
assert [d.title for d in plan.related_docs] == ["Doc 5", "Doc 6"]
|
||||
assert plan.summary_hits == 1
|
||||
|
||||
|
||||
def test_low_branch_counts_summary_hit_on_selected_doc() -> None:
|
||||
def test_low_branch_counts_summary_hit_on_suggested_doc() -> None:
|
||||
"""LOW (deflected) branch records ``summary_hits`` too: the weak hit's
|
||||
parent is still the selected (weak-hit) document."""
|
||||
parent is still the suggested (weak-hit) document (no floor, A3)."""
|
||||
a = _doc("Gamma", "GAMMA_DOC_CONTENT")
|
||||
b = _doc("Delta", "DELTA_DOC_CONTENT")
|
||||
chunks = [
|
||||
@@ -683,7 +739,43 @@ def test_high_path_unaffected() -> None:
|
||||
assert "Reply in plain text only" not in plan.system_prompt
|
||||
assert "ALPHA_DOC_CONTENT" in plan.system_prompt
|
||||
assert "BETA_DOC_CONTENT" in plan.system_prompt
|
||||
assert [d.title for d in plan.docs] == ["Kubernetes Homelab Cluster", "Backup Strategy"]
|
||||
# Phase 118 (A3): both docs are suggested (no floor) — the durable-
|
||||
# record input — with nothing left for the related tier (rank 6+).
|
||||
assert [d.title for d in plan.suggested_docs] == [
|
||||
"Kubernetes Homelab Cluster",
|
||||
"Backup Strategy",
|
||||
]
|
||||
assert plan.related_docs == []
|
||||
|
||||
|
||||
# ---------- deflected branch: byte-identical (A8 gate untouched) ----------
|
||||
|
||||
|
||||
def test_low_prompt_byte_identical_to_pre_task_sha_pin() -> None:
|
||||
"""Phase 118 (A8 gate untouched): the deflected prompt is
|
||||
BYTE-IDENTICAL to pre-task on the same chunks — it is exactly
|
||||
``build_deflect_prompt(weak_hit_titles(…))`` (the tiering feeds the
|
||||
TurnPlan's durable-record fields, never the LOW prompt). The sha256
|
||||
pin makes any future LOW-body drift loud; the suggestions/chips and
|
||||
the deflected flag are unchanged."""
|
||||
a = _doc("Kubernetes Homelab Cluster", "ALPHA_DOC_CONTENT")
|
||||
b = _doc("Backup Strategy", "BETA_DOC_CONTENT")
|
||||
chunks = [_chunk(b, 0.10), _chunk(a, 0.20)]
|
||||
plan = chat_api.plan_turn(chunks, _settings())
|
||||
assert plan.deflected is True
|
||||
expected = build_deflect_prompt(weak_hit_titles(chunks))
|
||||
assert plan.system_prompt == expected # byte-identical to the pre-task build
|
||||
assert (
|
||||
hashlib.sha256(plan.system_prompt.encode("utf-8")).hexdigest()
|
||||
== "603395e013c97be8a13837eda533c0b7bd5da4f7a0806b0ae8ad7d6c52420913"
|
||||
)
|
||||
assert plan.suggestions # the "Maybe try" chips are unchanged
|
||||
# Both tiers still ride the plan (the durable-record inputs, A3).
|
||||
assert [d.title for d in plan.suggested_docs] == [
|
||||
"Kubernetes Homelab Cluster",
|
||||
"Backup Strategy",
|
||||
]
|
||||
assert plan.related_docs == []
|
||||
|
||||
|
||||
# ---------- weak_hit_titles (fake retriever mapping) ----------
|
||||
@@ -1059,3 +1151,55 @@ def test_endpoint_no_kb_row_prompt_unchanged(
|
||||
assert "<knowledge_base>" not in system["content"]
|
||||
log_lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
|
||||
assert log_lines and "kb_chars=0" in log_lines[-1]
|
||||
|
||||
|
||||
def test_endpoint_log_line_records_suggested_after_summary_hits(
|
||||
client: TestClient,
|
||||
gate_env: tuple[_FakeSession, _CannedLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""Phase 118 (PLAN §9 extension): the per-turn log line gains
|
||||
``suggested=N`` immediately AFTER ``summary_hits=N`` — the field
|
||||
order of every existing field is untouched (the phase-114
|
||||
``retries=N scaffold_stripped=N`` tail stays last). The value is the
|
||||
seeded suggestion tier's size (``len(plan.suggested_docs)``)."""
|
||||
_session, _llm = gate_env
|
||||
doc = _doc("Kubernetes Homelab Cluster", "TALOS_DOC_SENT")
|
||||
chunks = [_chunk(doc, 0.30), _chunk(doc, 0.20, is_summary=True)]
|
||||
monkeypatch.setattr(chat_api, "retrieve", _fake_retriever(chunks))
|
||||
|
||||
with caplog.at_level("INFO", logger="app.chat"):
|
||||
_ask(client, "How is my Kubernetes cluster set up?")
|
||||
|
||||
log_lines = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
|
||||
assert log_lines
|
||||
line = log_lines[-1]
|
||||
# The new slot: suggested=N right after summary_hits=N (one suggested
|
||||
# doc — the single fixture doc — and one summary hit on it).
|
||||
assert "summary_hits=1 suggested=1" in line
|
||||
# The full field order (the phase-114 tail stays last).
|
||||
order = (
|
||||
"question=",
|
||||
"embed_ms=",
|
||||
"top_score=",
|
||||
"fts_hits=",
|
||||
"summary_hits=",
|
||||
"suggested=",
|
||||
"tuning=",
|
||||
"kb_chars=",
|
||||
"history_msgs=",
|
||||
"threshold=",
|
||||
"deflected=",
|
||||
"sources=",
|
||||
"thinking_chars=",
|
||||
"tool_calls=",
|
||||
"total_ms=",
|
||||
"retries=",
|
||||
"scaffold_stripped=",
|
||||
)
|
||||
idx = -1
|
||||
for field in order:
|
||||
pos = line.find(field)
|
||||
assert pos > idx, f"{field} out of order in the per-turn log line"
|
||||
idx = pos
|
||||
|
||||
@@ -282,6 +282,59 @@ def test_related_max_docs_rejects_negative(
|
||||
_settings()
|
||||
|
||||
|
||||
def test_suggested_docs_default_and_env_override(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 118 (LOCKED A3): the start-here suggestion tier cap —
|
||||
default 5 (the owner directive, TODO L3), env-tunable."""
|
||||
monkeypatch.delenv("BOR_SUGGESTED_DOCS", raising=False)
|
||||
assert _settings().suggested_docs == 5
|
||||
monkeypatch.setenv("BOR_SUGGESTED_DOCS", "3")
|
||||
assert _settings().suggested_docs == 3
|
||||
|
||||
|
||||
def test_suggested_docs_rejects_zero_and_negative(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""``0`` (no starting points) and a negative cap are typos (the
|
||||
``agent_max_rounds`` pattern); ``1`` is the minimum legal value."""
|
||||
monkeypatch.setenv("BOR_SUGGESTED_DOCS", "0")
|
||||
with pytest.raises(ValidationError, match="suggested_docs"):
|
||||
_settings()
|
||||
monkeypatch.setenv("BOR_SUGGESTED_DOCS", "-2")
|
||||
with pytest.raises(ValidationError, match="suggested_docs"):
|
||||
_settings()
|
||||
monkeypatch.setenv("BOR_SUGGESTED_DOCS", "1")
|
||||
assert _settings().suggested_docs == 1
|
||||
|
||||
|
||||
def test_suggestion_preview_chars_default_and_env_override(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Phase 118, task 03 (LOCKED A5): the NULL-summary preview cap —
|
||||
default 400, env-tunable via ``BOR_SUGGESTION_PREVIEW_CHARS``."""
|
||||
monkeypatch.delenv("BOR_SUGGESTION_PREVIEW_CHARS", raising=False)
|
||||
assert _settings().suggestion_preview_chars == 400
|
||||
monkeypatch.setenv("BOR_SUGGESTION_PREVIEW_CHARS", "800")
|
||||
assert _settings().suggestion_preview_chars == 800
|
||||
|
||||
|
||||
def test_suggestion_preview_chars_rejects_zero_and_negative(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""``0``/negative would preview an empty/absent prefix — typos that
|
||||
fail loudly (the ``agent_max_rounds`` pattern); ``1`` is the minimum
|
||||
legal value."""
|
||||
monkeypatch.setenv("BOR_SUGGESTION_PREVIEW_CHARS", "0")
|
||||
with pytest.raises(ValidationError, match="suggestion_preview_chars"):
|
||||
_settings()
|
||||
monkeypatch.setenv("BOR_SUGGESTION_PREVIEW_CHARS", "-5")
|
||||
with pytest.raises(ValidationError, match="suggestion_preview_chars"):
|
||||
_settings()
|
||||
monkeypatch.setenv("BOR_SUGGESTION_PREVIEW_CHARS", "1")
|
||||
assert _settings().suggestion_preview_chars == 1
|
||||
|
||||
|
||||
def test_read_max_chars_default_and_env_override(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
|
||||
+272
-22
@@ -4,8 +4,10 @@ The walk tests are pure filesystem (``tmp_path``); the delta and summary
|
||||
tests run against the local compose Postgres (preferred — a real vector
|
||||
table), skipping with clear instructions when the stack is not up.
|
||||
|
||||
Summaries (phase 30): non-markdown files get a ``lite``-model summary via
|
||||
the fake's deterministic ``chat`` (``"Summary of <first token>"``); the
|
||||
Summaries (phase 30; phase 118, A2: every file, markdown included):
|
||||
every file gets a ``lite``-model summary via the fake's deterministic
|
||||
``chat`` (``"Summary of <first token>"``); an unchanged doc whose summary
|
||||
is NULL is backfilled on the next run (``summary_backfilled``); the
|
||||
sentinel word ``SUMMARY-BLOWUP`` makes ``chat`` raise :class:`LLMError`
|
||||
for the fail-soft path.
|
||||
"""
|
||||
@@ -13,10 +15,12 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import func, select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
import app.rag.importer as importer
|
||||
from app.config import Settings
|
||||
@@ -29,7 +33,7 @@ from app.rag.importer import (
|
||||
iter_importable_files,
|
||||
match_extension,
|
||||
)
|
||||
from app.rag.llm import EmbeddingError
|
||||
from app.rag.llm import EmbeddingError, LLMError
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
#: The original seven A9 formats as dotted suffixes (pre-phase-47 default
|
||||
@@ -47,6 +51,15 @@ class _PoisonEmbedder(FakeEmbedder):
|
||||
return await super().embed(texts)
|
||||
|
||||
|
||||
class _FailingChatEmbedder(FakeEmbedder):
|
||||
"""A ``lite`` model that always fails (drives the summary fail-soft
|
||||
path — including the phase-118 backfill)."""
|
||||
|
||||
async def chat(self, messages: list[dict[str, str]], model: str | None = None) -> str:
|
||||
self.chat_calls.append(list(messages))
|
||||
raise LLMError("simulated lite-model failure (test sentinel)")
|
||||
|
||||
|
||||
class _CapEmbedder(FakeEmbedder):
|
||||
"""Simulates the endpoint's ~1024-token input cap at ~1.1 chars/token:
|
||||
any single text over 1000 chars is rejected (URL-dense worst case)."""
|
||||
@@ -266,16 +279,19 @@ def test_added_then_unchanged_then_updated_then_pruned(db, tmp_path: Path) -> No
|
||||
try:
|
||||
s1 = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (s1.files, s1.added, s1.unchanged, s1.updated, s1.pruned) == (2, 2, 0, 0, 0)
|
||||
# a.md has two sections (2 chunks), b.md one (1 chunk).
|
||||
# a.md has two sections (2 chunks), b.md one (1 chunk) — content
|
||||
# chunks only; the ``is_summary`` chunks live in ``summaries``.
|
||||
assert s1.chunks == 3
|
||||
# Embeddings are stored with the configured dimension.
|
||||
assert s1.summaries == 2 # phase 118 (A2): markdown is summarized too
|
||||
# Embeddings are stored with the configured dimension: 3 content
|
||||
# chunks + 2 ``is_summary`` chunks (one per doc, phase 118 A2).
|
||||
n = db.scalar(
|
||||
select(func.count())
|
||||
.select_from(Chunk)
|
||||
.join(Document, Document.id == Chunk.document_id)
|
||||
.where(Document.source == root.name)
|
||||
)
|
||||
assert n == 3
|
||||
assert n == 5
|
||||
for c in db.scalars(
|
||||
select(Chunk)
|
||||
.join(Document, Document.id == Chunk.document_id)
|
||||
@@ -300,14 +316,15 @@ def test_added_then_unchanged_then_updated_then_pruned(db, tmp_path: Path) -> No
|
||||
assert db.scalar(
|
||||
select(Document).where(Document.source == root.name, Document.path == "a.md")
|
||||
) is None
|
||||
# Chunks of the pruned document are gone (FK cascade).
|
||||
# Chunks of the pruned document are gone (FK cascade); b.md's
|
||||
# content chunk + its ``is_summary`` chunk survive (phase 118 A2).
|
||||
n_after = db.scalar(
|
||||
select(func.count())
|
||||
.select_from(Chunk)
|
||||
.join(Document, Document.id == Chunk.document_id)
|
||||
.where(Document.source == root.name)
|
||||
)
|
||||
assert n_after == 1
|
||||
assert n_after == 2
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
@@ -438,8 +455,11 @@ def test_chunk_positions_and_titles(db, tmp_path: Path) -> None:
|
||||
select(Document).where(Document.source == root.name, Document.path == "multi.md")
|
||||
)
|
||||
assert doc is not None
|
||||
positions = sorted(c.position for c in doc.chunks)
|
||||
assert positions == list(range(len(doc.chunks))) and len(doc.chunks) >= 2
|
||||
# 0-based CONTENT positions (the ``is_summary`` chunk sits at −1,
|
||||
# phase 30/118).
|
||||
content = [c for c in doc.chunks if not c.is_summary]
|
||||
positions = sorted(c.position for c in content)
|
||||
assert positions == list(range(len(content))) and len(content) >= 2
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
@@ -535,7 +555,7 @@ def test_quadlet_and_j2_files_get_stem_titles_and_per_format_counts(
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
# ---------- phase 30: lite-model summaries for non-markdown files ----------
|
||||
# ---------- phase 30: lite-model summaries (phase 118: every file) ----------
|
||||
|
||||
|
||||
def test_non_markdown_file_gets_stored_and_indexed_summary(db, tmp_path: Path) -> None:
|
||||
@@ -571,9 +591,11 @@ def test_non_markdown_file_gets_stored_and_indexed_summary(db, tmp_path: Path) -
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_markdown_file_never_gets_summary(db, tmp_path: Path) -> None:
|
||||
"""Markdown is already natural language: no summary, no ``is_summary``
|
||||
chunk, and the ``lite`` model is never called."""
|
||||
def test_markdown_file_gets_stored_summary(db, tmp_path: Path) -> None:
|
||||
"""Phase 118 (A2): markdown is summarized too (the phase-30 exclusion
|
||||
is retired) — ``documents.summary`` is set and one ``is_summary``
|
||||
chunk (position −1, embedded) is indexed alongside the content
|
||||
chunks."""
|
||||
root = tmp_path / "mdsrc"
|
||||
root.mkdir()
|
||||
(root / "note.md").write_text("# Note\n\nmarkdown body\n")
|
||||
@@ -581,14 +603,240 @@ def test_markdown_file_never_gets_summary(db, tmp_path: Path) -> None:
|
||||
try:
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert summary.added == 1
|
||||
assert summary.summaries == 0 and summary.summary_errors == 0
|
||||
assert llm.chat_calls == [] # the model was never asked
|
||||
assert summary.summaries == 1 and summary.summary_errors == 0
|
||||
assert llm.chat_calls # the lite model WAS asked (phase 118)
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == root.name, Document.path == "note.md")
|
||||
)
|
||||
assert doc is not None
|
||||
assert doc.summary is None
|
||||
assert doc.chunks and all(not c.is_summary for c in doc.chunks)
|
||||
assert doc.summary is not None
|
||||
# Deterministic fake reply + the code-appended pointer line.
|
||||
assert doc.summary.startswith("Summary of")
|
||||
assert doc.summary.endswith(f"Source: {root.name}/note.md")
|
||||
schunks = [c for c in doc.chunks if c.is_summary]
|
||||
assert len(schunks) == 1
|
||||
assert schunks[0].position == -1
|
||||
assert schunks[0].content == doc.summary
|
||||
assert schunks[0].embedding is not None and len(schunks[0].embedding) == 768
|
||||
# Content chunks stay 0-based and are never flagged as summaries.
|
||||
content = [c for c in doc.chunks if not c.is_summary]
|
||||
assert sorted(c.position for c in content) == list(range(len(content)))
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
# ---------- phase 118 (A2): NULL-summary backfill on the unchanged path ----------
|
||||
|
||||
|
||||
def _clear_stored_summary(db: Session, doc: Document) -> None:
|
||||
"""Simulate a NULL-summary row (a pre-phase-30 row, or a cleared
|
||||
summary): the content stays, only the summary + its chunk go away."""
|
||||
doc.summary = None
|
||||
for c in [c for c in doc.chunks if c.is_summary]:
|
||||
doc.chunks.remove(c)
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_unchanged_doc_with_null_summary_is_backfilled(db, tmp_path: Path) -> None:
|
||||
"""Phase 118 (A2): an unchanged doc whose summary is NULL gets a
|
||||
summary-only backfill on the next sync: summary stored + one embedded
|
||||
``is_summary`` chunk, counted ``summary_backfilled`` — never
|
||||
``summaries``, never added/updated/pruned, no content re-embed."""
|
||||
root = tmp_path / "bfill"
|
||||
root.mkdir()
|
||||
(root / "note.md").write_text("# Note\n\nmarkdown body\n")
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
first = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (first.added, first.summaries) == (1, 1)
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == root.name, Document.path == "note.md")
|
||||
)
|
||||
assert doc is not None and doc.summary is not None
|
||||
_clear_stored_summary(db, doc) # the NULL-summary row the backfill targets
|
||||
|
||||
embed_before = len(llm.calls)
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (second.added, second.updated, second.pruned) == (0, 0, 0)
|
||||
assert second.unchanged == 1
|
||||
assert second.summary_backfilled == 1
|
||||
assert second.summaries == 0 and second.summary_errors == 0
|
||||
# No content re-embed: exactly one new embed batch, the summary
|
||||
# text only.
|
||||
assert len(llm.calls) == embed_before + 1
|
||||
|
||||
db.expire_all()
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == root.name, Document.path == "note.md")
|
||||
)
|
||||
assert doc is not None
|
||||
assert doc.summary is not None
|
||||
assert llm.calls[-1] == [doc.summary] # only the backfilled summary
|
||||
schunks = [c for c in doc.chunks if c.is_summary]
|
||||
assert len(schunks) == 1
|
||||
assert schunks[0].position == -1
|
||||
assert schunks[0].content == doc.summary
|
||||
assert schunks[0].embedding is not None and len(schunks[0].embedding) == 768
|
||||
# The content chunk is untouched.
|
||||
content = [c for c in doc.chunks if not c.is_summary]
|
||||
assert len(content) == 1 and content[0].embedding is not None
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_unchanged_doc_with_stored_summary_never_resummarizes(db, tmp_path: Path) -> None:
|
||||
"""Phase 118 (A2): an unchanged doc that ALREADY has a summary (the
|
||||
third sync of the lifecycle) makes no summary LLM call at all and
|
||||
gains no chunks — owner-edited (non-NULL) summaries are never
|
||||
touched."""
|
||||
root = tmp_path / "noref"
|
||||
root.mkdir()
|
||||
(root / "note.md").write_text("# Note\n\nmarkdown body\n")
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
chat_before = len(llm.chat_calls)
|
||||
embed_before = len(llm.calls)
|
||||
chunk_before = db.scalar(
|
||||
select(func.count())
|
||||
.select_from(Chunk)
|
||||
.join(Document, Document.id == Chunk.document_id)
|
||||
.where(Document.source == root.name)
|
||||
)
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert second.unchanged == 1
|
||||
assert second.summary_backfilled == 0 and second.summaries == 0
|
||||
assert second.summary_errors == 0
|
||||
assert len(llm.chat_calls) == chat_before # the model was never asked
|
||||
assert len(llm.calls) == embed_before # no embedding of any kind
|
||||
chunk_after = db.scalar(
|
||||
select(func.count())
|
||||
.select_from(Chunk)
|
||||
.join(Document, Document.id == Chunk.document_id)
|
||||
.where(Document.source == root.name)
|
||||
)
|
||||
assert chunk_after == chunk_before # no new chunk of any kind
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_unchanged_doc_with_empty_string_summary_is_never_backfilled(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 118 (A2, strict ``is None``): an empty-string summary is
|
||||
owner-set (phase 57) — the backfill skips it, the ``lite`` model is
|
||||
never called, and the value stays byte-identical."""
|
||||
root = tmp_path / "emptysum"
|
||||
root.mkdir()
|
||||
(root / "note.md").write_text("# Note\n\nmarkdown body\n")
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == root.name, Document.path == "note.md")
|
||||
)
|
||||
assert doc is not None
|
||||
doc.summary = "" # the owner-set empty string (never NULL)
|
||||
db.commit()
|
||||
|
||||
chat_before = len(llm.chat_calls)
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert second.unchanged == 1
|
||||
assert second.summary_backfilled == 0 and second.summaries == 0
|
||||
assert second.summary_errors == 0
|
||||
assert len(llm.chat_calls) == chat_before # the model was never asked
|
||||
|
||||
db.expire_all()
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == root.name, Document.path == "note.md")
|
||||
)
|
||||
assert doc is not None
|
||||
assert doc.summary == "" # byte-identical — never overwritten
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_backfill_runs_on_manually_dated_doc_without_touching_the_date(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 118 (A2, assumption 7): ``created_at_manual`` protects the
|
||||
DATE only (phase 106, D1) — a manually-dated, NULL-summary doc still
|
||||
gets its backfilled summary, and the stored date stays byte-untouched
|
||||
even though a refresh was due."""
|
||||
root = tmp_path / "manualdate"
|
||||
root.mkdir()
|
||||
(root / "note.md").write_text("# Note\n\nmarkdown body\n")
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == root.name, Document.path == "note.md")
|
||||
)
|
||||
assert doc is not None
|
||||
manual = datetime(2020, 5, 4, 12, 0, 0, tzinfo=UTC)
|
||||
doc.created_at = manual
|
||||
doc.created_at_manual = True
|
||||
_clear_stored_summary(db, doc) # the NULL-summary row the backfill targets
|
||||
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert second.unchanged == 1
|
||||
assert second.summary_backfilled == 1 and second.summary_errors == 0
|
||||
# A date refresh WAS due (the mtime differs from the 2020
|
||||
# correction) but the manual flag withheld it — the backfill
|
||||
# never touches the date either.
|
||||
assert second.dates_updated == 0
|
||||
|
||||
db.expire_all()
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == root.name, Document.path == "note.md")
|
||||
)
|
||||
assert doc is not None
|
||||
assert doc.summary is not None # the backfill landed
|
||||
assert doc.created_at == manual # byte-untouched
|
||||
assert doc.created_at_manual is True
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_backfill_failure_is_fail_soft_and_date_still_refreshes(
|
||||
db, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 118 (A2): a backfill whose ``lite`` call fails rolls back
|
||||
its own session work only — ``summary_errors=1``, the doc row
|
||||
untouched — while the UNCHANGED path's date refresh still runs
|
||||
afterwards."""
|
||||
root = tmp_path / "bfillfail"
|
||||
root.mkdir()
|
||||
(root / "note.md").write_text("# Note\n\nmarkdown body\n")
|
||||
llm = FakeEmbedder()
|
||||
try:
|
||||
asyncio.run(import_sources([root], llm, session=db))
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == root.name, Document.path == "note.md")
|
||||
)
|
||||
assert doc is not None
|
||||
_clear_stored_summary(db, doc) # the NULL-summary row the backfill targets
|
||||
# Force a date drift so the refresh is DUE on this run.
|
||||
doc.created_at = datetime(2020, 1, 1, tzinfo=UTC)
|
||||
db.commit()
|
||||
|
||||
second = asyncio.run(import_sources([root], _FailingChatEmbedder(), session=db))
|
||||
assert second.unchanged == 1
|
||||
assert (second.added, second.updated, second.pruned) == (0, 0, 0)
|
||||
assert second.summary_errors == 1
|
||||
assert second.summary_backfilled == 0 and second.summaries == 0
|
||||
|
||||
db.expire_all()
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == root.name, Document.path == "note.md")
|
||||
)
|
||||
assert doc is not None
|
||||
assert doc.summary is None # the failed backfill left the row untouched
|
||||
assert not any(c.is_summary for c in doc.chunks)
|
||||
# …but the date refresh ran (the failure only rolled back the
|
||||
# summary's own session work).
|
||||
assert second.dates_updated == 1
|
||||
assert doc.created_at != datetime(2020, 1, 1, tzinfo=UTC)
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
@@ -720,11 +968,13 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
) -> None:
|
||||
"""PLAN §9 summary line: the phase-30 counters sit between
|
||||
``embed_batches`` and ``formats``; the phase-106 date-refresh
|
||||
counter sits between ``summary_errors`` and ``formats``."""
|
||||
``embed_batches`` and ``formats``; the phase-118 backfill counter
|
||||
sits between ``summary_errors`` and ``dates_updated``; the
|
||||
phase-106 date-refresh counter sits before ``formats``."""
|
||||
s = ImportSummary()
|
||||
s.files, s.added, s.chunks, s.embed_batches = 3, 3, 5, 4
|
||||
s.summaries, s.summary_errors = 2, 1
|
||||
s.summary_backfilled = 1
|
||||
s.dates_updated = 0
|
||||
s.formats = {"md": 1, "yaml": 2}
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
@@ -732,8 +982,8 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
line = caplog.records[-1].getMessage()
|
||||
assert line == (
|
||||
"import: summary files=3 added=3 updated=0 unchanged=0 pruned=0 errors=0 "
|
||||
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 dates_updated=0 "
|
||||
"formats=yaml:2,md:1"
|
||||
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
|
||||
"dates_updated=0 formats=yaml:2,md:1"
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -16,6 +16,7 @@ import hashlib
|
||||
|
||||
from app.rag.prompts import (
|
||||
PERSONA,
|
||||
SUGGEST_INTRO,
|
||||
TOOLS_SECTION,
|
||||
_base,
|
||||
build_deflect_prompt,
|
||||
@@ -67,9 +68,16 @@ def test_high_and_low_bases_byte_locked() -> None:
|
||||
|
||||
# ---------- TOOLS_SECTION (the HIGH prompt's locked ``<tools>`` copy) ----------
|
||||
|
||||
#: Pre-phase-112 anchors for ``TOOLS_SECTION``.
|
||||
TOOLS_SECTION_SHA256 = "b834cbe368055e65da82ae3e37a91e6c658c713954703fc79b849a6ebdf4aa53"
|
||||
TOOLS_SECTION_LEN = 2273
|
||||
#: Anchors for ``TOOLS_SECTION`` — pre-phase-112 values, RE-CUT for
|
||||
#: phase 118 (task 04, A6, owner directive 2026-09-15): the ``read``
|
||||
#: clause was rewritten for the summary-seed mode (the ``<documents>``
|
||||
#: section holds SUMMARIES — ``read`` adds the full text; only an
|
||||
#: already-read document is refused). Only that clause moved — the
|
||||
#: prefix (the ``ls``-clause opening) and the suffix (the
|
||||
#: discipline-rules ending) survived byte-identical, so they are the
|
||||
#: same anchors as pre-phase-118.
|
||||
TOOLS_SECTION_SHA256 = "87ee80faf0170da6ab1de518177313def6460785613fa074312a2a5f9f071750"
|
||||
TOOLS_SECTION_LEN = 2465
|
||||
TOOLS_SECTION_PREFIX = (
|
||||
"<tools>\n"
|
||||
"You may extend your context with three tools. `ls` lists the "
|
||||
@@ -120,3 +128,51 @@ def test_deflect_body_byte_locked() -> None:
|
||||
assert prompt.index("DEFLECT_MODE") < prompt.index(
|
||||
"Reply in plain text only"
|
||||
) # the marker precedes the plain-text line
|
||||
|
||||
|
||||
#: The full LOW prompt build on the canonical fixture titles — sha-pinned
|
||||
#: (phase 118, task 03): the deflection path is UNTOUCHED by the
|
||||
#: summary-seed re-revision (LOCKED A8) — the same inputs must produce
|
||||
#: the pre-phase bytes, so this anchor is a pre-phase-118 value.
|
||||
LOW_PROMPT_SHA256 = "726eddb4eb3bcc26c840011f6f8635d6af55aa09d4d064bbb9e256caa9665837"
|
||||
LOW_PROMPT_LEN = 968
|
||||
|
||||
|
||||
def test_low_prompt_build_byte_identical_to_pre_phase() -> None:
|
||||
"""Phase 118 contract: the LOW prompt output is byte-identical to
|
||||
pre-phase for identical inputs — the summary seeding (and the new
|
||||
``SUGGEST_INTRO`` line) never leaks into the deflection path."""
|
||||
prompt = build_deflect_prompt(["T1", "T2"])
|
||||
assert len(prompt) == LOW_PROMPT_LEN
|
||||
assert _sha256(prompt) == LOW_PROMPT_SHA256
|
||||
assert "SUGGEST_INTRO" not in prompt and "<documents>" not in prompt
|
||||
|
||||
|
||||
# ---------- SUGGEST_INTRO (phase 118, task 03 — the start-here framing) ----------
|
||||
|
||||
#: Phase-118 anchors for ``SUGGEST_INTRO`` — the ``<documents>``
|
||||
#: section's intro line (the owner's "start here if these summaries seem
|
||||
#: right to you" framing, TODO L3). The E2E mock's ``_document_block``
|
||||
#: parser is regex-based over the block markup (which stays byte-stable
|
||||
#: around the intro), so this constant is a prompt-copy lock, pinned the
|
||||
#: way ``TOOLS_SECTION`` is: sha256 + prefix + total length.
|
||||
SUGGEST_INTRO_SHA256 = "7b14d2dedc6ebc4e440d32dd1edb979a7461c94034b9541c9f37b9042f394d3a"
|
||||
SUGGEST_INTRO_LEN = 323
|
||||
SUGGEST_INTRO_PREFIX = (
|
||||
"The blocks below are the summaries of the top-ranked documents for "
|
||||
"your question — start here if one seems right to you: "
|
||||
)
|
||||
|
||||
|
||||
def test_suggest_intro_byte_locked() -> None:
|
||||
"""The start-here framing is LOCKED copy (phase 118): sha256 + exact
|
||||
prefix + total length; the three contracts it must carry (summaries
|
||||
are the starting points; ``read`` adds the full text, which is NOT
|
||||
in the prompt until read; cite by path) are pinned as substrings."""
|
||||
assert len(SUGGEST_INTRO) == SUGGEST_INTRO_LEN
|
||||
assert _sha256(SUGGEST_INTRO) == SUGGEST_INTRO_SHA256
|
||||
assert SUGGEST_INTRO.startswith(SUGGEST_INTRO_PREFIX)
|
||||
assert "call `read`" in SUGGEST_INTRO
|
||||
assert "combined `source/path`" in SUGGEST_INTRO
|
||||
assert "its full text is not in the prompt until you read it" in SUGGEST_INTRO
|
||||
assert "Cite the document(s) you used, by path." in SUGGEST_INTRO
|
||||
|
||||
+303
-24
@@ -17,9 +17,13 @@ stated up front — the combined ``source/path`` identity for
|
||||
the drill-down tree contract — one level per call, sources at the
|
||||
top, folders + files below, ``grep`` as the without-listing locator —
|
||||
while the ``read``/``grep`` clauses and the discipline rules are
|
||||
byte-identical): the teaching refusals in :mod:`app.rag.agent`
|
||||
re-state the same contract; the ``<tools>`` marker keying (HIGH
|
||||
only) is unchanged.
|
||||
byte-identical; phase 118, task 04: the ``read`` clause rewritten
|
||||
for the summary-seed mode — the ``<documents>`` section holds
|
||||
SUMMARIES, ``read`` adds the full text, and only an already-read
|
||||
document is refused — while the ``ls``/``grep`` clauses and the
|
||||
discipline rules stay byte-identical): the teaching refusals in
|
||||
:mod:`app.rag.agent` re-state the same contract; the ``<tools>``
|
||||
marker keying (HIGH only) is unchanged.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -32,6 +36,7 @@ from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag.prompts import (
|
||||
PERSONA,
|
||||
SUGGEST_INTRO,
|
||||
TOOLS_SECTION,
|
||||
_base,
|
||||
build_deflect_prompt,
|
||||
@@ -55,7 +60,12 @@ KB_INTRO = "The basic categories of everything in this knowledge base (generated
|
||||
_FIXTURE_CREATED_AT = datetime(2024, 6, 15, 12, 0, 0, tzinfo=UTC)
|
||||
|
||||
|
||||
def _doc(path: str, content: str, title: str) -> Document:
|
||||
def _doc(
|
||||
path: str,
|
||||
content: str,
|
||||
title: str,
|
||||
summary: str | None = None,
|
||||
) -> Document:
|
||||
return Document(
|
||||
id=uuid.uuid4(),
|
||||
source="Homelab",
|
||||
@@ -65,6 +75,7 @@ def _doc(path: str, content: str, title: str) -> Document:
|
||||
content=content,
|
||||
content_hash="0" * 64,
|
||||
created_at=_FIXTURE_CREATED_AT,
|
||||
summary=summary,
|
||||
)
|
||||
|
||||
|
||||
@@ -93,23 +104,64 @@ def test_persona_owner_edits_are_preserved() -> None:
|
||||
assert "HONESTY GATE" in PERSONA # the gate itself is intact
|
||||
|
||||
|
||||
def test_high_prompt_carries_relevance_marker_and_full_documents() -> None:
|
||||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||||
def test_high_prompt_carries_relevance_marker_and_summary_block() -> None:
|
||||
"""Phase 118 (LOCKED A6): the grounded turn seeds the document's
|
||||
SUMMARY — the full content never reaches the prompt (the ``read``
|
||||
tool is the only full-text path)."""
|
||||
doc = _doc(
|
||||
"kubernetes.md",
|
||||
"Talos Linux on three nodes. FULL_CONTENT_SENTINEL_987654",
|
||||
"Kubernetes Homelab Cluster",
|
||||
summary="A Talos Linux cluster on three nodes.",
|
||||
)
|
||||
prompt = build_high_prompt([doc])
|
||||
assert "<relevance>HIGH</relevance>" in prompt
|
||||
assert "DEFLECT_MODE" not in prompt
|
||||
assert "<documents>" in prompt and "</documents>" in prompt
|
||||
assert 'path="kubernetes.md"' in prompt
|
||||
assert "Talos Linux on three nodes." in prompt
|
||||
assert "A Talos Linux cluster on three nodes." in prompt # the summary
|
||||
assert "FULL_CONTENT_SENTINEL_987654" not in prompt # never the content
|
||||
assert "Talos Linux on three nodes." not in prompt # nor the full sentence
|
||||
assert "HONESTY GATE" in prompt # persona intact
|
||||
|
||||
|
||||
def test_high_prompt_lists_multiple_documents_in_order() -> None:
|
||||
a = _doc("a.md", "CONTENT_A", "Title A")
|
||||
b = _doc("b.md", "CONTENT_B", "Title B")
|
||||
a = _doc("a.md", "CONTENT_A", "Title A", summary="Summary A.")
|
||||
b = _doc("b.md", "CONTENT_B", "Title B", summary="Summary B.")
|
||||
prompt = build_high_prompt([a, b])
|
||||
assert prompt.index("CONTENT_A") < prompt.index("CONTENT_B")
|
||||
assert prompt.index("Summary A.") < prompt.index("Summary B.")
|
||||
assert 'title="Title B"' in prompt
|
||||
assert "CONTENT_A" not in prompt and "CONTENT_B" not in prompt
|
||||
|
||||
|
||||
def test_high_prompt_seeds_summaries_of_all_suggested_docs() -> None:
|
||||
"""The phase-118 completion pin: a grounded prompt built from five
|
||||
summary-bearing documents contains ALL FIVE summaries + the intro,
|
||||
and ZERO full-content characters (the full texts are not in the
|
||||
prompt at all)."""
|
||||
docs = [
|
||||
_doc(
|
||||
f"doc{i}.md",
|
||||
f"FULL_CONTENT_SENTINEL_{i} " + "x" * 100,
|
||||
f"Title {i}",
|
||||
summary=f"SUMMARY_{i} of the document.",
|
||||
)
|
||||
for i in range(5)
|
||||
]
|
||||
prompt = build_high_prompt(docs)
|
||||
assert SUGGEST_INTRO in prompt
|
||||
for i in range(5):
|
||||
assert f"SUMMARY_{i} of the document.\n" in prompt
|
||||
assert f"FULL_CONTENT_SENTINEL_{i}" not in prompt
|
||||
assert f'path="doc{i}.md"' in prompt
|
||||
# One block per document, in the given order.
|
||||
assert prompt.count("<document ") == 5
|
||||
positions = [prompt.index(f"SUMMARY_{i}") for i in range(5)]
|
||||
assert positions == sorted(positions)
|
||||
# The intro leads the section: <documents> → intro → first block.
|
||||
i_open = prompt.index("<documents>")
|
||||
i_block = prompt.index("<document ")
|
||||
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n\n"
|
||||
|
||||
|
||||
def test_high_prompt_without_documents_stays_honest() -> None:
|
||||
@@ -143,7 +195,9 @@ def test_zero_note_prompt_is_byte_identical_to_pre_steering() -> None:
|
||||
"""Phase 15 contract: with no steering notes the prompt is exactly what
|
||||
it was before the <tuning> section existed. (Phase 37: the HIGH prompt
|
||||
additionally carries the ``<tools>`` section after the mode body — the
|
||||
fixtures account for it; the LOW prompt is untouched.)"""
|
||||
fixtures account for it; phase 118: the ``<documents>`` section leads
|
||||
with the ``SUGGEST_INTRO`` line — the fixture accounts for it; the
|
||||
LOW prompt is untouched.)"""
|
||||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||||
block = (
|
||||
'<document source="Homelab" path="kubernetes.md" '
|
||||
@@ -152,7 +206,14 @@ def test_zero_note_prompt_is_byte_identical_to_pre_steering() -> None:
|
||||
"</document>"
|
||||
)
|
||||
assert build_high_prompt([doc]) == (
|
||||
_base("HIGH") + "\n<documents>\n" + block + "\n</documents>" + "\n" + TOOLS_SECTION
|
||||
_base("HIGH")
|
||||
+ "\n<documents>\n"
|
||||
+ SUGGEST_INTRO
|
||||
+ "\n\n"
|
||||
+ block
|
||||
+ "\n</documents>"
|
||||
+ "\n"
|
||||
+ TOOLS_SECTION
|
||||
)
|
||||
# Phase 71: the LOW prompt carries the owner-permitted plain-text
|
||||
# line after the DEFLECT_MODE sentence (the marker-keying contract
|
||||
@@ -257,6 +318,86 @@ def test_tools_section_phase95_read_truncation_clause() -> None:
|
||||
assert "Very large documents are capped" in build_high_prompt([doc])
|
||||
|
||||
|
||||
def test_tools_section_phase118_summary_seed_read_clause() -> None:
|
||||
"""Phase 118 (task 04, A6): the ``read`` clause is rewritten for
|
||||
the summary-seed mode — the ``<documents>`` section holds
|
||||
SUMMARIES (a suggested document's full text is not in the prompt
|
||||
until ``read`` adds it); do not re-read an already-read document
|
||||
(answer from the text already in the prompt); if the user asks to
|
||||
open or read a suggested document, ``read`` it. The phase-72 "do
|
||||
not call ``read`` for a ``<documents>`` document" copy is retired.
|
||||
The ``ls`` clause (phase 94 drill-down contract), the ``grep``
|
||||
clause, and the discipline rules stay byte-identical — the
|
||||
prompt lock's prefix/suffix anchors survive (see
|
||||
``test_prompt_lock``)."""
|
||||
# The new summary-seed copy (pinned byte-for-byte).
|
||||
assert (
|
||||
"The <documents> section holds SUMMARIES — the full text of a "
|
||||
"suggested document is not in your prompt until you `read` it. "
|
||||
"Do not re-read a document you have already read — its full "
|
||||
"text is already in your prompt; answer directly from it. If "
|
||||
"the user asks you to open or read a suggested document, "
|
||||
"`read` it — that is the point of the section."
|
||||
) in TOOLS_SECTION
|
||||
# The read identity handoff now also names the <documents> summary
|
||||
# blocks (the combined identity is shown there), keeping the
|
||||
# "including the source name" contract.
|
||||
assert (
|
||||
"exactly as shown in the `ls` output — including the source "
|
||||
"name — or in the <documents> summary blocks — adding its "
|
||||
"full content to your context"
|
||||
) in TOOLS_SECTION
|
||||
# The retired pre-phase-118 copy is gone.
|
||||
assert "Do not call `read` for a document already shown in" not in TOOLS_SECTION
|
||||
assert "even when the user asks you to open or read it" not in TOOLS_SECTION
|
||||
# The ls clause (phase 94 drill-down contract) — byte-identical.
|
||||
assert (
|
||||
"`ls` lists the knowledge base as a tree, one level at a "
|
||||
"time: with no `path` it lists every synced source with its "
|
||||
"document count and a summary of its contents; with a source "
|
||||
"name (e.g. 'homelab') it lists that source's top-level "
|
||||
"folders and files; with a `source/folder` path it drills one "
|
||||
"level deeper. A listing shows only that level's subfolders "
|
||||
"and its own files — never the whole knowledge base in one "
|
||||
"call — and each folder line's summary says what the folder "
|
||||
"contains before you drill into it. File lines are `source: X "
|
||||
"| path: Y | title: Z`; to find one specific document without "
|
||||
"listing, use `grep`."
|
||||
) in TOOLS_SECTION
|
||||
# The grep clause — byte-identical.
|
||||
assert (
|
||||
"`grep` locates an exact string (case-insensitive) in the "
|
||||
"indexed documents and returns up to 20 matching "
|
||||
"`source/path:line: text` lines — a locator, not a "
|
||||
"context-adder: read the winner with `read`. A grep pattern "
|
||||
"is a plain substring, NEVER a regex — '.*' and '\\.' are "
|
||||
"literal text there; if such a pattern returns no matches, "
|
||||
"retry with the plain text you expect to see. For a normal "
|
||||
"search pass only `pattern` — its optional `path` argument "
|
||||
"limits the search to one document you already know, by the "
|
||||
"same combined `source/path` string; never a source name — a "
|
||||
"bare document path (without the source name) will not "
|
||||
"resolve there either."
|
||||
) in TOOLS_SECTION
|
||||
# The discipline rules — byte-identical.
|
||||
assert (
|
||||
"Make exactly one tool call per reply — a reply carrying two "
|
||||
"tool calls runs only the first, the second is discarded — "
|
||||
"and wait for the result before the next call. Never repeat a "
|
||||
"call that was refused or already succeeded — the refusal "
|
||||
"already told you the correct form. Answer as soon as you "
|
||||
"have what you need."
|
||||
) in TOOLS_SECTION
|
||||
# The new read clause rides the built HIGH prompt and never the
|
||||
# LOW (deflected) prompt.
|
||||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes", summary="K8S")
|
||||
high = build_high_prompt([doc])
|
||||
assert "The <documents> section holds SUMMARIES" in high
|
||||
assert "`read` it — that is the point of the section." in high
|
||||
low = build_deflect_prompt(["T1"])
|
||||
assert "The <documents> section holds SUMMARIES" not in low
|
||||
|
||||
|
||||
def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
|
||||
"""Phase 72/94: the contract clauses ride the HIGH prompt with the
|
||||
rest of the section and never leak into the LOW/deflection prompt
|
||||
@@ -273,20 +414,37 @@ def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
|
||||
assert "including the source name" not in low
|
||||
|
||||
|
||||
def test_documents_section_has_no_leading_intro() -> None:
|
||||
"""Phase 72, task 05 (gate iterations 2-3, reverted): the
|
||||
``<documents>`` section must NOT lead with an in-context reminder
|
||||
or name the ``<document>`` blocks — the live telemetry showed that
|
||||
copy primed the model to latch the seed documents' paths as
|
||||
``ls`` scopes (the incident turn regressed to a cap-reached loop
|
||||
on run 2 and re-trapped on run 5), and the reminder never flipped
|
||||
the seed-doc ``read``s (15/15 across gate runs 1-5). The section
|
||||
is exactly the document blocks again."""
|
||||
def test_documents_section_leads_with_the_suggest_intro() -> None:
|
||||
"""Phase 118, task 03: the ``<documents>`` section leads with the
|
||||
start-here :data:`SUGGEST_INTRO` line BEFORE the first block (the
|
||||
phase-15 ``_STEERING_INTRO`` / phase-31 ``_KB_INTRO`` precedent) —
|
||||
only when at least one block is present. This is the summary-as-
|
||||
starting-point framing, not the reverted phase-72 do-not-read
|
||||
reminder (that copy taught the seed texts as already-read context;
|
||||
with A6's summary seeding the blocks are starting points the model
|
||||
may ``read`` through to full text)."""
|
||||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||||
high = build_high_prompt([doc])
|
||||
i_open = high.index("<documents>")
|
||||
i_block = high.index('<document source="Homelab"')
|
||||
assert high[i_open : i_block] == "<documents>\n" # no intro line
|
||||
assert high[i_open : i_block] == f"<documents>\n{SUGGEST_INTRO}\n\n"
|
||||
# The intro occurs exactly once and sits inside the section.
|
||||
assert high.count(SUGGEST_INTRO) == 1
|
||||
assert high.index(SUGGEST_INTRO) > high.index("<documents>")
|
||||
assert high.index(SUGGEST_INTRO) < high.index("</documents>")
|
||||
|
||||
|
||||
def test_documents_section_without_blocks_has_no_intro() -> None:
|
||||
"""The intro rides only on present blocks: an empty ``<documents>``
|
||||
section is exactly the fallback line again (no intro, no blocks)."""
|
||||
high = build_high_prompt([])
|
||||
assert SUGGEST_INTRO not in high
|
||||
i_open = high.index("<documents>")
|
||||
i_close = high.index("</documents>")
|
||||
assert (
|
||||
high[i_open : i_close]
|
||||
== "<documents>\n(no documents matched — do not invent specifics)\n"
|
||||
)
|
||||
|
||||
|
||||
def test_tools_section_old_names_and_budget_copy_gone() -> None:
|
||||
@@ -314,6 +472,117 @@ def test_high_prompt_still_ends_with_tools_section() -> None:
|
||||
assert old not in prompt
|
||||
|
||||
|
||||
# ---------- phase 118 (task 03): the NULL-summary preview fallback (A5) ----------
|
||||
|
||||
|
||||
def _content_only_doc(content: str) -> Document:
|
||||
return _doc("kubernetes.md", content, "Kubernetes Homelab Cluster")
|
||||
|
||||
|
||||
def test_null_summary_falls_back_to_preview_plus_marker() -> None:
|
||||
"""A doc whose ``summary`` is None (a fail-soft import miss) seeds the
|
||||
first ``suggestion_preview_chars`` (default 400) content characters +
|
||||
the shared ``[…truncated…]`` marker on its own line — never the full
|
||||
content, never an LLM call."""
|
||||
content = "A" * 400 + "B" * 600 # 1000 chars
|
||||
prompt = build_high_prompt([_content_only_doc(content)])
|
||||
body_start = prompt.index('date="2024-06-15">\n') + len('date="2024-06-15">\n')
|
||||
body_end = prompt.index("\n</document>", body_start)
|
||||
assert prompt[body_start:body_end] == content[:400] + "\n" + TRUNCATION_MARKER
|
||||
assert "B" * 40 not in prompt # beyond the 400-char cut
|
||||
assert prompt.count(TRUNCATION_MARKER) == 1 # only the block's marker
|
||||
|
||||
|
||||
def test_whitespace_summary_uses_the_same_preview_fallback() -> None:
|
||||
for blank in ("", " ", "\n \t "):
|
||||
doc = _doc("kubernetes.md", "A" * 500 + "B" * 500, "T", summary=blank)
|
||||
prompt = build_high_prompt([doc])
|
||||
assert "A" * 400 in prompt
|
||||
assert "B" * 40 not in prompt
|
||||
assert TRUNCATION_MARKER in prompt
|
||||
|
||||
|
||||
def test_short_content_preview_is_the_whole_content_unmarked() -> None:
|
||||
"""Content at or under the cap rides whole — nothing was cut, so no
|
||||
marker (the marker signals truncation, not the fallback)."""
|
||||
for content in ("short body", "C" * 399, "D" * 400):
|
||||
prompt = build_high_prompt([_content_only_doc(content)])
|
||||
assert content + "\n</document>" in prompt
|
||||
assert TRUNCATION_MARKER not in prompt
|
||||
|
||||
|
||||
def test_summary_stripped_and_never_truncated_by_the_preview_cap() -> None:
|
||||
"""The summary body is the stripped ``doc.summary`` — even when it is
|
||||
longer than the preview cap (the cap bounds only the FALLBACK; a
|
||||
stored summary is trusted context, LOCKED A5)."""
|
||||
summary = "S" * 900
|
||||
content = "CONTENT_SENTINEL " + "x" * 50
|
||||
prompt = build_high_prompt([_doc("kubernetes.md", content, "T", summary=f" {summary}\n")])
|
||||
assert summary in prompt
|
||||
assert content not in prompt
|
||||
assert TRUNCATION_MARKER not in prompt
|
||||
|
||||
|
||||
def test_preview_cap_setting_is_honored_on_the_fallback_path(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""A non-default ``suggestion_preview_chars`` (``BOR_SUGGESTION_PREVIEW_CHARS``
|
||||
— the env mapping is pinned in :mod:`tests.unit.test_config`) cuts the
|
||||
preview at the setting's cap on the fallback path."""
|
||||
from app.rag import prompts as prompts_mod
|
||||
|
||||
monkeypatch.setattr(
|
||||
prompts_mod,
|
||||
"get_settings",
|
||||
lambda: Settings(_env_file=None, suggestion_preview_chars=10), # pyright: ignore[reportCallIssue]
|
||||
)
|
||||
content = "A" * 100 + "B" * 100
|
||||
prompt = build_high_prompt([_content_only_doc(content)])
|
||||
assert "A" * 10 in prompt
|
||||
assert "A" * 11 not in prompt
|
||||
assert TRUNCATION_MARKER in prompt
|
||||
|
||||
|
||||
def test_preview_fallback_never_reads_settings_for_summarized_docs(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""House pattern (``build_steering_section`` / ``build_kb_section``):
|
||||
the fallback cap is read from settings on the fallback path ONLY — a
|
||||
prompt built from summary-bearing docs makes no settings call for
|
||||
it (a dead ``get_settings`` must not break such a build)."""
|
||||
from app.rag import prompts as prompts_mod
|
||||
|
||||
doc = _doc("kubernetes.md", "CONTENT_SENTINEL", "T", summary="the summary")
|
||||
|
||||
def _dead() -> None: # pragma: no cover - must never be called
|
||||
raise AssertionError("get_settings() called for a summarized doc")
|
||||
|
||||
monkeypatch.setattr(prompts_mod, "get_settings", _dead)
|
||||
prompt = build_high_prompt([doc])
|
||||
assert "the summary\n</document>" in prompt
|
||||
assert "CONTENT_SENTINEL" not in prompt
|
||||
|
||||
|
||||
# ---------- phase 118: the LOW prompt stays byte-identical ----------
|
||||
|
||||
|
||||
def test_low_prompt_byte_identical_to_pre_task() -> None:
|
||||
"""LOCKED A8 surface: the deflection prompt is untouched by the
|
||||
summary seeding — byte-identical build on the same inputs (the full
|
||||
sha pin lives in :mod:`tests.unit.test_prompt_lock`)."""
|
||||
for titles, notes, kb in (
|
||||
(["T1", "T2"], None, None),
|
||||
(["T1"], ["be concise"], None),
|
||||
([], None, OVERVIEW),
|
||||
(["T1", "T2"], ["be concise"], OVERVIEW),
|
||||
):
|
||||
prompt = build_deflect_prompt(titles, notes=notes, kb_overview=kb)
|
||||
assert "DEFLECT_MODE" in prompt
|
||||
assert "<documents>" not in prompt
|
||||
assert SUGGEST_INTRO not in prompt
|
||||
assert "<tools>" not in prompt
|
||||
|
||||
|
||||
def test_relevance_placeholder_rejected_for_garbage() -> None:
|
||||
with pytest.raises(ValueError, match="HIGH or LOW"):
|
||||
_base("MEDIUM")
|
||||
@@ -401,7 +670,9 @@ def test_no_overview_prompt_is_byte_identical_to_pre_phase() -> None:
|
||||
every prompt is exactly what it was before the ``<knowledge_base>``
|
||||
section existed — with or without steering notes. (Phase 37: the HIGH
|
||||
prompt additionally carries the ``<tools>`` section after the mode
|
||||
body — the fixtures account for it; the LOW prompt is untouched.)"""
|
||||
body; phase 118: the ``<documents>`` section leads with the
|
||||
``SUGGEST_INTRO`` line — the fixtures account for both; the LOW
|
||||
prompt is untouched.)"""
|
||||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||||
block = (
|
||||
'<document source="Homelab" path="kubernetes.md" '
|
||||
@@ -409,7 +680,15 @@ def test_no_overview_prompt_is_byte_identical_to_pre_phase() -> None:
|
||||
"Talos Linux on three nodes.\n"
|
||||
"</document>"
|
||||
)
|
||||
docs_block = "\n<documents>\n" + block + "\n</documents>" + "\n" + TOOLS_SECTION
|
||||
docs_block = (
|
||||
"\n<documents>\n"
|
||||
+ SUGGEST_INTRO
|
||||
+ "\n\n"
|
||||
+ block
|
||||
+ "\n</documents>"
|
||||
+ "\n"
|
||||
+ TOOLS_SECTION
|
||||
)
|
||||
high_plain = _base("HIGH") + docs_block
|
||||
high_steered = _base("HIGH") + "\n" + build_steering_section(["be concise"]) + docs_block
|
||||
# Phase 71: the owner-permitted plain-text line is part of the
|
||||
|
||||
@@ -58,6 +58,7 @@ def _doc(
|
||||
title: str = "T",
|
||||
content: str = "CONTENT",
|
||||
created_at: datetime = CREATED_AT,
|
||||
summary: str | None = None,
|
||||
) -> Document:
|
||||
return Document(
|
||||
id=uuid.uuid4(),
|
||||
@@ -68,6 +69,7 @@ def _doc(
|
||||
content=content,
|
||||
content_hash="0" * 64,
|
||||
created_at=created_at,
|
||||
summary=summary,
|
||||
)
|
||||
|
||||
|
||||
@@ -171,6 +173,32 @@ def test_high_block_date_always_present_for_every_document() -> None:
|
||||
assert prompt.count("<document ") == prompt.count(' date="')
|
||||
|
||||
|
||||
def test_high_block_date_survives_the_summary_body_change() -> None:
|
||||
"""Phase 118 (task 03): the block BODY became the document's summary
|
||||
(never the full content) — the D5 identity attributes, including
|
||||
``date`` after ``title``, survive byte-identical around the new
|
||||
body (the E2E mock's block parser keys off exactly these)."""
|
||||
doc = _doc(
|
||||
source="S",
|
||||
path="P",
|
||||
title="T",
|
||||
content="FULL_CONTENT_SENTINEL_42",
|
||||
created_at=CREATED_AT,
|
||||
summary="The stored summary.",
|
||||
)
|
||||
prompt = build_high_prompt([doc])
|
||||
block = (
|
||||
f'<document source="S" path="P" title="T" date="{DATE}">\n'
|
||||
"The stored summary.\n"
|
||||
"</document>"
|
||||
)
|
||||
assert block in prompt
|
||||
# Attribute order pinned: date directly after title, body after.
|
||||
assert f'title="T" date="{DATE}">' in prompt
|
||||
# The full content stays out (A6) — only the summary rides the block.
|
||||
assert "FULL_CONTENT_SENTINEL_42" not in prompt
|
||||
|
||||
|
||||
# --------------------------------------------------------------------
|
||||
# The deflection prompt — byte-identical to the pre-phase text (A8)
|
||||
# --------------------------------------------------------------------
|
||||
|
||||
@@ -11,12 +11,16 @@ from types import SimpleNamespace
|
||||
|
||||
import pytest
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag import retriever
|
||||
from app.rag.retriever import (
|
||||
TRUNCATION_MARKER,
|
||||
RetrievedChunk,
|
||||
select_documents,
|
||||
select_documents_tiered,
|
||||
select_related,
|
||||
select_suggested,
|
||||
)
|
||||
|
||||
|
||||
@@ -258,6 +262,231 @@ def test_select_documents_wrapper_is_legacy_tiering() -> None:
|
||||
)[0]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 118 — select_suggested: the top-N "start here" tier, NO floor (A3)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_suggested_rank_order_by_first_seen_chunk() -> None:
|
||||
"""The SAME stable walk as ``select_documents_tiered``: a document's
|
||||
rank is fixed by its FIRST seen chunk in score-descending order — a
|
||||
doc whose best chunk appears later in the input list still ranks
|
||||
where that chunk falls."""
|
||||
a = _doc("a.md", "A" * 50)
|
||||
b = _doc("b.md", "B" * 50)
|
||||
c = _doc("c.md", "C" * 50)
|
||||
chunks = [
|
||||
_chunk(a, 0.4, position=0), # a's weak chunk comes first
|
||||
_chunk(b, 0.8),
|
||||
_chunk(a, 0.9, position=2), # a's best chunk comes last
|
||||
_chunk(c, 0.5),
|
||||
]
|
||||
out = select_suggested(chunks, n=5)
|
||||
assert [d.path for d in out] == ["a.md", "b.md", "c.md"]
|
||||
# The rows carry the full content byte-identical (A6: the content is
|
||||
# what ``read`` serves later — never truncated).
|
||||
assert out[0].content == "A" * 50
|
||||
assert TRUNCATION_MARKER not in out[0].content
|
||||
|
||||
|
||||
def test_suggested_dedupes_multiple_chunks_to_one_row() -> None:
|
||||
"""Multiple hit chunks of one document collapse to a single row."""
|
||||
a = _doc("a.md", "A" * 50)
|
||||
b = _doc("b.md", "B" * 50)
|
||||
chunks = [
|
||||
_chunk(a, 0.2),
|
||||
_chunk(b, 0.7),
|
||||
_chunk(a, 0.9, position=2),
|
||||
_chunk(a, 0.5),
|
||||
]
|
||||
out = select_suggested(chunks, n=5)
|
||||
assert [d.path for d in out] == ["a.md", "b.md"] # one row per document
|
||||
assert out[0] is a
|
||||
|
||||
|
||||
def test_suggested_caps_at_n_in_rank_order() -> None:
|
||||
docs_in = [_doc(f"d{i}.md", "X" * 20) for i in range(7)]
|
||||
chunks = [_chunk(d, 0.9 - 0.1 * i) for i, d in enumerate(docs_in)]
|
||||
out = select_suggested(chunks, n=3)
|
||||
assert [d.path for d in out] == ["d0.md", "d1.md", "d2.md"]
|
||||
|
||||
|
||||
def test_suggested_default_cap_is_the_settings_value(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""``n`` omitted → ``BOR_SUGGESTED_DOCS`` caps the walk — default 5
|
||||
(LOCKED A3, the top-5 "start here" directive, TODO L3) — and the cap
|
||||
is the setting's LIVE value, not a frozen constant."""
|
||||
docs_in = [_doc(f"d{i}.md", "X" * 20) for i in range(7)]
|
||||
chunks = [_chunk(d, 0.9 - 0.1 * i) for i, d in enumerate(docs_in)]
|
||||
settings = Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
||||
assert settings.suggested_docs == 5 # the production default
|
||||
monkeypatch.setattr(retriever, "get_settings", lambda: settings)
|
||||
assert [d.path for d in select_suggested(chunks)] == [
|
||||
f"d{i}.md" for i in range(5)
|
||||
]
|
||||
small = Settings(_env_file=None, suggested_docs=2) # pyright: ignore[reportCallIssue]
|
||||
monkeypatch.setattr(retriever, "get_settings", lambda: small)
|
||||
assert [d.path for d in select_suggested(chunks)] == ["d0.md", "d1.md"]
|
||||
|
||||
|
||||
def test_suggested_never_filters_on_cosine_floor() -> None:
|
||||
"""NO floor (LOCKED A3): a lexical-only hit (cosine 0.0 by
|
||||
construction) is a suggestion when it ranks — the contrast pin
|
||||
against ``select_documents_tiered``'s floored cited tier on the SAME
|
||||
input, which demotes it to the related tier."""
|
||||
a = _doc("a.md", "A" * 50)
|
||||
b = _doc("b.md", "B" * 50)
|
||||
lexical_only = RetrievedChunk(
|
||||
chunk_id=uuid.uuid4(),
|
||||
position=0,
|
||||
content="X" * 10,
|
||||
score=0.9, # top fused rank (the FTS hit)
|
||||
document=a,
|
||||
cosine=0.0, # no vector rank — lexical-only
|
||||
fts_hit=True,
|
||||
)
|
||||
chunks = [lexical_only, _cos_chunk(b, 0.8, 0.5)]
|
||||
# Suggested: the floor never filters — a leads, b follows in rank order.
|
||||
assert [d.path for d in select_suggested(chunks, n=5)] == ["a.md", "b.md"]
|
||||
# Contrast: the same input through the phase-113 cited tier — the
|
||||
# 0.35 usefulness bar demotes the lexical-only doc to related.
|
||||
cited, related = select_documents_tiered(chunks, n=2, floor=0.35, related_cap=2)
|
||||
assert [d.path for d in cited] == ["b.md"]
|
||||
assert [d.path for d in related] == ["a.md"]
|
||||
|
||||
|
||||
def test_suggested_tie_break_inherited_from_fused_order() -> None:
|
||||
"""Equal fused scores keep the input (fused) order — the stable
|
||||
score-only walk inherits ``fuse()``'s (-score, -cosine, path,
|
||||
position) tie-break; the selector never re-sorts it away."""
|
||||
a = _doc("a.md", "A" * 50)
|
||||
b = _doc("b.md", "B" * 50)
|
||||
chunks = [_cos_chunk(a, 0.7, 0.5), _cos_chunk(b, 0.7, 0.4)] # a wins on cosine
|
||||
assert [d.path for d in select_suggested(chunks, n=5)] == ["a.md", "b.md"]
|
||||
# A FULL tie (score AND cosine): the input position — ``fuse()``'s
|
||||
# path/position tie-break already applied — decides. ``m.md`` sorts
|
||||
# AFTER ``b2.md`` alphabetically, so any re-sort by path would flip
|
||||
# the order; the fused input order must win.
|
||||
m = _doc("m.md", "M" * 50)
|
||||
b2 = _doc("b2.md", "B" * 50)
|
||||
chunks = [_cos_chunk(m, 0.7, 0.4), _cos_chunk(b2, 0.7, 0.4)]
|
||||
assert [d.path for d in select_suggested(chunks, n=5)] == ["m.md", "b2.md"]
|
||||
|
||||
|
||||
def test_suggested_empty_chunks_yield_no_documents() -> None:
|
||||
assert select_suggested([], n=5) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 118, task 05 — select_related: the rank-6+ tier after the suggested set
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_related_walk_order_after_excluded_set() -> None:
|
||||
"""The SAME stable score-descending walk as ``select_suggested``:
|
||||
a document's rank is fixed by its FIRST seen chunk; documents in
|
||||
*excluded_ids* (the suggested set) are skipped and the rest come
|
||||
back in rank order."""
|
||||
a = _doc("a.md", "A" * 50)
|
||||
b = _doc("b.md", "B" * 50)
|
||||
c = _doc("c.md", "C" * 50)
|
||||
d = _doc("d.md", "D" * 50)
|
||||
chunks = [
|
||||
_chunk(a, 0.4, position=0), # a's weak chunk comes first
|
||||
_chunk(b, 0.8),
|
||||
_chunk(a, 0.9, position=2), # a's best chunk last — a ranks first
|
||||
_chunk(c, 0.5),
|
||||
_chunk(d, 0.3),
|
||||
]
|
||||
out = select_related(chunks, {a.id, b.id}, cap=2)
|
||||
assert [x.path for x in out] == ["c.md", "d.md"]
|
||||
|
||||
|
||||
def test_related_skips_excluded_documents() -> None:
|
||||
"""Every document in *excluded_ids* is skipped, even when it would
|
||||
rank inside the cap — the suggested set never rides the related row."""
|
||||
a = _doc("a.md", "A" * 50)
|
||||
b = _doc("b.md", "B" * 50)
|
||||
c = _doc("c.md", "C" * 50)
|
||||
chunks = [_chunk(a, 0.9), _chunk(b, 0.8), _chunk(c, 0.5)]
|
||||
out = select_related(chunks, {a.id, b.id}, cap=5)
|
||||
assert out == [c]
|
||||
# Every doc excluded → empty, even with room left in the cap.
|
||||
assert select_related(chunks, {a.id, b.id, c.id}, cap=5) == []
|
||||
|
||||
|
||||
def test_related_caps_at_cap_in_rank_order() -> None:
|
||||
"""The phase-118 turn wiring on 9 docs: suggested = the top 5,
|
||||
related = rank 6–7 (capped at 2), disjoint from the suggested set."""
|
||||
docs_in = [_doc(f"d{i}.md", "X" * 20) for i in range(9)]
|
||||
chunks = [_chunk(d, 0.9 - 0.1 * i) for i, d in enumerate(docs_in)]
|
||||
suggested = select_suggested(chunks, n=5)
|
||||
out = select_related(chunks, {d.id for d in suggested}, cap=2)
|
||||
assert [x.path for x in out] == ["d5.md", "d6.md"] # rank 6–7, capped
|
||||
suggested_paths = {d.path for d in suggested}
|
||||
assert suggested_paths.isdisjoint({x.path for x in out})
|
||||
|
||||
|
||||
def test_related_cap_zero_yields_empty() -> None:
|
||||
"""cap=0 is the kill switch (related_max_docs=0): no related docs,
|
||||
the pre-phase-113 visibility."""
|
||||
a = _doc("a.md", "A" * 50)
|
||||
b = _doc("b.md", "B" * 50)
|
||||
c = _doc("c.md", "C" * 50)
|
||||
chunks = [_chunk(a, 0.9), _chunk(b, 0.8), _chunk(c, 0.5)]
|
||||
assert select_related(chunks, {a.id}, cap=0) == []
|
||||
|
||||
|
||||
def test_related_never_filters_on_cosine_floor() -> None:
|
||||
"""NO floor: a lexical-only (cosine 0.0) doc is related when it
|
||||
ranks after the excluded set — the related tier is visibility, not
|
||||
citation (phase 118 applies no cosine floor to it)."""
|
||||
a = _doc("a.md", "A" * 50)
|
||||
b = _doc("b.md", "B" * 50)
|
||||
lexical_only = RetrievedChunk(
|
||||
chunk_id=uuid.uuid4(),
|
||||
position=0,
|
||||
content="X" * 10,
|
||||
score=0.9, # top fused rank (the FTS hit)
|
||||
document=a,
|
||||
cosine=0.0, # no vector rank — lexical-only
|
||||
fts_hit=True,
|
||||
)
|
||||
chunks = [lexical_only, _cos_chunk(b, 0.8, 0.5)]
|
||||
out = select_related(chunks, set(), cap=5)
|
||||
assert [x.path for x in out] == ["a.md", "b.md"]
|
||||
|
||||
|
||||
def test_related_dedupes_multiple_chunks_to_one_row() -> None:
|
||||
"""Multiple hit chunks of one document collapse to a single row
|
||||
(first-seen-chunk rank, dedupe by document.id — the shared walk)."""
|
||||
a = _doc("a.md", "A" * 50)
|
||||
b = _doc("b.md", "B" * 50)
|
||||
chunks = [_chunk(a, 0.2), _chunk(b, 0.7), _chunk(a, 0.9, position=2)]
|
||||
out = select_related(chunks, set(), cap=5)
|
||||
assert [x.path for x in out] == ["a.md", "b.md"] # one row per document
|
||||
assert out[0] is a
|
||||
|
||||
|
||||
def test_related_tie_break_inherited_from_fused_order() -> None:
|
||||
"""Equal fused scores keep the input (fused) order — the stable
|
||||
score-only walk inherits ``fuse()``'s (-score, -cosine, path,
|
||||
position) tie-break; the selector never re-sorts it away."""
|
||||
a = _doc("a.md", "A" * 50)
|
||||
b = _doc("b.md", "B" * 50)
|
||||
chunks = [_cos_chunk(a, 0.7, 0.5), _cos_chunk(b, 0.7, 0.4)] # a wins on cosine
|
||||
assert [x.path for x in select_related(chunks, set(), cap=5)] == ["a.md", "b.md"]
|
||||
m = _doc("m.md", "M" * 50)
|
||||
b2 = _doc("b2.md", "B" * 50)
|
||||
chunks = [_cos_chunk(m, 0.7, 0.4), _cos_chunk(b2, 0.7, 0.4)] # full tie
|
||||
assert [x.path for x in select_related(chunks, set(), cap=5)] == ["m.md", "b2.md"]
|
||||
|
||||
|
||||
def test_related_empty_chunks_yield_no_documents() -> None:
|
||||
assert select_related([], set(), cap=5) == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Hybrid retrieval (A7): RRF fusion + lexical tsquery
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -331,7 +331,16 @@ def test_extra_keys_still_forbidden() -> None:
|
||||
def test_minimal_message_still_validates() -> None:
|
||||
"""Optional keys may be ABSENT exactly as pre-phase-83."""
|
||||
msg = ChatMessage.model_validate({"who": "user", "text": "hi"})
|
||||
assert (msg.sources, msg.deflected, msg.suggestions, msg.thinking, msg.tools, msg.stopped) == (
|
||||
assert (
|
||||
msg.sources,
|
||||
msg.related,
|
||||
msg.deflected,
|
||||
msg.suggestions,
|
||||
msg.thinking,
|
||||
msg.tools,
|
||||
msg.stopped,
|
||||
) == (
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
None,
|
||||
@@ -358,6 +367,7 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
|
||||
"who": "user",
|
||||
"text": "How did I install k3s on the new node?",
|
||||
"sources": None,
|
||||
"related": None,
|
||||
"deflected": None,
|
||||
"suggestions": None,
|
||||
"thinking": None,
|
||||
@@ -371,6 +381,14 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
|
||||
{"source": "Homelab", "path": "kubernetes.md", "title": "Kubernetes Cluster"},
|
||||
{"source": "Deployments", "path": "k3s-install.md", "title": "k3s Install Notes"},
|
||||
],
|
||||
# Phase 113 related-doc tier — an ACCEPTED key (the phase-113
|
||||
# omission of this field made extra="forbid" 422 every
|
||||
# done-time auto-save carrying it, so grounded turns' brain
|
||||
# messages never persisted — the A2 quiet failure swallowed
|
||||
# the 422). Round-trips like sources.
|
||||
"related": [
|
||||
{"source": "Homelab", "path": "traefik.md", "title": "Traefik Notes"}
|
||||
],
|
||||
"deflected": False,
|
||||
"suggestions": None,
|
||||
"thinking": "The kubernetes doc covers the cluster layout…",
|
||||
@@ -398,6 +416,7 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
|
||||
"who": "user",
|
||||
"text": "And what ports does Traefik expose?",
|
||||
"sources": None,
|
||||
"related": None,
|
||||
"deflected": None,
|
||||
"suggestions": None,
|
||||
"thinking": None,
|
||||
@@ -408,6 +427,7 @@ def test_realistic_bor_chat_v1_payload_round_trips() -> None:
|
||||
"who": "brain",
|
||||
"text": "Traefik exposes 80/443 on every node.",
|
||||
"sources": None,
|
||||
"related": None,
|
||||
"deflected": None,
|
||||
"suggestions": ["What is the Traefik dashboard password?"],
|
||||
"thinking": None,
|
||||
@@ -433,6 +453,7 @@ def test_realistic_payload_round_trips_through_update_model() -> None:
|
||||
"who": "brain",
|
||||
"text": "answer",
|
||||
"sources": [_source_ref()],
|
||||
"related": None,
|
||||
"deflected": None,
|
||||
"suggestions": ["follow-up?"],
|
||||
"thinking": "scratchpad",
|
||||
|
||||
@@ -1,13 +1,15 @@
|
||||
"""Unit: the phase-113 source-chip-quality contract (TODO L5 + L2c —
|
||||
"the 2nd chip is often noise the answer never used").
|
||||
|
||||
Phase 113 demotes sub-floor hits out of the citation surface: the
|
||||
done frame carries the cited tier in ``sources`` (rendered by
|
||||
``appendSources`` as ``.source-chip`` pills, UNCHANGED) and the
|
||||
related tier in ``related`` (rendered by the NEW ``appendRelated`` as
|
||||
the de-emphasized labeled row — ``.related-doc`` links, never
|
||||
``.source-chip``). A deflected turn carries ``sources: []`` → zero
|
||||
chips; its weak hits live in the related row only.
|
||||
Phase 118 re-tiers the same frame (LOCKED A3/A4): ``sources`` carries
|
||||
the suggested tier (top-5, NO floor) + the agent-read docs (deduped)
|
||||
— rendered by ``appendSources`` as ``.source-chip`` pills, UNCHANGED —
|
||||
and ``related`` carries rank 6+ after the suggested set (rendered by
|
||||
``appendRelated`` as the de-emphasized labeled row — ``.related-doc``
|
||||
links, never ``.source-chip``). A deflected turn carries
|
||||
``sources: []`` → zero chips; its weak hits are the suggested tier
|
||||
(the durable record), and with ≤5 retrieved docs the related row is
|
||||
empty.
|
||||
|
||||
This module pins the STATIC SOURCES the UI contract stands on, in the
|
||||
house source-pin pattern (the test_chip_sizing_question_cap.py
|
||||
@@ -26,24 +28,23 @@ house source-pin pattern (the test_chip_sizing_question_cap.py
|
||||
|
||||
1. **both docs weak** ("What is the capital of Mongolia?" →
|
||||
``Trooper_Nagraz.pl`` + ``Trooper_Begzei.pl``, both unrelated) —
|
||||
cited tier empty, the weak hits ride the related tier, capped at
|
||||
``related_max_docs``; the FTS hit without vector corroboration
|
||||
stays LOW (the A8-revised "Mongolia" case);
|
||||
the weak hits are SUGGESTED (no floor, A3; the durable record),
|
||||
nothing left for the related tier; the FTS hit without vector
|
||||
corroboration stays LOW (the A8-revised "Mongolia" case);
|
||||
2. **one strong + one weak** (the phase-gate question answered from
|
||||
``brain-of-reese/.agents/validate.sh``; the 2nd chip
|
||||
``ServMon/README.md`` unused) — exactly ONE cited ref, the weak
|
||||
doc in ``related``;
|
||||
``brain-of-reese/.agents/validate.sh``; ``ServMon/README.md``
|
||||
alongside) — BOTH suggested (no floor) ⇒ both cited refs (A4),
|
||||
no related tier;
|
||||
3. **the Nagraz case** (``Trooper_Nagraz.pl`` strong,
|
||||
``Trooper_Byzin.pl`` weak — same shape, different fixtures);
|
||||
4. **the meta/history question** (no doc clears the bar, the agent
|
||||
reads nothing — chips ``app/api/suggestions.py`` +
|
||||
``108_history_wire_check/00_phase.md``, neither used) — pinned on
|
||||
the DONE FRAME (endpoint-level, fake retriever/LLM/session): the
|
||||
frame is row-only — ``sources: []`` (the UI's chip list — zero
|
||||
chips) + the weak hits in ``related``;
|
||||
5. **the agent-read exemption** (LOCKED A2): a below-floor doc the
|
||||
agent ``read`` via the tool joins ``sources`` (cited, last) and is
|
||||
excluded from ``related``.
|
||||
4. **the meta/history question** (the agent reads nothing —
|
||||
``app/api/suggestions.py`` + ``108_history_wire_check/00_phase.md``
|
||||
alongside) — pinned on the DONE FRAME (endpoint-level, fake
|
||||
retriever/LLM/session): ``sources: []`` (zero chips) and an EMPTY
|
||||
related row (both weak docs are suggested, ≤5 docs retrieved);
|
||||
5. **the agent-read exemption** (LOCKED A4): a rank-6+ (related-
|
||||
tier) doc the agent ``read`` via the tool joins ``sources``
|
||||
(cited, last) and is excluded from ``related``.
|
||||
|
||||
The browser behavior (chip counts on a single-source question, zero
|
||||
chips on a deflected turn) is E2E-gated by
|
||||
@@ -411,14 +412,18 @@ def _shape_settings() -> Settings:
|
||||
"""The PRODUCTION calibration (the code defaults, explicit) — the
|
||||
four shapes were observed LIVE under this threshold/floor pair.
|
||||
``_env_file=None`` keeps the mock-calibrated values from
|
||||
``tests/conftest.py`` (and any local ``.env``) out of the pin."""
|
||||
``tests/conftest.py`` (and any local ``.env``) out of the pin.
|
||||
``source_usefulness_floor`` / ``top_n_docs`` are legacy phase-113
|
||||
settings — phase 118 retired their seeding role (A6; ``plan_turn``
|
||||
never consults them), they are carried here for completeness."""
|
||||
return Settings(
|
||||
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||
relevance_threshold=0.62,
|
||||
lexical_support_floor=0.35,
|
||||
source_usefulness_floor=0.35, # LOCKED A2 default
|
||||
related_max_docs=2, # LOCKED A4 default
|
||||
top_n_docs=2, # the ceiling — never a quota (LOCKED A2)
|
||||
source_usefulness_floor=0.35, # retired by phase 118 (A6) — not consulted
|
||||
related_max_docs=2, # the rank-6+ row cap (LOCKED A4)
|
||||
top_n_docs=2, # retired by phase 118 (A6) — not consulted
|
||||
suggested_docs=5, # the "start here" cap (LOCKED A3)
|
||||
)
|
||||
|
||||
|
||||
@@ -461,12 +466,12 @@ def _chunk(
|
||||
def test_shape_1_mongolia_both_docs_weak_cite_nothing() -> None:
|
||||
"""Observed shape 1 (TODO L110–113): "What is the capital of
|
||||
Mongolia?" → chips ``Trooper_Nagraz.pl`` + ``Trooper_Begzei.pl``,
|
||||
BOTH unrelated. Both below the bar: the cited tier is EMPTY (zero
|
||||
citation chips) and the weak hits ride the related tier — in rank
|
||||
order, capped at ``related_max_docs`` (the 3rd weak doc drops out).
|
||||
The FTS hit without vector corroboration (0.20 < the 0.35 lexical
|
||||
floor) stays LOW — the A8-revised "Mongolia" case; the weak content
|
||||
never reaches the LLM."""
|
||||
BOTH unrelated. Phase 118 (A3): the floor never filters — the weak
|
||||
hits are the SUGGESTED tier (the durable record's input), and with
|
||||
three retrieved docs nothing is left for the related tier (rank
|
||||
6+). The FTS hit without vector corroboration (0.20 < the 0.35
|
||||
lexical floor) stays LOW — the A8-revised "Mongolia" case; the weak
|
||||
content never reaches the LLM (titles only)."""
|
||||
nagraz = _doc("scripts", "scripts/Trooper_Nagraz.pl", "Trooper_Nagraz.pl",
|
||||
"NAGRAZ_PL_CONTENT")
|
||||
begzei = _doc("scripts", "scripts/Trooper_Begzei.pl", "Trooper_Begzei.pl",
|
||||
@@ -476,64 +481,76 @@ def test_shape_1_mongolia_both_docs_weak_cite_nothing() -> None:
|
||||
chunks = [
|
||||
_chunk(nagraz, 0.033, cosine=0.20, fts_hit=True), # rank 1, lexical hit
|
||||
_chunk(begzei, 0.031, cosine=0.12),
|
||||
_chunk(third, 0.030, cosine=0.10), # below the cap — related drops it
|
||||
_chunk(third, 0.030, cosine=0.10),
|
||||
]
|
||||
plan = chat_api.plan_turn(chunks, _shape_settings())
|
||||
assert plan.deflected is True # 0.20 < 0.62 AND 0.20 < the 0.35 lex floor
|
||||
assert plan.docs == [] # NO citation slot below the bar
|
||||
assert [d.title for d in plan.related_docs] == [
|
||||
# No floor (A3): the weak hits are suggested, in rank order (≤5).
|
||||
assert [d.title for d in plan.suggested_docs] == [
|
||||
"Trooper_Nagraz.pl",
|
||||
"Trooper_Begzei.pl",
|
||||
] # rank order, capped at related_max_docs (2)
|
||||
assert len(plan.related_docs) <= 2
|
||||
"Trooper_Third.pl",
|
||||
]
|
||||
assert plan.related_docs == [] # no rank-6+ doc among 3 retrieved
|
||||
# The LOW prompt is titles only — none of the weak content is sent.
|
||||
assert "NAGRAZ_PL_CONTENT" not in plan.system_prompt
|
||||
assert "Trooper_Nagraz.pl" in plan.system_prompt # weak-hit titles carried
|
||||
assert plan.suggestions # the "Maybe try" chips are unchanged
|
||||
|
||||
|
||||
def test_shape_2_validate_sh_strong_plus_unused_second_chip() -> None:
|
||||
def test_shape_2_validate_sh_strong_plus_weak_second_suggested() -> None:
|
||||
"""Observed shape 2 (TODO L114–116): the phase-gate question is
|
||||
answered from ``brain-of-reese/.agents/validate.sh`` — the 2nd chip
|
||||
``ServMon/README.md`` was NEVER used. The strong doc clears the bar
|
||||
and takes the only cited slot (top_n_docs is a ceiling, not a
|
||||
quota); the weak 2nd doc demotes to related — never a citation.
|
||||
The HIGH prompt carries the cited content only."""
|
||||
answered from ``brain-of-reese/.agents/validate.sh`` with
|
||||
``ServMon/README.md`` retrieved alongside (weak cosine). Phase 118
|
||||
(A3): the floor never filters — the weak 2nd doc is SUGGESTED too
|
||||
(both docs seed the HIGH prompt as summaries; the A5 fallback
|
||||
carries the short fixture content whole), and A4 makes both
|
||||
citation refs on the done frame — the "unused 2nd chip" is the
|
||||
phase-113 shape, retired by the owner directive."""
|
||||
validate = _doc("brain-of-reese", ".agents/validate.sh", "validate.sh",
|
||||
"VALIDATE_SH_CONTENT")
|
||||
servmon = _doc("ServMon", "README.md", "ServMon README",
|
||||
"SERVMON_README_CONTENT")
|
||||
chunks = [
|
||||
_chunk(validate, 0.90, cosine=0.70), # clears threshold AND bar
|
||||
_chunk(validate, 0.90, cosine=0.70), # clears the threshold
|
||||
_chunk(servmon, 0.80, cosine=0.20), # high fused rank, weak cosine
|
||||
]
|
||||
plan = chat_api.plan_turn(chunks, _shape_settings())
|
||||
assert plan.deflected is False # 0.70 >= 0.62
|
||||
assert [d.title for d in plan.docs] == ["validate.sh"] # exactly ONE cited
|
||||
assert [d.title for d in plan.related_docs] == ["ServMon README"]
|
||||
assert "VALIDATE_SH_CONTENT" in plan.system_prompt
|
||||
assert "SERVMON_README_CONTENT" not in plan.system_prompt
|
||||
assert [d.title for d in plan.suggested_docs] == [
|
||||
"validate.sh",
|
||||
"ServMon README",
|
||||
] # both suggested (no floor), rank order
|
||||
assert plan.related_docs == [] # nothing beyond rank 2 for 2 docs
|
||||
assert "VALIDATE_SH_CONTENT" in plan.system_prompt # A5 preview fallback
|
||||
assert "SERVMON_README_CONTENT" in plan.system_prompt # ditto
|
||||
|
||||
|
||||
def test_shape_3_nagraz_answered_by_own_doc_byzin_uncited() -> None:
|
||||
def test_shape_3_nagraz_answered_by_own_doc_byzin_suggested() -> None:
|
||||
"""Observed shape 3 (TODO L117–119): the Trooper_Nagraz question is
|
||||
answered from ``Trooper_Nagraz.pl`` — the 2nd chip
|
||||
``Trooper_Byzin.pl`` uncited. The SAME shape as shape 2 with
|
||||
different fixtures — the bar filters the 2nd chip; it is not a
|
||||
coincidence of the validate.sh pair."""
|
||||
answered from ``Trooper_Nagraz.pl`` with ``Trooper_Byzin.pl``
|
||||
retrieved alongside (weak cosine). The SAME shape as shape 2 with
|
||||
different fixtures — phase 118's no-floor tiering suggests BOTH
|
||||
(the phase-113 "bar filters the 2nd chip" story is retired); the
|
||||
HIGH prompt seeds both summaries (A5 fallback for the short
|
||||
fixture content)."""
|
||||
nagraz = _doc("scripts", "scripts/Trooper_Nagraz.pl", "Trooper_Nagraz.pl",
|
||||
"NAGRAZ_PL_CONTENT")
|
||||
byzin = _doc("scripts", "scripts/Trooper_Byzin.pl", "Trooper_Byzin.pl",
|
||||
"BYZIN_PL_CONTENT")
|
||||
chunks = [
|
||||
_chunk(nagraz, 0.85, cosine=0.70),
|
||||
_chunk(byzin, 0.75, cosine=0.15), # below the bar
|
||||
_chunk(byzin, 0.75, cosine=0.15), # weak cosine — still suggested (A3)
|
||||
]
|
||||
plan = chat_api.plan_turn(chunks, _shape_settings())
|
||||
assert plan.deflected is False
|
||||
assert [d.title for d in plan.docs] == ["Trooper_Nagraz.pl"] # 1 cited
|
||||
assert [d.title for d in plan.related_docs] == ["Trooper_Byzin.pl"] # 1 related
|
||||
assert "BYZIN_PL_CONTENT" not in plan.system_prompt
|
||||
assert [d.title for d in plan.suggested_docs] == [
|
||||
"Trooper_Nagraz.pl",
|
||||
"Trooper_Byzin.pl",
|
||||
] # both suggested (no floor), rank order
|
||||
assert plan.related_docs == []
|
||||
assert "NAGRAZ_PL_CONTENT" in plan.system_prompt # A5 preview fallback
|
||||
assert "BYZIN_PL_CONTENT" in plan.system_prompt # ditto
|
||||
|
||||
|
||||
# ---------- task 03: done-frame wire (endpoint-level fakes, no stack) ----------
|
||||
@@ -664,21 +681,22 @@ def _fake_retriever(chunks: list[RetrievedChunk]) -> Any:
|
||||
return retrieve
|
||||
|
||||
|
||||
def test_shape_4_meta_question_deflected_frame_is_row_only(
|
||||
def test_shape_4_meta_question_deflected_frame_has_no_chips_or_row(
|
||||
client: TestClient,
|
||||
chip_env: tuple[_FakeSession, _CannedLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Observed shape 4 (TODO L120–123), pinned on the DONE FRAME: a
|
||||
meta question about the conversation's own history → chips
|
||||
meta question about the conversation's own history → weak hits
|
||||
``app/api/suggestions.py`` + ``108_history_wire_check/00_phase.md``,
|
||||
neither used. No doc clears the bar and the agent reads nothing —
|
||||
the frame is ROW-ONLY: ``sources: []`` (the UI chips every source
|
||||
entry — zero chips) with the weak hits in ``related`` (rank order,
|
||||
≤ ``related_max_docs``) — the de-emphasized row's links (the row's
|
||||
rendering itself is pinned by task 02's source tests + the E2E).
|
||||
The weak retrieval stays durably recorded (LOCKED A3); the weak
|
||||
content never reaches the LLM (LOW prompt, titles only)."""
|
||||
neither used. Phase 118: the agent reads nothing, the weak hits
|
||||
are the SUGGESTED tier (no floor, A3 — the durable record's input)
|
||||
and, with only two retrieved docs, nothing reaches rank 6+ — the
|
||||
frame carries ``sources: []`` (zero chips — a deflected answer
|
||||
cites nothing) AND an empty ``related`` row (the row's rendering
|
||||
itself is pinned by task 02's source tests + the E2E). The weak
|
||||
retrieval stays durably recorded (LOCKED A3); the weak content
|
||||
never reaches the LLM (LOW prompt, titles only)."""
|
||||
session, llm = chip_env
|
||||
suggestions = _doc("brain-of-reese", "app/api/suggestions.py",
|
||||
"suggestions.py", "SUGGESTIONS_PY_CONTENT")
|
||||
@@ -701,13 +719,9 @@ def test_shape_4_meta_question_deflected_frame_is_row_only(
|
||||
assert done["type"] == "done"
|
||||
assert done["deflected"] is True
|
||||
assert done["sources"] == [] # zero citation chips on the wire
|
||||
related = done["related"]
|
||||
assert [(s["source"], s["path"]) for s in related] == [
|
||||
("brain-of-reese", "app/api/suggestions.py"),
|
||||
("brain-of-reese", ".agents/108_history_wire_check/00_phase.md"),
|
||||
] # rank order
|
||||
assert len(related) <= 2 # related_max_docs
|
||||
assert all(s["title"] for s in related) # the row's links carry the identity
|
||||
# Phase 118 (A3): both weak docs are suggested (≤5, no floor) —
|
||||
# nothing reaches rank 6+, so the related row is empty.
|
||||
assert done["related"] == []
|
||||
assert done["suggestions"] # the "Maybe try" chips are unchanged
|
||||
|
||||
(system, _user) = llm.seen[0][0], llm.seen[0][1]
|
||||
@@ -722,16 +736,17 @@ def test_shape_4_meta_question_deflected_frame_is_row_only(
|
||||
assert "108_history_wire_check/00_phase.md" in row.sources
|
||||
|
||||
|
||||
def test_done_frame_single_cited_ref_strong_plus_weak(
|
||||
def test_done_frame_carries_suggested_refs_strong_plus_weak(
|
||||
client: TestClient,
|
||||
chip_env: tuple[_FakeSession, _CannedLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""Shape 2 on the wire — the single-document question's input to
|
||||
"exactly one citation chip" (the E2E asserts the rendered chip):
|
||||
the bar-clearing doc is the ONLY ``sources`` ref; the weak 2nd doc
|
||||
rides ``related``; the tiers are disjoint (the done frame's dedupe).
|
||||
The durable record keeps the FULL retrieval (LOCKED A3)."""
|
||||
"""Shape 2 on the wire under phase 118 (LOCKED A4): the citation
|
||||
surface is the suggested tier + the agent's reads (deduped) —
|
||||
with two retrieved docs and no read, BOTH docs are ``sources``
|
||||
refs (no floor — A3); nothing reaches rank 6+, so ``related`` is
|
||||
empty; the tiers stay disjoint (the done frame's dedupe). The
|
||||
durable record keeps the FULL retrieval (LOCKED A3)."""
|
||||
session, _llm = chip_env
|
||||
validate = _doc("brain-of-reese", ".agents/validate.sh", "validate.sh",
|
||||
"VALIDATE_SH_CONTENT")
|
||||
@@ -753,10 +768,9 @@ def test_done_frame_single_cited_ref_strong_plus_weak(
|
||||
assert done["deflected"] is False
|
||||
assert [(s["source"], s["path"]) for s in done["sources"]] == [
|
||||
("brain-of-reese", ".agents/validate.sh"),
|
||||
] # EXACTLY one citation chip on the wire
|
||||
assert [(s["source"], s["path"]) for s in done["related"]] == [
|
||||
("ServMon", "README.md"),
|
||||
("ServMon", "README.md"), # A4: suggested + read — both suggested (A3)
|
||||
]
|
||||
assert done["related"] == [] # nothing reaches rank 6+ for 2 docs
|
||||
cited = {(s["source"], s["path"]) for s in done["sources"]}
|
||||
related = {(s["source"], s["path"]) for s in done["related"]}
|
||||
assert cited.isdisjoint(related)
|
||||
@@ -769,20 +783,25 @@ def test_done_frame_single_cited_ref_strong_plus_weak(
|
||||
assert "ServMon/README.md" in row.sources
|
||||
|
||||
|
||||
def test_agent_read_below_floor_doc_joins_sources(
|
||||
def test_agent_read_related_doc_is_cited_not_related(
|
||||
client: TestClient,
|
||||
chip_env: tuple[_FakeSession, _CannedLLM],
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""The agent-read exemption (LOCKED A2): a doc UNDER the bar that
|
||||
the agent ``read`` via the tool is cited by definition — the model
|
||||
read it, so it was used. It joins ``sources`` (after the retrieved
|
||||
cited docs, deduped) and is EXCLUDED from ``related`` (a used doc
|
||||
must never read as "nearby"); the other below-floor doc stays in
|
||||
the tier. The read content reached the model (the tool result in
|
||||
the follow-up request)."""
|
||||
"""The agent-read exemption (LOCKED A4, phase-118 tiering): a
|
||||
rank-6+ doc — the related tier ("nearby docs") — that the agent
|
||||
``read`` via the tool is cited by definition: the model read it, so
|
||||
it was used. It joins ``sources`` (after the suggested docs — it
|
||||
was not suggested, so the read appends it last, deduped) and is
|
||||
EXCLUDED from ``related`` (a used doc must never read as "nearby");
|
||||
the other rank-6+ doc stays in the tier. The read content reached
|
||||
the model (the tool result in the follow-up request)."""
|
||||
session, _default_llm = chip_env
|
||||
strong = _doc("docs", "strong.md", "Strong", "STRONG_DOC_CONTENT")
|
||||
fillers = [
|
||||
_doc("docs", f"filler{i}.md", f"Filler {i}", f"FILLER_{i}_CONTENT")
|
||||
for i in range(1, 5) # ranks 2–5 — fill the suggested tier
|
||||
]
|
||||
weak_b = _doc("docs", "weak-b.md", "Weak B", "WEAK_B_READ_BY_AGENT")
|
||||
weak_c = _doc("docs", "weak-c.md", "Weak C", "WEAK_C_CONTENT")
|
||||
monkeypatch.setattr(
|
||||
@@ -790,9 +809,13 @@ def test_agent_read_below_floor_doc_joins_sources(
|
||||
"retrieve",
|
||||
_fake_retriever(
|
||||
[
|
||||
_chunk(strong, 0.90, cosine=0.70), # clears the bar
|
||||
_chunk(weak_b, 0.80, cosine=0.20), # below the bar — read by the agent
|
||||
_chunk(weak_c, 0.70, cosine=0.10), # below the bar — nobody reads it
|
||||
_chunk(strong, 0.90, cosine=0.70), # clears the threshold (rank 1)
|
||||
_chunk(fillers[0], 0.85, cosine=0.30), # ranks 2–5: suggested
|
||||
_chunk(fillers[1], 0.80, cosine=0.30),
|
||||
_chunk(fillers[2], 0.75, cosine=0.30),
|
||||
_chunk(fillers[3], 0.72, cosine=0.30),
|
||||
_chunk(weak_b, 0.70, cosine=0.20), # rank 6 — related; read by the agent
|
||||
_chunk(weak_c, 0.65, cosine=0.10), # rank 7 — related; nobody reads it
|
||||
]
|
||||
),
|
||||
)
|
||||
@@ -815,7 +838,10 @@ def test_agent_read_below_floor_doc_joins_sources(
|
||||
assert "WEAK_B_READ_BY_AGENT" in tool_msgs[0]["content"]
|
||||
|
||||
sources = [(s["source"], s["path"]) for s in done["sources"]]
|
||||
assert sources == [("docs", "strong.md"), ("docs", "weak-b.md")] # read ⇒ cited, last
|
||||
# A4: suggested (5) + the read doc (last — it was not suggested).
|
||||
assert sources[-1] == ("docs", "weak-b.md") # read ⇒ cited, last
|
||||
assert len(sources) == 6
|
||||
assert ("docs", "weak-c.md") not in sources # never suggested, never read
|
||||
related = [(s["source"], s["path"]) for s in done["related"]]
|
||||
assert related == [("docs", "weak-c.md")] # the read doc is not "nearby"
|
||||
assert set(sources).isdisjoint(set(related))
|
||||
@@ -823,3 +849,4 @@ def test_agent_read_below_floor_doc_joins_sources(
|
||||
(row,) = session.added
|
||||
assert isinstance(row, QueryLog)
|
||||
assert "weak-b.md" in row.sources # the full retrieval is recorded (A3)
|
||||
assert "weak-c.md" in row.sources # … rank 6+ included (suggested + related + read)
|
||||
|
||||
Reference in New Issue
Block a user