Files
brain-of-reese/tests/unit/test_prompts.py
T
ducoterra eebc824636
Build and Push Containers / build-and-push-app (push) Successful in 1m50s
Build and Push Containers / build-and-push-db (push) Successful in 14s
phase: 162_search_recall_small_sources
**Phase 162 — final verification pass (all tasks already complete; verified + docs pass)**

- Verified: `_fold` (NFKD+casefold, stdlib) inside `_normalize_name` — `BESTÅ`→`besta` question side, `BÅsta.md`→`basta` path side; A2 match classes/ranking/caps/bonus untouched; FTS/cosine/A8 untouched.
- Verified: `grep_document` folds both sides via the imported `_fold` (no second copy); regex-teaching lines byte-identical; 3 copy sites updated.
- Verified: `regenerate_overview` appends the deterministic `Sources: <name> (<k> doc|docs)` tail (happy path only; zero-doc + `LLMError` rows byte-identical; `build_overview_prompt`/`load_kb_overview`/`build_kb_section` untouched).
- Verified: `TOOLS_SECTION` grep sentence updated + D4 search-discipline sentence present **verbatim**, immediately after the grep sentence; `PERSONA`/LOW pins byte-identical (`test_prompt_lock.py`).
- Measured evidence (in `.agents/reports/162_…/task04_prefix_evidence_and_battery.md` + re-measured every run by `test_under_rank_is_measured_not_assumed`): pre-fix tokens `['build','best']` → IKEA doc rank 6 (outside top-5); post-fix `['build','besta']` → suggested #1, cosine unchanged 0.382.
- `uv run pytest tests/e2e/test_search_recall_small_sources.py -v --no-cov` → **7 passed** (isolated); neighbours `test_kb_overview{,_freshness} test_name_signal_read_chips test_grep_regex_teaching test_harness_aligned_tools test_agent_document_tools test_tool_call_compact` → **24 passed**.
- `uv run pytest --cov=app --cov-report=term-missing` → **3756 passed**, app coverage **99%** (>90%).
- `uv run ruff check .` → All checks passed; `uv run pyright` → **0 errors**.
- Completion criteria: fold both ways ✅ · `Sources:` tail ✅ · TOOLS_SECTION + D4 verbatim + re-cut pins ✅ · E2E green/battery recorded/full suite/coverage/lint ✅ · docs pass ✅ · commit + dir move = harness.
- Battery: 5 phase-119 questions byte-identical to the phase-119 record (GATE PASS); the 2 Besta questions **skipped loudly** — verified the 2026-09-05 replica dump has zero `ikea` matches (`grep -c -i ikea data/bor_live_kb_replica.dump.sql` = 0); rank gate stays with the owner's live KB.
- Docs fixed: README — overview-tail doc + `BOR_KB_OVERVIEW_MAX_CHARS` tail-cut constraint, the phase-162 search-discipline paragraph, per-phase (not per-story) E2E wording, new "Retrieval regression battery" section; AGENTS.md — rule 4 per-phase wording + battery command in Quick reference. (The overview's expected "AGENTS.md §6 tool table" does not exist in this repo — the grep contract line lives in README's tool table, already updated.)
- Deviations (recorded, not assumed): `_fold` adds `casefold` (required for the pinned `ß → ss`); `①` NFKD-decomposes to `1` (the overview's parenthetical was wrong) and is excluded by `NAME_TOKEN_MIN_LEN`; pre-fix evidence lives in the reports dir because task files are read-only for this pass.
- Owner follow-up A6 (runtime, not code): add the Tuning-tab steering note "IKEA's BESTÅ assembly PDF is a live source — search it as `BESTÅ` or `AA-1272068`".
- Next pending phase: none — 162 is the only entry in `todo/` (harness moves it to `complete/`).
2026-10-07 01:42:01 -04:00

1772 lines
81 KiB
Python

"""Unit: locked persona prompt builder (PLAN §6 verbatim + both modes).
Also covers the phase-31 ``<knowledge_base>`` section (the stored,
lite-generated KB outline): its own builder contract (empty → ``""``,
char budget + ``[…truncated…]`` marker, pathological budgets), its
placement between ``<relevance>`` and ``<tuning>`` in both modes, and
the byte-identical-when-absent convention (phase 15 precedent).
(Phase 142, task 03: the section's intro is the honest-copy rewrite —
a compact, possibly partial map that favors recent documents, with the
verify-with-``ls``/``grep`` teaching for "latest / complete set"
questions — pinned verbatim; the false "(generated at import time)"
phrase is gone.)
And the phase-71 deflection plain-text line (owner-permitted
2026-09-03): the LOW prompt = pre-phase text + exactly the one new
line; the ``DEFLECT_MODE`` marker-keying contract is unchanged and
the line never leaks into the HIGH prompt.
And the ``<tools>`` copy (phase 72: the document-identity contract
stated up front — the combined ``source/path`` identity for
``read``/``grep``; phase 94, task 03: the ``ls`` clause rewritten to
the drill-down tree contract — one level per call, sources at the
top, folders + files below, ``grep`` as the without-listing locator —
while the ``read``/``grep`` clauses and the discipline rules are
byte-identical; phase 118, task 04: the ``read`` clause rewritten
for the summary-seed mode — the ``<documents>`` section holds
SUMMARIES, ``read`` adds the full text, and only an already-read
document is refused — while the ``ls``/``grep`` clauses and the
discipline rules stay byte-identical): the teaching refusals in
:mod:`app.rag.agent` re-state the same contract; the ``<tools>``
marker keying (HIGH only) is unchanged.
And the phase-119 cite discipline (task 04, LOCKED A5): the intro's
final sentence ("cite the document(s) you used, by path", phase 118)
is REPLACED — the HIGH prompt carries the discipline sentence exactly
once, inside ``<documents>`` after the intro's start-here framing;
the LOW prompt and ``PERSONA`` / ``TOOLS_SECTION`` stay byte-identical
(the full sha pins live in :mod:`tests.unit.test_prompt_lock`).
"""
from __future__ import annotations
import inspect
import uuid
from datetime import UTC, datetime
import pytest
from app.config import Settings
from app.models import Document
from app.rag.prompts import (
ORIGIN_RULE,
PERSONA,
SEARCH_DISCIPLINE_RULE,
SHOW_IMAGE_CLAUSE,
SUGGEST_INTRO,
TOOLS_SECTION,
_base,
_document_body,
_extract_context_line,
build_deflect_prompt,
build_high_prompt,
build_kb_section,
build_steering_section,
)
from app.rag.retriever import TRUNCATION_MARKER
#: A small multi-line outline standing in for the lite-generated one.
OVERVIEW = "- Homelab\n - Kubernetes (k3s)\n- Deployments\n - Borg backups"
#: The locked one-line intro of the ``<knowledge_base>`` section
#: (phase 142, task 03 — the honest-intro rewrite, pinned verbatim in
#: :func:`test_kb_intro_phase142_honest_copy`) — the intro line WITHOUT
#: its trailing newline (the section renderers / the f-strings below
#: supply the line break after it, as the pre-phase-142 constant did).
KB_INTRO = (
'A compact picture of the knowledge base\'s categories and topics — '
'it favors recent documents and may omit older or lesser ones: a map '
'of what exists, not a complete inventory. For questions about the '
'latest state or the complete set ("the last N …", "everything '
'about …", "all the …"), verify with `ls` (drill into the folders) '
'or `grep` before answering from the outline:'
)
#: The fixture documents' fixed creation date (phase 106, D5) — the
#: ``<document>`` block formats its UTC date part (the detached fixture
#: rows carry it exactly as the NOT NULL DB column guarantees it for
#: real rows).
_FIXTURE_CREATED_AT = datetime(2024, 6, 15, 12, 0, 0, tzinfo=UTC)
def _doc(
path: str,
content: str,
title: str,
summary: str | None = None,
*,
is_image: bool = False,
image_context: str | None = None,
) -> Document:
return Document(
id=uuid.uuid4(),
source="Homelab",
path=path,
full_path=f"/tmp/{path}",
title=title,
content=content,
content_hash="0" * 64,
created_at=_FIXTURE_CREATED_AT,
summary=summary,
is_image=is_image,
image_path="/tmp/pic.png" if is_image else None,
image_context=image_context,
)
def test_persona_rules_present_verbatim() -> None:
# Aligned to the owner's working-tree persona edits (PLAN §6 revision,
# 2026-08-22): no "you've got this" tagline, no mandated deflection
# opening. The honesty gate itself (rule 3) is unchanged.
for fragment in (
'You are "Brain of Reese" — the digital brain of Reese, a self-hoster and',
"optimistic about the user's ability to do things",
"Answer ONLY from the provided document context. Cite which document(s)",
"you used, by path.",
"Be concrete: names, versions, ports, hosts, schedules",
'HONESTY GATE: if <relevance> is "LOW", you must NOT pretend to know',
"Offer 2-3 alternative questions about things you DO have notes on.",
"Never invent facts, hosts, or steps that are not in the context.",
"Keep answers tight: short paragraphs, bullets where helpful.",
):
assert fragment in PERSONA
def test_persona_owner_edits_are_preserved() -> None:
"""PLAN §6 revision (2026-08-22): the removed elements must stay out."""
assert 'you\'ve got this' not in PERSONA # tagline removed by the owner
assert "Start your answer with a variant of" not in PERSONA # no mandated opening
assert "HONESTY GATE" in PERSONA # the gate itself is intact
def test_high_prompt_carries_relevance_marker_and_summary_block() -> None:
"""Phase 118 (LOCKED A6): the grounded turn seeds the document's
SUMMARY — the full content never reaches the prompt (the ``read``
tool is the only full-text path)."""
doc = _doc(
"kubernetes.md",
"Talos Linux on three nodes. FULL_CONTENT_SENTINEL_987654",
"Kubernetes Homelab Cluster",
summary="A Talos Linux cluster on three nodes.",
)
prompt = build_high_prompt([doc])
assert "<relevance>HIGH</relevance>" in prompt
assert "DEFLECT_MODE" not in prompt
assert "<documents>" in prompt and "</documents>" in prompt
assert 'path="kubernetes.md"' in prompt
assert "A Talos Linux cluster on three nodes." in prompt # the summary
assert "FULL_CONTENT_SENTINEL_987654" not in prompt # never the content
assert "Talos Linux on three nodes." not in prompt # nor the full sentence
assert "HONESTY GATE" in prompt # persona intact
def test_high_prompt_lists_multiple_documents_in_order() -> None:
a = _doc("a.md", "CONTENT_A", "Title A", summary="Summary A.")
b = _doc("b.md", "CONTENT_B", "Title B", summary="Summary B.")
prompt = build_high_prompt([a, b])
assert prompt.index("Summary A.") < prompt.index("Summary B.")
assert 'title="Title B"' in prompt
assert "CONTENT_A" not in prompt and "CONTENT_B" not in prompt
def test_high_prompt_seeds_summaries_of_all_suggested_docs() -> None:
"""The phase-118 completion pin: a grounded prompt built from five
summary-bearing documents contains ALL FIVE summaries + the intro,
and ZERO full-content characters (the full texts are not in the
prompt at all)."""
docs = [
_doc(
f"doc{i}.md",
f"FULL_CONTENT_SENTINEL_{i} " + "x" * 100,
f"Title {i}",
summary=f"SUMMARY_{i} of the document.",
)
for i in range(5)
]
prompt = build_high_prompt(docs)
assert SUGGEST_INTRO in prompt
for i in range(5):
assert f"SUMMARY_{i} of the document.\n" in prompt
assert f"FULL_CONTENT_SENTINEL_{i}" not in prompt
assert f'path="doc{i}.md"' in prompt
# One block per document, in the given order.
assert prompt.count("<document ") == 5
positions = [prompt.index(f"SUMMARY_{i}") for i in range(5)]
assert positions == sorted(positions)
# The intro leads the section: <documents> → intro → first block.
i_open = prompt.index("<documents>")
i_block = prompt.index("<document ")
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n\n"
def test_high_prompt_without_documents_stays_honest() -> None:
prompt = build_high_prompt([])
assert "<documents>" in prompt
assert "do not invent specifics" in prompt
def test_low_prompt_has_deflect_mode_and_titles_only() -> None:
titles = ["Kubernetes Homelab Cluster", "Backup Strategy"]
prompt = build_deflect_prompt(titles)
assert "<relevance>LOW</relevance>" in prompt
assert "DEFLECT_MODE" in prompt # marker the E2E mock keys on
assert "- Kubernetes Homelab Cluster" in prompt
assert "- Backup Strategy" in prompt
def test_low_prompt_never_contains_document_content() -> None:
secret = "SECRET_DOCUMENT_CONTENT_12345"
prompt = build_deflect_prompt(["Some Title"])
assert secret not in prompt
assert "<documents>" not in prompt
assert "HONESTY GATE" in prompt # the LOW rule is what the model must follow
def test_low_prompt_with_no_titles() -> None:
assert "nothing close at all" in build_deflect_prompt([])
def test_zero_note_prompt_is_byte_identical_to_pre_steering() -> None:
"""Phase 15 contract: with no steering notes the prompt is exactly what
it was before the <tuning> section existed. (Phase 37: the HIGH prompt
additionally carries the ``<tools>`` section after the mode body — the
fixtures account for it; phase 118: the ``<documents>`` section leads
with the ``SUGGEST_INTRO`` line — the fixture accounts for it; the
LOW prompt is untouched.)"""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
block = (
'<document source="Homelab" path="kubernetes.md" '
'title="Kubernetes Homelab Cluster" date="2024-06-15">\n'
"Talos Linux on three nodes.\n"
"</document>"
)
assert build_high_prompt([doc]) == (
_base("HIGH")
+ "\n<documents>\n"
+ SUGGEST_INTRO
+ "\n\n"
+ block
+ "\n</documents>"
+ "\n"
+ TOOLS_SECTION
)
# Phase 71: the LOW prompt carries the owner-permitted plain-text
# line after the DEFLECT_MODE sentence (the marker-keying contract
# is unchanged — the E2E mock keys on the marker's presence).
assert build_deflect_prompt(["T1", "T2"]) == (
_base("LOW")
+ "\nDEFLECT_MODE: retrieval was weak — the titles below are the closest "
"your notes come to the question. They are titles only; do not pretend "
"they answer it. Use them to propose 2-3 alternative questions.\n"
"Reply in plain text only — you have no tools in this mode.\n"
+ "- T1\n- T2"
)
assert "<tuning>" not in build_high_prompt([doc])
assert "<tuning>" not in build_deflect_prompt([])
# Phase 70: the rewritten <tools> copy stays out of the LOW path —
# the byte-identical equality above already proves it; this names
# the contract (no <tools>, no new copy) on both empty/non-empty LOW
# builds.
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
assert "<tools>" not in low
assert TOOLS_SECTION not in low
# ---------- <tools> section copy (phase 70: ls / read / grep) ----------
def test_tools_section_markers_and_new_tool_names() -> None:
"""Phase 70: the section keeps the ``<tools>``/``</tools>`` markers
the E2E mock keys on and teaches the harness-aligned tool names
(backticked, exactly as the ``AGENT_TOOLS`` schemas name them)."""
assert TOOLS_SECTION.startswith("<tools>\n")
assert TOOLS_SECTION.rstrip().endswith("</tools>")
for tool in ("`ls`", "`grep`", "`read`"):
assert tool in TOOLS_SECTION
def test_tools_section_teaches_the_harness_shapes() -> None:
"""Copy pins: ``ls``'s phase-63 catalog-line format, ``grep``'s
case- and accent-insensitive exact-string locator contract (up to 20
``source/path:line: text`` lines, a locator not a context-adder),
and ``read``'s combined ``source/path`` + full content. (Phase 72:
the source-name scope clause and the combined-identity clause are
pinned byte-for-byte in
:func:`test_tools_section_phase72_contract_clauses`.)"""
assert "source: X | path: Y | title: Z" in TOOLS_SECTION
assert "case- and accent-insensitive" in TOOLS_SECTION # phase 162 fold
assert "up to 20" in TOOLS_SECTION
assert "source/path:line: text" in TOOLS_SECTION
assert "locator, not a context-adder" in TOOLS_SECTION
assert "combined `source/path`" in TOOLS_SECTION
assert "full content" in TOOLS_SECTION
assert "Answer as soon as you have what you need" in TOOLS_SECTION
def test_tools_section_phase72_contract_clauses() -> None:
"""Phase 72 + phase 94: the contract clauses the teaching refusals
re-state after the fact, pinned byte-for-byte in the constant —
the ``ls`` clause (phase 94: the drill-down tree contract — one
level per call, sources at the top, folders + files below, never
the whole KB in one call, ``grep`` as the without-listing locator)
and the ``read``/``grep`` combined-identity clause (the combined
``source/path`` string exactly as shown in the ``ls`` output,
*including the source name*; a bare document path will not
resolve)."""
# The ls drill-down clauses (phase 94, task 03).
assert "one level at a time" in TOOLS_SECTION
assert "lists every synced source with its document count" in TOOLS_SECTION
assert "that source's top-level folders and files" in TOOLS_SECTION
assert "never the whole knowledge base in one call" in TOOLS_SECTION
assert "to find one specific document without listing, use `grep`" in TOOLS_SECTION
# The read combined-identity clause (byte-identical across phases).
assert (
"combined `source/path` string, exactly as shown in the `ls` "
"output — including the source name"
) in TOOLS_SECTION
# The bare-path note: read clause AND grep clause (exactly twice).
assert TOOLS_SECTION.count(
"a bare document path (without the source name) will not resolve"
) == 2
# The pre-phase-70 scope wording is gone — replaced by the
# explicit source-name contract (and the phase-72 source-name-only
# clause by the phase-94 drill-down contract).
assert "pass a source name as `path`" not in TOOLS_SECTION
assert "not a directory or file path" not in TOOLS_SECTION
def test_tools_section_phase95_read_truncation_clause() -> None:
"""Phase 95 (task 01): the ``read`` teaching gains exactly one line —
very large documents are capped, a cut read returns the first part
plus the TRUNCATED notice (the document did not end where it
stopped), and ``grep`` is the follow-up (it searches the whole
document). The ``ls``/``grep`` teaching is untouched (phase 94 owns
``ls``) — the clause is pinned byte-for-byte in the constant."""
assert (
"Very large documents are capped: a cut read returns the first "
"part plus a TRUNCATED notice — the document did not end where "
"it stopped; use `grep` (pattern) to find the rest, it searches "
"the whole document."
) in TOOLS_SECTION
# It rides the HIGH prompt and never the LOW (deflected) prompt.
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes")
assert "Very large documents are capped" in build_high_prompt([doc])
def test_tools_section_phase118_summary_seed_read_clause() -> None:
"""Phase 118 (task 04, A6): the ``read`` clause is rewritten for
the summary-seed mode — the ``<documents>`` section holds
SUMMARIES (a suggested document's full text is not in the prompt
until ``read`` adds it); do not re-read a document already read
IN FULL (answer from the text already in the prompt); a PARTIAL
document (a range read, or a truncated read — re-pinned phase
141, task 02: the re-read sentence is the coverage-true form —
read the other parts with ``offset``/``limit``, the document is
not complete in the context until read all); if the user asks to
open or read a suggested document, ``read`` it. The phase-72 "do
not call ``read`` for a ``<documents>`` document" copy is retired.
The ``ls`` clause (phase 94 drill-down contract), the ``grep``
clause, and the discipline rules stay byte-identical — the
prompt lock's prefix/suffix anchors survive (see
``test_prompt_lock``)."""
# The summary-seed copy (pinned byte-for-byte — the phase-141
# coverage-true re-read sentence re-pins the pre-phase-141 line;
# the section's other bytes are unchanged).
assert (
"The <documents> section holds SUMMARIES — the full text of a "
"suggested document is not in your prompt until you `read` it. "
"Do not re-read a document you have already read in full — "
"its full text is in your prompt, answer from it. If you have "
"read only part of a document (a range read, or a truncated "
"read), read the other parts with `offset` and `limit` — the "
"document is not complete in your context until you have read "
"it all. If the user asks you to open or read a suggested "
"document, `read` it — that is the point of the section."
) in TOOLS_SECTION
# The retired pre-phase-141 re-read copy is gone (it claimed the
# FULL text for a document that may be only partially in the
# context — the owner's stuck-turn bug).
assert (
"Do not re-read a document you have already read — its full "
"text is already in your prompt; answer directly from it"
) not in TOOLS_SECTION
# The read identity handoff now also names the <documents> summary
# blocks (the combined identity is shown there), keeping the
# "including the source name" contract.
assert (
"exactly as shown in the `ls` output — including the source "
"name — or in the <documents> summary blocks — adding its "
"full content to your context"
) in TOOLS_SECTION
# The retired pre-phase-118 copy is gone.
assert "Do not call `read` for a document already shown in" not in TOOLS_SECTION
assert "even when the user asks you to open or read it" not in TOOLS_SECTION
# The ls clause (phase 94 drill-down contract) — byte-identical.
assert (
"`ls` lists the knowledge base as a tree, one level at a "
"time: with no `path` it lists every synced source with its "
"document count and a summary of its contents; with a source "
"name (e.g. 'homelab') it lists that source's top-level "
"folders and files; with a `source/folder` path it drills one "
"level deeper. A listing shows only that level's subfolders "
"and its own files — never the whole knowledge base in one "
"call — and each folder line's summary says what the folder "
"contains before you drill into it. File lines are `source: X "
"| path: Y | title: Z`; to find one specific document without "
"listing, use `grep`."
) in TOOLS_SECTION
# The grep clause — folded contract (phase 162, LOCKED A3), up to the
# search-discipline sentence, byte-identical.
assert (
"`grep` locates an exact string (case- and accent-insensitive — "
"accents fold: `besta` matches `BESTÅ`) in the "
"indexed documents and returns up to 20 matching "
"`source/path:line: text` lines — a locator, not a "
"context-adder: read the winner with `read`. A grep pattern "
"is a plain substring, NEVER a regex — '.*' and '\\.' are "
"literal text there; if such a pattern returns no matches, "
"retry with the plain text you expect to see. "
) in TOOLS_SECTION
# The LOCKED search-discipline sentence sits immediately after the
# grep sentence, before the path clause (phase 162, LOCKED A4).
assert (
"retry with the plain text you expect to see. "
+ SEARCH_DISCIPLINE_RULE
+ " For a normal search pass only `pattern` — its optional "
"`path` argument limits the search to one document you already "
"know, by the same combined `source/path` string; never a "
"source name — a bare document path (without the source name) "
"will not resolve there either."
) in TOOLS_SECTION
# The discipline rules — byte-identical.
assert (
"Make exactly one tool call per reply — a reply carrying two "
"tool calls runs only the first, the second is discarded — "
"and wait for the result before the next call. Never repeat a "
"call that was refused or already succeeded — the refusal "
"already told you the correct form. Answer as soon as you "
"have what you need."
) in TOOLS_SECTION
# The new read clause rides the built HIGH prompt and never the
# LOW (deflected) prompt.
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes", summary="K8S")
high = build_high_prompt([doc])
assert "The <documents> section holds SUMMARIES" in high
assert "`read` it — that is the point of the section." in high
low = build_deflect_prompt(["T1"])
assert "The <documents> section holds SUMMARIES" not in low
def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
"""Phase 72/94: the contract clauses ride the HIGH prompt with the
rest of the section and never leak into the LOW/deflection prompt
(whose byte-identity is pinned in
:func:`test_zero_note_prompt_is_byte_identical_to_pre_steering`)."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
high = build_high_prompt([doc])
assert "<tools>" in high
assert "one level at a time" in high # the phase-94 ls clause
assert "including the source name" in high
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
assert "<tools>" not in low
assert "one level at a time" not in low
assert "including the source name" not in low
def test_tools_section_phase145_origin_rule() -> None:
"""Phase 145 (task 02): the ``<tools>`` ``read`` clause carries the
original-source rule (LOCKED verbatim in :data:`ORIGIN_RULE`) — when
the answer uses a document whose read result carries an ``origin:``
line, it includes the original source (a link to the repository
from the origin line, with the document's path and the pinned branch
where shown) so the user opens the complete original in one step.
The rule rides the HIGH prompt (``<tools>`` is HIGH-only) and never
the LOW/deflection prompt. The citation discipline is UNCHANGED —
the rule extends the ANSWERS of cited documents; it does not widen
the cite set (the "cite only what you read" sentence stays
byte-identical — pinned in ``test_prompt_lock``)."""
# The rule is present verbatim in the constant, exactly once, and at
# its exact position (the read clause's end, before the grep clause).
assert TOOLS_SECTION.count(ORIGIN_RULE) == 1
assert (
"it searches the whole document. "
+ ORIGIN_RULE
+ " `grep` locates an exact string"
) in TOOLS_SECTION
# It rides the built HIGH prompt with the rest of the section.
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes")
high = build_high_prompt([doc])
assert ORIGIN_RULE in high
# ...and never the LOW/deflection prompt (``<tools>`` is HIGH-only).
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
assert ORIGIN_RULE not in low
# The citation discipline is UNCHANGED: the rule does not widen the
# cite set (the "cite only what you read" sentence is untouched).
assert (
"Cite only the document(s) you read — or, if you answered from a "
"suggested summary without reading it, cite that suggested "
"document — never a document you neither read nor used."
) in high
assert "Cite only the document(s) you read" in SUGGEST_INTRO
def test_documents_section_leads_with_the_suggest_intro() -> None:
"""Phase 118, task 03: the ``<documents>`` section leads with the
start-here :data:`SUGGEST_INTRO` line BEFORE the first block (the
phase-15 ``_STEERING_INTRO`` / phase-31 ``_KB_INTRO`` precedent) —
only when at least one block is present. This is the summary-as-
starting-point framing, not the reverted phase-72 do-not-read
reminder (that copy taught the seed texts as already-read context;
with A6's summary seeding the blocks are starting points the model
may ``read`` through to full text)."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
high = build_high_prompt([doc])
i_open = high.index("<documents>")
i_block = high.index('<document source="Homelab"')
assert high[i_open : i_block] == f"<documents>\n{SUGGEST_INTRO}\n\n"
# The intro occurs exactly once and sits inside the section.
assert high.count(SUGGEST_INTRO) == 1
assert high.index(SUGGEST_INTRO) > high.index("<documents>")
assert high.index(SUGGEST_INTRO) < high.index("</documents>")
def test_documents_section_without_blocks_has_no_intro() -> None:
"""The intro rides only on present blocks: an empty ``<documents>``
section is exactly the fallback line again (no intro, no blocks)."""
high = build_high_prompt([])
assert SUGGEST_INTRO not in high
i_open = high.index("<documents>")
i_close = high.index("</documents>")
assert (
high[i_open : i_close]
== "<documents>\n(no documents matched — do not invent specifics)\n"
)
# ---------- phase 119 (D3, LOCKED A4): the suggested-folder lines ----------
def test_high_prompt_folder_lines_after_intro_before_first_block() -> None:
"""The folder lines ride the ``<documents>`` section immediately
AFTER the ``SUGGEST_INTRO`` line — each on its own line — then a
blank line, then the first ``<document>`` block. Plain lines: no
new markup/tag anywhere (the E2E mock keys off the ``<documents>``
marker and the LAST block's tail)."""
doc = _doc(
"deploy/Deployments/reeseapps/gitea/app/gitea-web.env.j2",
"FULL_CONTENT_SENTINEL_119",
"Gitea Web Env",
summary="Gitea web env file.",
)
lines = [
"Homelab/deploy/Deployments/reeseapps/gitea/: app/ (5 docs), README.md",
"Homelab/deploy/Deployments/reeseapps/: gateway/ (3 docs), gitea/ (8 docs)",
]
prompt = build_high_prompt([doc], folder_lines=lines)
i_open = prompt.index("<documents>")
i_block = prompt.index("<document ")
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n{lines[0]}\n{lines[1]}\n\n"
# Each line occurs exactly once, inside the section, before the
# first block (never after a summary — the E2E tail echo is safe).
for line in lines:
assert prompt.count(line) == 1
assert prompt.index("<documents>") < prompt.index(line) < prompt.index("<document ")
assert prompt.index(line) < prompt.index("</documents>")
# The <document> block markup AND body stay byte-identical: from the
# first block on, the prompt equals the no-folder-line build.
plain = build_high_prompt([doc])
assert prompt[prompt.index("<document ") :] == plain[plain.index("<document ") :]
# No new markup: the section still opens/closes exactly once (the
# ``<documents>`` MENTION in the TOOLS_SECTION copy is plain text —
# count the tag + newline, not the bare substring).
assert prompt.count("<documents>\n") == 1
assert prompt.count("</documents>") == 1
assert prompt.count("<document ") == 1
def test_high_prompt_single_folder_line_exact_shape() -> None:
"""One folder line: the exact slice between the section open and the
first block is ``<documents>\n`` + intro + ``\n`` + line +
``\n\n``."""
doc = _doc("a.md", "CONTENT", "Title A", summary="Summary A.")
line = "Homelab/: a.md (2 docs), b/ (1 doc)"
prompt = build_high_prompt([doc], folder_lines=[line])
i_open = prompt.index("<documents>")
i_block = prompt.index("<document ")
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n{line}\n\n"
def test_high_prompt_empty_folder_lines_byte_identical_to_phase_118() -> None:
"""LOCKED A4: empty ``folder_lines`` (the default) ⇒ the phase-118
build, byte-identical — the existing pins keep passing and this
pins the default explicitly (both an explicit ``()`` and the
omitted parameter)."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
block = (
'<document source="Homelab" path="kubernetes.md" '
'title="Kubernetes Homelab Cluster" date="2024-06-15">\n'
"Talos Linux on three nodes.\n"
"</document>"
)
expected = (
_base("HIGH")
+ "\n<documents>\n"
+ SUGGEST_INTRO
+ "\n\n"
+ block
+ "\n</documents>"
+ "\n"
+ TOOLS_SECTION
)
# An explicit empty tuple AND the omitted parameter (the default):
assert build_high_prompt([doc], folder_lines=()) == expected
assert build_high_prompt([doc]) == expected
# The LOW (deflected) prompt has no folder_lines parameter at all —
# the deflected build stays byte-identical (the phase-118 pin in
# test_zero_note_prompt_is_byte_identical_to_pre_steering stands).
def test_folder_lines_ride_only_on_present_blocks() -> None:
"""Like the intro, the folder lines ride on present blocks ONLY: an
empty ``<documents>`` section is unchanged (no lines, no blocks)."""
prompt = build_high_prompt([], folder_lines=["X/: y.md"])
assert prompt == build_high_prompt([])
assert "X/: y.md" not in prompt
# ---------- phase 119 (D4, LOCKED A5): the cite-discipline sentence ----------
#: The LOCKED A5 sentence (phase 119, task 04) — the exact replacement
#: for the retired phase-118 final sentence of :data:`SUGGEST_INTRO`
#: ("Cite the document(s) you used, by path."). Closes the live
#: confabulation: the answer's "Docs used:" line cited a file the
#: agent never read.
CITE_DISCIPLINE = (
"Cite only the document(s) you read — or, if you answered from a "
"suggested summary without reading it, cite that suggested "
"document — never a document you neither read nor used."
)
def test_high_prompt_carries_cite_discipline_exactly_once_in_documents() -> None:
"""LOCKED A5: the HIGH prompt carries the discipline sentence
EXACTLY ONCE — it IS the intro's final sentence, so it sits inside
``<documents>``, AFTER the intro's start-here framing and BEFORE
the first block (both the plain build and the folder-line build;
the sentence rides the intro line, which never moves)."""
doc = _doc(
"kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster", summary="K8S"
)
assert CITE_DISCIPLINE in SUGGEST_INTRO
assert SUGGEST_INTRO.endswith(CITE_DISCIPLINE) # it is the final sentence
for high in (build_high_prompt([doc]), build_high_prompt([doc], folder_lines=["X/: y.md"])):
assert high.count(CITE_DISCIPLINE) == 1
i_docs = high.index("<documents>")
i_sentence = high.index(CITE_DISCIPLINE)
i_block = high.index("<document ")
i_close = high.index("</documents>")
assert i_docs < i_sentence < i_close
assert i_sentence > high.index("start here if one seems right")
assert i_sentence < i_block # before the first block, never after a summary
# The retired phase-118 sentence is gone from the prompt (it is
# gone from the constant — pinned in test_prompt_lock too).
high = build_high_prompt([doc])
assert "Cite the document(s) you used, by path." not in high
# The sentence rides on present blocks ONLY: an empty ``<documents>``
# section (no intro) carries none of it.
assert CITE_DISCIPLINE not in build_high_prompt([])
def test_cite_discipline_absent_from_low_prompt() -> None:
"""LOCKED A5: the discipline sentence belongs to ``SUGGEST_INTRO``
(the HIGH path) — it never leaks into the LOW (deflected) prompt,
whose byte-identity is pinned separately (the LOW anchors in
``test_prompt_lock`` pass unchanged)."""
for prompt in (
build_deflect_prompt(["T1", "T2"]),
build_deflect_prompt(["T1"], notes=["be concise"], kb_overview=OVERVIEW),
build_deflect_prompt([]),
):
assert CITE_DISCIPLINE not in prompt
assert "Cite only the document(s) you read" not in prompt
assert "Cite the document(s) you used, by path." not in prompt
def test_tools_section_old_names_and_budget_copy_gone() -> None:
"""The phase-37/68 tool names and the phase-37 per-tool budget line
(phase 45: the round cap is the bound — the prompt does not
re-state it) are out of the copy."""
for old in ("list_documents", "read_document", "search_documents"):
assert old not in TOOLS_SECTION
assert "more than one" not in TOOLS_SECTION
assert "extra document" not in TOOLS_SECTION
def test_high_prompt_still_ends_with_tools_section() -> None:
"""Mock keying intact: the HIGH prompt still ends with the
``<tools>`` section after ``</documents>``, now in the phase-70
copy — new names in, old names out."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
prompt = build_high_prompt([doc])
assert TOOLS_SECTION in prompt
assert prompt.index("</documents>") < prompt.index("<tools>")
assert prompt.rstrip().endswith("</tools>")
for tool in ("`ls`", "`grep`", "`read`"):
assert tool in prompt
for old in ("list_documents", "read_document", "search_documents"):
assert old not in prompt
# ---------- phase 118 (task 03): the NULL-summary preview fallback (A5) ----------
def _content_only_doc(content: str) -> Document:
return _doc("kubernetes.md", content, "Kubernetes Homelab Cluster")
def test_null_summary_falls_back_to_preview_plus_marker() -> None:
"""A doc whose ``summary`` is None (a fail-soft import miss) seeds the
first ``suggestion_preview_chars`` (default 400) content characters +
the shared ``[…truncated…]`` marker on its own line — never the full
content, never an LLM call."""
content = "A" * 400 + "B" * 600 # 1000 chars
prompt = build_high_prompt([_content_only_doc(content)])
body_start = prompt.index('date="2024-06-15">\n') + len('date="2024-06-15">\n')
body_end = prompt.index("\n</document>", body_start)
assert prompt[body_start:body_end] == content[:400] + "\n" + TRUNCATION_MARKER
assert "B" * 40 not in prompt # beyond the 400-char cut
assert prompt.count(TRUNCATION_MARKER) == 1 # only the block's marker
def test_whitespace_summary_uses_the_same_preview_fallback() -> None:
for blank in ("", " ", "\n \t "):
doc = _doc("kubernetes.md", "A" * 500 + "B" * 500, "T", summary=blank)
prompt = build_high_prompt([doc])
assert "A" * 400 in prompt
assert "B" * 40 not in prompt
assert TRUNCATION_MARKER in prompt
def test_short_content_preview_is_the_whole_content_unmarked() -> None:
"""Content at or under the cap rides whole — nothing was cut, so no
marker (the marker signals truncation, not the fallback)."""
for content in ("short body", "C" * 399, "D" * 400):
prompt = build_high_prompt([_content_only_doc(content)])
assert content + "\n</document>" in prompt
assert TRUNCATION_MARKER not in prompt
def test_summary_stripped_and_never_truncated_by_the_preview_cap() -> None:
"""The summary body is the stripped ``doc.summary`` — even when it is
longer than the preview cap (the cap bounds only the FALLBACK; a
stored summary is trusted context, LOCKED A5)."""
summary = "S" * 900
content = "CONTENT_SENTINEL " + "x" * 50
prompt = build_high_prompt([_doc("kubernetes.md", content, "T", summary=f" {summary}\n")])
assert summary in prompt
assert content not in prompt
assert TRUNCATION_MARKER not in prompt
def test_preview_cap_setting_is_honored_on_the_fallback_path(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""A non-default ``suggestion_preview_chars`` (``BOR_SUGGESTION_PREVIEW_CHARS``
— the env mapping is pinned in :mod:`tests.unit.test_config`) cuts the
preview at the setting's cap on the fallback path."""
from app.rag import prompts as prompts_mod
monkeypatch.setattr(
prompts_mod,
"get_settings",
lambda: Settings(_env_file=None, suggestion_preview_chars=10), # pyright: ignore[reportCallIssue]
)
content = "A" * 100 + "B" * 100
prompt = build_high_prompt([_content_only_doc(content)])
assert "A" * 10 in prompt
assert "A" * 11 not in prompt
assert TRUNCATION_MARKER in prompt
def test_preview_fallback_never_reads_settings_for_summarized_docs(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""House pattern (``build_steering_section`` / ``build_kb_section``):
the fallback cap is read from settings on the fallback path ONLY — a
prompt built from summary-bearing docs makes no settings call for
it (a dead ``get_settings`` must not break such a build)."""
from app.rag import prompts as prompts_mod
doc = _doc("kubernetes.md", "CONTENT_SENTINEL", "T", summary="the summary")
def _dead() -> None: # pragma: no cover - must never be called
raise AssertionError("get_settings() called for a summarized doc")
monkeypatch.setattr(prompts_mod, "get_settings", _dead)
prompt = build_high_prompt([doc])
assert "the summary\n</document>" in prompt
assert "CONTENT_SENTINEL" not in prompt
# ---------- phase 118: the LOW prompt stays byte-identical ----------
def test_low_prompt_byte_identical_to_pre_task() -> None:
"""LOCKED A8 surface: the deflection prompt is untouched by the
summary seeding — byte-identical build on the same inputs (the full
sha pin lives in :mod:`tests.unit.test_prompt_lock`)."""
for titles, notes, kb in (
(["T1", "T2"], None, None),
(["T1"], ["be concise"], None),
([], None, OVERVIEW),
(["T1", "T2"], ["be concise"], OVERVIEW),
):
prompt = build_deflect_prompt(titles, notes=notes, kb_overview=kb)
assert "DEFLECT_MODE" in prompt
assert "<documents>" not in prompt
assert SUGGEST_INTRO not in prompt
assert "<tools>" not in prompt
def test_relevance_placeholder_rejected_for_garbage() -> None:
with pytest.raises(ValueError, match="HIGH or LOW"):
_base("MEDIUM")
# ---------- <knowledge_base> section (phase 31) ----------
def test_kb_section_empty_when_no_overview() -> None:
assert build_kb_section("") == ""
assert build_kb_section(" \n\t ") == ""
def test_kb_section_format_intro_and_content() -> None:
assert build_kb_section(OVERVIEW) == (
f"<knowledge_base>\n{KB_INTRO}\n{OVERVIEW}\n</knowledge_base>"
)
def test_kb_intro_phase142_honest_copy() -> None:
"""Phase 142 (task 03): the intro is pinned verbatim — it says what
the outline actually is (a compact picture of the KB's categories
and topics that favors recent documents and may omit older or lesser
ones — a map of what exists, not a complete inventory) and gives the
one behavior that fixes the "the last N phases" question class: for
latest-state / complete-set questions, verify with ``ls`` (drill
into the folders) or ``grep`` BEFORE answering from the outline. The
"(generated at import time)" phrase is gone — it was false (the sync
regenerates the row on every KB change) and it taught the model to
treat a partial outline as the whole frozen KB."""
from app.rag.prompts import _KB_INTRO
# The app's constant, pinned verbatim (the trailing newline included):
assert _KB_INTRO == (
'A compact picture of the knowledge base\'s categories and topics — '
'it favors recent documents and may omit older or lesser ones: a map '
'of what exists, not a complete inventory. For questions about the '
'latest state or the complete set ("the last N …", "everything '
'about …", "all the …"), verify with `ls` (drill into the folders) '
'or `grep` before answering from the outline:\n'
)
# The local pin is the same text minus the trailing newline.
assert KB_INTRO + "\n" == _KB_INTRO
# The false claim is gone from the constant (and thus the prompt).
assert "(generated at import time)" not in KB_INTRO
assert "generated at import time" not in KB_INTRO
# The three contracts it must carry: the recency bias (task 02's
# newest-first input is what makes it true), the map-not-inventory
# framing, and the verify-first teaching with both tool names.
assert "favors recent documents" in KB_INTRO
assert "not a complete inventory" in KB_INTRO
assert "verify with `ls` (drill into the folders) or `grep`" in KB_INTRO
assert "before answering from the outline" in KB_INTRO
# It rides the rendered section exactly where the old intro did:
# between the open tag and the outline, colon-terminated.
section = build_kb_section(OVERVIEW)
assert section.startswith(f"<knowledge_base>\n{KB_INTRO}")
assert section.index(KB_INTRO) < section.index(OVERVIEW)
# And it reaches BOTH modes' prompts (the section is mode-agnostic).
doc = _doc("kubernetes.md", "CONTENT", "T", summary="S.")
for prompt in (
build_high_prompt([doc], kb_overview=OVERVIEW),
build_deflect_prompt(["T1"], kb_overview=OVERVIEW),
):
assert KB_INTRO in prompt
assert "(generated at import time)" not in prompt
def test_kb_section_trims_overview_edges() -> None:
assert build_kb_section(f" {OVERVIEW} \n") == build_kb_section(OVERVIEW)
def test_kb_section_fits_budget_exactly_no_marker() -> None:
exact = f"<knowledge_base>\n{KB_INTRO}\n{OVERVIEW}\n</knowledge_base>"
section = build_kb_section(OVERVIEW, max_chars=len(exact))
assert TRUNCATION_MARKER not in section
assert section == exact
def test_kb_section_over_budget_capped_with_marker() -> None:
text = "- " + "x" * 500
# The section frame alone is 422 chars (the tag + the phase-142
# honest intro + the marker + the close tag), so the cap must clear
# it for any outline prefix to fit (pathological budgets are tested
# below).
cap = 600
section = build_kb_section(text, max_chars=cap)
assert len(section) <= cap # the budget is never exceeded
assert TRUNCATION_MARKER in section
assert section.startswith(f"<knowledge_base>\n{KB_INTRO}\n-")
assert section.endswith(f"{TRUNCATION_MARKER}\n</knowledge_base>")
# The body is the kept prefix + the marker on its own line, and the
# kept part must be a true prefix of the outline (longest-fitting).
body = section.removeprefix(f"<knowledge_base>\n{KB_INTRO}\n").removesuffix(
"\n</knowledge_base>"
)
kept, marker = body.rsplit("\n", 1)
assert marker == TRUNCATION_MARKER
assert kept.startswith("- ")
assert text.startswith(kept), "the kept part must be a prefix of the outline"
# And it is the longest such prefix: one more char would not fit.
assert len(section) > cap - 2, "the cut must sit as close to the cap as possible"
def test_kb_section_default_budget_from_settings(monkeypatch: pytest.MonkeyPatch) -> None:
from app.rag import prompts as prompts_mod
monkeypatch.setattr(
prompts_mod, "get_settings", lambda: Settings(_env_file=None) # pyright: ignore[reportCallIssue]
)
text = "y" * 9_000 # > the 4 000-char default
section = build_kb_section(text)
assert TRUNCATION_MARKER in section
assert len(section) <= 4_000
def test_kb_section_nonpositive_budget_is_empty() -> None:
assert build_kb_section(OVERVIEW, max_chars=0) == ""
assert build_kb_section(OVERVIEW, max_chars=-10) == ""
def test_kb_section_tiny_budget_never_exceeds_cap() -> None:
# Pathological budget (steering precedent, phase 15): the section must
# never exceed the cap — bare marker when it fits, no section at all
# when even that doesn't.
assert build_kb_section("a" * 500, max_chars=10) == "" # marker (13) > 10
fits_marker = build_kb_section("a" * 500, max_chars=len(TRUNCATION_MARKER))
assert fits_marker == TRUNCATION_MARKER
# ---------- <knowledge_base> placement (both modes) ----------
def test_no_overview_prompt_is_byte_identical_to_pre_phase() -> None:
"""Phase 31 contract: with no KB overview (None, empty, or blank)
every prompt is exactly what it was before the ``<knowledge_base>``
section existed — with or without steering notes. (Phase 37: the HIGH
prompt additionally carries the ``<tools>`` section after the mode
body; phase 118: the ``<documents>`` section leads with the
``SUGGEST_INTRO`` line — the fixtures account for both; the LOW
prompt is untouched.)"""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
block = (
'<document source="Homelab" path="kubernetes.md" '
'title="Kubernetes Homelab Cluster" date="2024-06-15">\n'
"Talos Linux on three nodes.\n"
"</document>"
)
docs_block = (
"\n<documents>\n"
+ SUGGEST_INTRO
+ "\n\n"
+ block
+ "\n</documents>"
+ "\n"
+ TOOLS_SECTION
)
high_plain = _base("HIGH") + docs_block
high_steered = _base("HIGH") + "\n" + build_steering_section(["be concise"]) + docs_block
# Phase 71: the owner-permitted plain-text line is part of the
# DEFLECT_MODE body in every LOW build (with or without steering).
low_plain = (
_base("LOW")
+ "\nDEFLECT_MODE: retrieval was weak — the titles below are the closest "
"your notes come to the question. They are titles only; do not pretend "
"they answer it. Use them to propose 2-3 alternative questions.\n"
"Reply in plain text only — you have no tools in this mode.\n"
+ "- T1\n- T2"
)
low_steered = (
_base("LOW")
+ "\n"
+ build_steering_section(["be concise"])
+ "\nDEFLECT_MODE: retrieval was weak — the titles below are the closest "
"your notes come to the question. They are titles only; do not pretend "
"they answer it. Use them to propose 2-3 alternative questions.\n"
"Reply in plain text only — you have no tools in this mode.\n"
+ "- T1\n- T2"
)
for kb in (None, "", " \n\t "):
assert build_high_prompt([doc], kb_overview=kb) == high_plain
assert build_high_prompt([doc], notes=["be concise"], kb_overview=kb) == high_steered
assert build_deflect_prompt(["T1", "T2"], kb_overview=kb) == low_plain
assert build_deflect_prompt(
["T1", "T2"], notes=["be concise"], kb_overview=kb
) == low_steered
assert "<knowledge_base>" not in build_high_prompt(
[doc], notes=["be concise"], kb_overview=kb
)
assert "<knowledge_base>" not in build_deflect_prompt(
["T1"], notes=["be concise"], kb_overview=kb
)
def test_high_prompt_kb_section_ordered_between_relevance_and_tuning() -> None:
doc = _doc("kubernetes.md", "TALOS_DOC_CONTENT", "Kubernetes Homelab Cluster")
prompt = build_high_prompt([doc], notes=["be concise"], kb_overview=OVERVIEW)
i_rel = prompt.index("<relevance>HIGH</relevance>")
i_kb_open = prompt.index("<knowledge_base>")
i_kb_close = prompt.index("</knowledge_base>")
i_tuning = prompt.index("<tuning>")
i_docs = prompt.index("<documents>")
assert i_rel < i_kb_open < i_kb_close < i_tuning < i_docs
assert KB_INTRO in prompt
assert OVERVIEW in prompt # outline intact within the section
assert "1. be concise" in prompt # steering still there
assert "TALOS_DOC_CONTENT" in prompt # documents still full
def test_high_prompt_kb_section_without_steering() -> None:
doc = _doc("kubernetes.md", "TALOS_DOC_CONTENT", "Kubernetes Homelab Cluster")
prompt = build_high_prompt([doc], kb_overview=OVERVIEW)
i_rel = prompt.index("<relevance>HIGH</relevance>")
i_kb_close = prompt.index("</knowledge_base>")
i_docs = prompt.index("<documents>")
assert i_rel < i_kb_close < i_docs
assert "<tuning>" not in prompt # no notes → no steering section
assert build_kb_section(OVERVIEW) in prompt
def test_deflect_prompt_kb_section_ordered_between_relevance_and_tuning() -> None:
prompt = build_deflect_prompt(
["Title A", "Title B"], notes=["be concise"], kb_overview=OVERVIEW
)
i_rel = prompt.index("<relevance>LOW</relevance>")
i_kb_open = prompt.index("<knowledge_base>")
i_kb_close = prompt.index("</knowledge_base>")
i_tuning = prompt.index("<tuning>")
i_mode = prompt.index("DEFLECT_MODE")
assert i_rel < i_kb_open < i_kb_close < i_tuning < i_mode
assert KB_INTRO in prompt
assert OVERVIEW in prompt
assert "1. be concise" in prompt
assert "- Title A" in prompt # weak-hit titles still carried
def test_prompt_kb_section_over_settings_budget_capped_with_marker(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Prompt-level budget: an overview longer than
``kb_overview_max_chars`` is capped with the shared marker — in both
modes."""
from app.rag import prompts as prompts_mod
monkeypatch.setattr(
prompts_mod,
"get_settings",
# 600 > the 422-char section frame, so a prefix + marker can fit.
lambda: Settings(_env_file=None, kb_overview_max_chars=600), # pyright: ignore[reportCallIssue]
)
text = "- " + "z" * 500
doc = _doc("kubernetes.md", "TALOS_DOC_CONTENT", "Kubernetes Homelab Cluster")
for prompt in (
build_high_prompt([doc], kb_overview=text),
build_deflect_prompt(["Title A"], kb_overview=text),
):
assert TRUNCATION_MARKER in prompt
assert prompt.index("<knowledge_base>") < prompt.index(TRUNCATION_MARKER)
# The capped section (open tag through close tag) fits the budget.
section = prompt[prompt.index("<knowledge_base>") :]
close = section.index("</knowledge_base>")
section = section[: close + len("</knowledge_base>")]
assert len(section) <= 600
# ---------- phase 71: the deflection plain-text line (prevention) ----------
#: The owner-permitted (2026-09-03) line appended to the ``DEFLECT_MODE``
#: body — the LOW prompt's only phase-71 change. The E2E mock keys on
#: the ``DEFLECT_MODE`` marker's *presence*, not the wording, so the
#: marker-keying contract is unchanged by the appended line.
PLAIN_TEXT_ONLY_LINE = "Reply in plain text only — you have no tools in this mode."
def _pre_phase71_low_body() -> str:
"""The ``DEFLECT_MODE`` body exactly as it was before phase 71."""
return (
"DEFLECT_MODE: retrieval was weak — the titles below are the closest "
"your notes come to the question. They are titles only; do not pretend "
"they answer it. Use them to propose 2-3 alternative questions.\n"
)
def test_low_prompt_is_pre_phase_plus_exactly_the_plain_text_line() -> None:
"""Diff pin: the LOW prompt = pre-phase text + exactly the one new
line, appended to the ``DEFLECT_MODE`` body; the weak-hit title list
follows exactly as before (and the line occurs exactly once)."""
prompt = build_deflect_prompt(["T1", "T2"])
assert prompt == (
_base("LOW")
+ "\n"
+ _pre_phase71_low_body()
+ PLAIN_TEXT_ONLY_LINE
+ "\n"
+ "- T1\n- T2"
)
assert prompt.count(PLAIN_TEXT_ONLY_LINE) == 1
assert prompt.endswith("- T1\n- T2") # the title list is untouched
def test_low_prompt_carries_the_line_and_keeps_the_mock_marker() -> None:
"""The new line is present in the LOW prompt (inside the
``DEFLECT_MODE`` body, after the marker) and the ``DEFLECT_MODE``
marker the E2E mock keys on stays put."""
for titles, tail in ((["T1"], "- T1"), ([], "(nothing close at all)")):
prompt = build_deflect_prompt(titles)
assert "DEFLECT_MODE" in prompt
assert PLAIN_TEXT_ONLY_LINE in prompt
assert prompt.index("DEFLECT_MODE") < prompt.index(PLAIN_TEXT_ONLY_LINE)
# The title list (or the no-titles fallback) follows the line
# exactly as before.
assert prompt.endswith(tail)
def test_plain_text_line_never_leaks_into_high_prompt() -> None:
"""The line is the LOW prompt's: every HIGH build (with/without
steering/overview) is unchanged and carries none of it."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
for notes, kb in (
(None, None),
(["be concise"], None),
(None, OVERVIEW),
(["be concise"], OVERVIEW),
):
high = build_high_prompt([doc], notes=notes, kb_overview=kb)
assert PLAIN_TEXT_ONLY_LINE not in high
assert "you have no tools" not in high
# ---------- phase 124 (task 03, LOCKED A5): the show_image prompt clause ----------
def test_show_image_clause_states_the_four_gate_elements() -> None:
"""LOCKED A5: the clause states ALL FOUR gate elements, pinned on
stable phrases, and references the tool by its backticked name (the
``TOOLS_SECTION`` voice). (1) the image-doc nature — its text is a
description GENERATED from the image; (2) the "contains the answer"
bar — never a merely related image, never padding; (3) at most one
image per answer; (4) the text answer is still provided in full."""
assert "`show_image`" in SHOW_IMAGE_CLAUSE # backticked, like the other tools
assert "description GENERATED from the image" in SHOW_IMAGE_CLAUSE # (1)
assert "what the user actually wants to see" in SHOW_IMAGE_CLAUSE # (2)
assert "never for a merely related image" in SHOW_IMAGE_CLAUSE # (2)
assert "never to pad an answer" in SHOW_IMAGE_CLAUSE # (2)
assert "at most one image per answer" in SHOW_IMAGE_CLAUSE # (3)
assert "answer fully in text" in SHOW_IMAGE_CLAUSE # (4)
# It is a clause, not a section: no ``<tools>`` markup of its own
# (the ``<documents>`` mention is plain text, as in the section copy).
assert "<tools>" not in SHOW_IMAGE_CLAUSE
assert "</tools>" not in SHOW_IMAGE_CLAUSE
def test_high_prompt_default_stays_byte_identical_to_pre_task() -> None:
"""Completion pin: ``build_high_prompt(docs)`` (no new arg) and the
explicit ``show_image=False`` are byte-identical to each other and
end with the untouched ``TOOLS_SECTION`` — the pre-task pins above
(and the sha pins in :mod:`tests.unit.test_prompt_lock`) pass
UNMODIFIED, with or without documents."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
plain = build_high_prompt([doc])
assert build_high_prompt([doc], show_image=False) == plain
assert plain.endswith(TOOLS_SECTION)
assert SHOW_IMAGE_CLAUSE not in plain
empty = build_high_prompt([])
assert build_high_prompt([], show_image=False) == empty
assert empty.endswith(TOOLS_SECTION)
assert SHOW_IMAGE_CLAUSE not in empty
def test_high_prompt_show_image_clause_lands_inside_tools_after_the_copy() -> None:
"""Phase 124 (task 03): ``show_image=True`` builds EXACTLY the
toggle-off prompt with its final ``</tools>`` replaced by the clause
(on its own line) + the closing tag — the prefix up to the clause
is byte-identical to the toggle-off output (the three-tool copy
untouched), the clause lands AFTER the existing copy and BEFORE the
closing tag, and appears exactly once."""
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
off = build_high_prompt([doc])
on = build_high_prompt([doc], show_image=True)
# The full structural equation (the clause is the ONLY difference).
assert on == off[: -len("</tools>")] + SHOW_IMAGE_CLAUSE + "\n</tools>"
clause_start = on.index(SHOW_IMAGE_CLAUSE)
# The prefix up to the clause is byte-identical to the toggle-off
# build minus its closing tag.
assert on[:clause_start] == off[: -len("</tools>")]
# From the <tools> opener to the clause is TOOLS_SECTION's own text
# minus its final </tools> — the constant itself is unedited (sha-
# pinned in test_prompt_lock) and the three-tool copy is intact.
assert on[on.index("<tools>") : clause_start] == TOOLS_SECTION[: -len("</tools>")]
assert "Answer as soon as you have what you need." in on[:clause_start]
# The clause (then the closing tag) is the tail of the prompt, after
# the existing copy and before the closing tag — exactly once.
assert on.endswith(SHOW_IMAGE_CLAUSE + "\n</tools>")
assert on.index("Answer as soon as you have what you need.") < clause_start
assert clause_start < on.index("</tools>")
assert on.count(SHOW_IMAGE_CLAUSE) == 1
assert on.count("<tools>") == 1 and on.count("</tools>") == 1
def test_show_image_clause_rides_only_the_high_prompt() -> None:
"""The clause appears ONLY in the HIGH prompt: the deflection prompt
takes NO ``show_image`` flag at all and its output never carries the
clause (byte-identical builds with/without notes/overview — a
deflected turn has no tools and never shows an image, the phase's
locked design)."""
assert "show_image" not in inspect.signature(build_deflect_prompt).parameters
for prompt in (
build_deflect_prompt(["T1", "T2"]),
build_deflect_prompt(["T1"], notes=["be concise"], kb_overview=OVERVIEW),
build_deflect_prompt([]),
):
assert SHOW_IMAGE_CLAUSE not in prompt
assert "`show_image`" not in prompt
assert "at most one image per answer" not in prompt
def test_high_prompt_empty_documents_show_image_clause_still_in_tools() -> None:
"""The "(no documents matched…)" body + ``show_image=True``: the
clause still lands inside ``<tools>`` — the grounded turn may still
``ls``/``grep`` its way to an image doc and show it."""
off = build_high_prompt([])
on = build_high_prompt([], show_image=True)
assert on == off[: -len("</tools>")] + SHOW_IMAGE_CLAUSE + "\n</tools>"
assert "(no documents matched — do not invent specifics)" in on
assert on.index("<tools>") < on.index(SHOW_IMAGE_CLAUSE) < on.index("</tools>")
assert on.endswith(SHOW_IMAGE_CLAUSE + "\n</tools>")
def test_show_image_clause_is_last_for_every_high_shape() -> None:
"""Steering + KB overview + folder lines + documents all present:
the section order is UNCHANGED (``<relevance>`` →
``<knowledge_base>`` → ``<tuning>`` → ``<documents>`` → ``<tools>``)
and the clause is still the last thing the model reads, inside
``<tools>`` after the three-tool copy."""
doc = _doc("deploy/k8s.md", "CONTENT_SENTINEL", "K8s Deploy", summary="K8s deploy notes.")
lines = ["Homelab/deploy/: k8s.md (1 doc)"]
off = build_high_prompt([doc], notes=["be concise"], kb_overview=OVERVIEW, folder_lines=lines)
on = build_high_prompt(
[doc],
notes=["be concise"],
kb_overview=OVERVIEW,
folder_lines=lines,
show_image=True,
)
assert on == off[: -len("</tools>")] + SHOW_IMAGE_CLAUSE + "\n</tools>"
clause_start = on.index(SHOW_IMAGE_CLAUSE)
assert on[on.index("<tools>") : clause_start] == TOOLS_SECTION[: -len("</tools>")]
for section in ("<relevance>", "<knowledge_base>", "<tuning>", "<documents>"):
assert on.index(section) < on.index("<tools>")
assert on.index("K8s deploy notes.") < on.index("<tools>") # the summary still seeds
assert SHOW_IMAGE_CLAUSE not in off # the toggle-off twin carries none of it
# ---------- phase 125 (task 05, LOCKED A4): the image context line ----------
# An enriched image doc (tasks 03/04: non-NULL ``image_context`` in the
# "Referenced in …" block shape) leads its ``<document>`` suggestion
# block with ONE short context line — the referring markdown's identity
# + the reference's alt text — before the description, so the LLM knows
# WHY the image exists, not only what it shows. Every other doc (every
# text doc, every non-enriched image doc) is byte-identical to the
# pre-phase body.
#: The exact stored block shape (tasks 03/04): one "Referenced in"
#: block per referring markdown file, blank-line joined; the
#: ``Context:`` body may span many lines (the surrounding paragraphs).
CTX_SINGLE = (
"Referenced in Homelab/homelab/kubernetes.md:\n"
'- Alt text: "Cluster layout"\n'
" Context: The diagram shows the three Talos nodes and Cilium.\n"
"It is referenced again from the runbook."
)
CTX_TWO_BLOCKS = CTX_SINGLE + "\n\nReferenced in Homelab/ops/runbook.md:\n- Alt text: \"\""
CTX_ALT_EMPTY = "Referenced in Homelab/ops/runbook.md:\n- Alt text: \"\""
#: The task-04 pending record — stored on MARKDOWN (text) docs, with no
#: ``Referenced in`` header at all.
MD_PENDING_RECORD = (
'IMAGE_CONTEXT:\n- [image.png] (line 42): "Alt text here" — 2 paragraphs of context'
)
def _image_doc(
image_context: str | None = None, summary: str | None = "A diagram of the cluster."
) -> Document:
return _doc(
"cluster-diagram.png",
"A diagram of the cluster.",
"cluster-diagram",
summary=summary,
is_image=True,
image_context=image_context,
)
# ---------- _extract_context_line: the parser contract ----------
def test_extract_context_line_single_block_with_alt() -> None:
"""One block with an alt → the short one-line locator: the stored
identity (``<source>/<path>`` exactly as the header carries it) +
the reference's alt between quotes."""
assert _extract_context_line(CTX_SINGLE) == (
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"'
)
def test_extract_context_line_context_body_never_renders() -> None:
"""The ``Context:`` paragraphs (LOCKED A3: they already reached the
vision model at enrichment time) stay out of the line — the line is
the locator + purpose, short enough not to bloat the prompt."""
line = _extract_context_line(CTX_SINGLE)
assert line is not None
assert "The diagram shows" not in line
assert "referenced again" not in line
assert "Context:" not in line
assert line.count("\n") == 0 # genuinely ONE line
def test_extract_context_line_empty_alt_renders_the_bare_identity() -> None:
"""A reference without an alt (``![](path)``) → the clause is the
bare identity — nothing to quote after the colon."""
assert _extract_context_line(CTX_ALT_EMPTY) == "Referenced in Homelab/ops/runbook.md"
def test_extract_context_line_two_blocks_join_with_semicolon() -> None:
"""One clause per referring file after a single prefix — the second
block's empty alt renders bare."""
assert _extract_context_line(CTX_TWO_BLOCKS) == (
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"; '
"Homelab/ops/runbook.md"
)
def test_extract_context_line_uses_the_first_reference_only() -> None:
"""A block with two references (two ``- Alt text:`` bullets) → the
FIRST reference's alt is the purpose; the second stays out."""
text = (
"Referenced in Homelab/a.md:\n"
'- Alt text: "first purpose"\n'
" Context: some paragraph.\n"
'- Alt text: "second purpose"'
)
assert _extract_context_line(text) == 'Referenced in Homelab/a.md: "first purpose"'
assert "second purpose" not in (_extract_context_line(text) or "")
def test_extract_context_line_multiline_context_body_is_ignored() -> None:
"""The ``Context:`` value is paragraph-joined (blank lines inside)
— the parser walks it without breaking the block."""
text = (
"Referenced in Homelab/a.md:\n"
'- Alt text: "purpose"\n'
" Context: para one.\n"
"\n"
"para two, and it contains a line that\n"
"looks like context continuation."
)
assert _extract_context_line(text) == 'Referenced in Homelab/a.md: "purpose"'
def test_extract_context_line_alt_with_inner_quotes_survives() -> None:
"""The block writer does not escape quotes: the greedy alt capture
(between the opening quote and the line's final quote) keeps an
alt with inner ``"`` intact."""
text = 'Referenced in Homelab/x.md:\n- Alt text: "He said "hi""'
assert _extract_context_line(text) == 'Referenced in Homelab/x.md: "He said "hi""'
def test_extract_context_line_spurious_header_is_dropped() -> None:
"""A look-alike ``Referenced in …:`` line WITHOUT its ``- Alt text:``
bullet (context text that merely looks like a header) parses to
``None`` — never a clause from the stray line."""
assert _extract_context_line(
"Referenced in Homelab/fake.md:\nsome context line that is not a bullet"
) is None
# A header with NO following line at all (end of value) is spurious too.
assert _extract_context_line("Referenced in Homelab/fake.md:") is None
# Spurious then REAL: only the real block's clause survives.
mixed = (
"Referenced in Homelab/fake.md:\nsome context line\n\n"
+ CTX_SINGLE
)
assert _extract_context_line(mixed) == (
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"'
)
def test_extract_context_line_header_after_header_drops_the_bulletless_one() -> None:
"""Two headers in a row: the first (never followed by its bullet) is
dropped and the second block parses normally."""
text = "Referenced in Homelab/fake.md:\nReferenced in Homelab/real.md:\n- Alt text: \"alt\""
assert _extract_context_line(text) == 'Referenced in Homelab/real.md: "alt"'
def test_extract_context_line_markdown_pending_record_is_none() -> None:
"""The task-04 ``IMAGE_CONTEXT:`` record (the MARKDOWN doc's stored
``image_context``) has no ``Referenced in`` header — it parses to
``None`` (and it rides a text doc, which the ``is_image`` guard
excludes from the line anyway)."""
assert _extract_context_line(MD_PENDING_RECORD) is None
def test_extract_context_line_blank_and_empty_are_none() -> None:
assert _extract_context_line("") is None
assert _extract_context_line(" \n \t ") is None
# Garbage that contains the keyword somewhere is not a block.
assert _extract_context_line("I was referenced in Homelab/a.md: yesterday") is None
# ---------- _document_body: the block-body contract ----------
def test_document_body_enriched_image_doc_leads_with_the_context_line() -> None:
"""The completion criterion at the body level: the context line
leads, a blank line, then the description — the exact two-part
shape."""
body = _document_body(_image_doc(image_context=CTX_SINGLE))
assert body == (
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"\n\n'
"A diagram of the cluster."
)
def test_document_body_image_doc_null_context_is_byte_identical_to_pre_phase() -> None:
"""LOCKED A4 (the byte-identity criterion): a standalone / pre-phase
image doc (NULL ``image_context``) → the body is EXACTLY the
stripped summary — no line, no blank, no drift."""
doc = _image_doc(image_context=None)
assert _document_body(doc) == "A diagram of the cluster."
# And the built prompt matches a text twin (same fields, not an
# image) to the byte.
twin = _doc("cluster-diagram.png", "A diagram of the cluster.", "cluster-diagram")
assert build_high_prompt([doc]) == build_high_prompt([twin])
def test_document_body_text_doc_with_image_context_is_unchanged() -> None:
"""The guard is ``is_image``: a TEXT doc whose ``image_context``
carries the task-04 pending record (the markdown case) keeps the
pre-phase body — the record never renders."""
doc = _doc("guide.md", "CONTENT", "Guide", summary="Guide summary.",
image_context=MD_PENDING_RECORD)
assert _document_body(doc) == "Guide summary."
assert "IMAGE_CONTEXT" not in build_high_prompt([doc])
assert "Referenced in" not in build_high_prompt([doc])
def test_document_body_image_doc_unparsable_context_is_unchanged() -> None:
"""Graceful degradation: an image doc whose ``image_context`` parses
to nothing (owner-edited / unknown shape) → the description only,
exactly pre-phase — the prompt build never raises."""
for junk in ("", " ", "referenced in lowercase.md: no header shape", MD_PENDING_RECORD):
assert _document_body(_image_doc(image_context=junk)) == "A diagram of the cluster."
def test_document_body_enriched_image_doc_without_summary_precedes_the_preview() -> None:
"""The line leads the fallback body too (the preview + marker of
LOCKED A5) — the "context before the description" contract holds on
both body shapes."""
doc = _image_doc(image_context=CTX_SINGLE, summary=None)
doc.content = "A" * 400 + "B" * 600 # past the 400-char preview cap
body = _document_body(doc)
line = 'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"'
assert body.startswith(line + "\n\n")
assert body[len(line) + 2 :] == "A" * 400 + "\n" + TRUNCATION_MARKER
assert "B" * 40 not in body # beyond the cut, as pre-phase
def test_enriched_image_doc_with_summary_makes_no_settings_call(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""House pattern (the LOCKED A5 preview-cap contract): the
context-line path for a summary-bearing doc is pure — a dead
``get_settings`` must not break such a build (extraction is pure;
the cap is read on the fallback path only)."""
from app.rag import prompts as prompts_mod
doc = _image_doc(image_context=CTX_SINGLE)
def _dead() -> None: # pragma: no cover - must never be called
raise AssertionError("get_settings() called for a summarized doc")
monkeypatch.setattr(prompts_mod, "get_settings", _dead)
assert _document_body(doc) == (
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"\n\n'
"A diagram of the cluster."
)
# ---------- build_high_prompt: the line in the built block ----------
def test_high_prompt_enriched_image_doc_block_exact_shape() -> None:
"""The full block in the built prompt: identity attributes
unchanged (phase 106 date appended after title), then the context
line, a blank line, the description — the pre-phase block shape
with the line spliced in."""
doc = _image_doc(image_context=CTX_SINGLE)
prompt = build_high_prompt([doc])
assert (
'<document source="Homelab" path="cluster-diagram.png" '
'title="cluster-diagram" date="2024-06-15">\n'
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"\n'
"\n"
"A diagram of the cluster.\n"
"</document>"
) in prompt
# The line is the FIRST thing in the block body (before the
# description), inside <documents>.
line = 'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"'
block_start = prompt.index('<document source="Homelab"')
assert prompt.index("<documents>") < prompt.index(line)
assert prompt.index(line) < prompt.index("A diagram of the cluster.")
assert block_start < prompt.index(line) < prompt.index("</document>")
# The Context: body stays out of the whole prompt.
assert "The diagram shows the three Talos nodes" not in prompt
def test_high_prompt_enriched_image_doc_mixed_with_text_docs() -> None:
"""A suggested tier with both text docs and one enriched image doc:
the line rides ONLY the image doc's block; the text blocks are
byte-identical to their summaries (no line, no drift)."""
text_doc = _doc("kubernetes.md", "CONTENT", "K8s", summary="K8s summary.")
img = _image_doc(image_context=CTX_SINGLE)
prompt = build_high_prompt([text_doc, img])
assert prompt.count('Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"') == 1
# The text block is the bare summary, exactly as pre-phase.
assert (
'<document source="Homelab" path="kubernetes.md" '
'title="K8s" date="2024-06-15">\nK8s summary.\n</document>'
) in prompt
# The image block carries the line + description.
assert (
'title="cluster-diagram" date="2024-06-15">\n'
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"\n\n'
"A diagram of the cluster.\n</document>"
) in prompt
# Ordering unchanged: the text block (first doc) still precedes the
# image block (second doc) — the line never re-orders the tier.
assert prompt.index('path="kubernetes.md"') < prompt.index('path="cluster-diagram.png"')
def test_high_prompt_image_doc_without_context_is_byte_identical_to_pre_phase() -> None:
"""LOCKED A4 at the prompt level: an image doc with NULL
``image_context`` builds the EXACT pre-phase prompt (the text-twin
equality — no context line anywhere, not even a stray blank)."""
img = _image_doc(image_context=None)
twin = _doc("cluster-diagram.png", "A diagram of the cluster.", "cluster-diagram")
assert build_high_prompt([img]) == build_high_prompt([twin])
assert "Referenced in" not in build_high_prompt([img])
# ---------- phase 137 (task 04, ASSUMPTION A6): the provenance clause ----------
# The stored value of a PDF-embedded image doc (tasks 02/03) can carry
# the ``Extracted from <source>/<pdf path> (page N):`` provenance block
# — written FIRST (A2: provenance before purpose) — ahead of the
# markdown reference blocks. The renderer turns that header line into
# the one clause ``From <source>/<pdf> (page N)`` (the page-text body
# is NEVER rendered), joins clauses in stored order, and keeps the
# markdown-only values byte-identical (the phase-125 pin, unedited).
#: The exact stored provenance block shape with page text (tasks
#: 02/03): the header + the page text (capped, stripped).
PDF_CTX_SINGLE = (
"Extracted from Homelab/docs/guide.pdf (page 2):\n"
"The second page of the guide, with the diagram.\n"
"More page text on the next line."
)
#: The tasks 02/03 COMBINED shape (A2): the document block first, then
#: the markdown reference blocks, blank-line joined.
PDF_CTX_BOTH = (
"Extracted from Homelab/docs/guide.pdf (page 2):\n"
"The second page of the guide, with the diagram.\n"
"\n"
"Referenced in Homelab/ops/runbook.md:\n"
'- Alt text: "alt"'
)
def test_extract_context_line_pdf_block_alone() -> None:
"""One provenance block → the clause EXACTLY: the locator + page
number — the page-text body never renders."""
line = _extract_context_line(PDF_CTX_SINGLE)
assert line == "From Homelab/docs/guide.pdf (page 2)"
assert "The second page of the guide" not in line
assert "More page text" not in line
assert line.count("\n") == 0 # genuinely ONE line
def test_extract_context_line_pdf_block_textless_page_is_header_only() -> None:
"""A text-less page stores the header line alone (tasks 02/03) →
the same single clause."""
assert _extract_context_line(
"Extracted from Homelab/docs/guide.pdf (page 2):"
) == "From Homelab/docs/guide.pdf (page 2)"
def test_extract_context_line_pdf_locator_greedy_to_the_last_page_marker() -> None:
"""The pdf path may itself contain spaces — ``loc`` is greedy to
the LAST `` (page N):`` on the line."""
assert _extract_context_line(
"Extracted from Sources/My Docs/guide v2.pdf (page 10):"
) == "From Sources/My Docs/guide v2.pdf (page 10)"
def test_extract_context_line_pdf_block_page_text_body_is_skipped() -> None:
"""The page-text body (multi-line, with a blank line and a
look-alike ``Referenced in …:`` line WITHOUT the alt bullet) is
skipped / dropped (the existing spurious-header rule) — the clause
stays the single one."""
text = (
"Extracted from Homelab/docs/guide.pdf (page 2):\n"
"Page text line one.\n"
"\n"
"Referenced in the appendix: it goes on like this.\n"
"More page text after the look-alike header."
)
assert _extract_context_line(text) == "From Homelab/docs/guide.pdf (page 2)"
def test_extract_context_line_spurious_pdf_header_in_page_text_emits_a_clause() -> None:
"""The accepted asymmetry (documented in the docstring): a
page-text line that LOOKS like a provenance header is NOT guarded
(unlike the ``Referenced in`` look-alikes, which demand their
bullet) — it emits a clause. The stored value is the context the
importer wrote, not free model text."""
text = (
"Extracted from Homelab/guide.pdf (page 1):\n"
"Page text carrying a look-alike line:\n"
"Extracted from Somewhere/other.pdf (page 9):\n"
"more page text"
)
assert _extract_context_line(text) == (
"From Homelab/guide.pdf (page 1); From Somewhere/other.pdf (page 9)"
)
def test_extract_context_line_markdown_only_two_blocks_pin_unchanged() -> None:
"""The phase-125 byte-identity pin (the task's shape): a
MARKDOWN-ONLY value renders exactly as pre-task — ONE
``Referenced in `` prefix over the joined clauses (the second
block's empty alt renders bare); a single block is unchanged too."""
text = (
'Referenced in docs/a.md:\n- Alt text: "alt a"\n'
"\n"
'Referenced in docs/b.md:\n- Alt text: ""'
)
assert _extract_context_line(text) == 'Referenced in docs/a.md: "alt a"; docs/b.md'
assert _extract_context_line('Referenced in docs/a.md:\n- Alt text: "alt a"') == (
'Referenced in docs/a.md: "alt a"'
)
def test_extract_context_line_both_blocks_document_first() -> None:
"""The tasks 02/03 combined shape (A2: the document block first,
then the markdown blocks) renders the clauses IN STORED ORDER —
each with its own prefix; the page-text body stays out."""
line = _extract_context_line(PDF_CTX_BOTH)
assert line == (
'From Homelab/docs/guide.pdf (page 2); '
'Referenced in Homelab/ops/runbook.md: "alt"'
)
assert "The second page of the guide" not in line
def test_extract_context_line_markdown_first_then_pdf_keeps_stored_order() -> None:
"""The reverse stored order (a markdown block BEFORE the document
block — an owner-edited value) renders in stored order too: the
complete reference block flushes when the provenance header
arrives, and the mixed join gives each clause its own prefix."""
text = (
"Referenced in Homelab/ops/runbook.md:\n"
'- Alt text: "alt"\n'
"\n"
"Extracted from Homelab/docs/guide.pdf (page 2):\n"
"The second page of the guide."
)
assert _extract_context_line(text) == (
'Referenced in Homelab/ops/runbook.md: "alt"; '
"From Homelab/docs/guide.pdf (page 2)"
)
def test_extract_context_line_pdf_garbage_is_none() -> None:
"""No parsable header of either kind → ``None`` (unchanged
degradation): the provenance keyword without the exact header
shape is not a block."""
# No trailing colon ...
assert _extract_context_line("Extracted from Homelab/guide.pdf (page 2)") is None
# ... the lowercase keyword ...
assert _extract_context_line("extracted from Homelab/guide.pdf (page 2):") is None
# ... the missing parentheses.
assert _extract_context_line("Extracted from Homelab/guide.pdf page 2:") is None
def test_document_body_pdf_image_doc_leads_with_the_from_line() -> None:
"""End-to-end: a PDF-embedded image doc's ``<document>`` body
leads with the one-line rendering (provenance first, then the
reference), a blank line, then the description — the page-text
body never renders."""
body = _document_body(_image_doc(image_context=PDF_CTX_BOTH))
assert body == (
'From Homelab/docs/guide.pdf (page 2); '
'Referenced in Homelab/ops/runbook.md: "alt"\n'
"\n"
"A diagram of the cluster."
)
assert "The second page of the guide" not in body
def test_document_body_text_doc_with_pdf_context_is_unchanged() -> None:
"""The guard is ``is_image``: a TEXT doc carrying a provenance
context keeps the byte-identical pre-phase body — the context
never renders."""
doc = _doc("guide.pdf", "CONTENT", "Guide", summary="Guide summary.",
image_context=PDF_CTX_SINGLE)
assert _document_body(doc) == "Guide summary."
assert "Extracted from" not in build_high_prompt([doc])
assert "From Homelab/docs/guide.pdf (page 2)" not in build_high_prompt([doc])
def test_high_prompt_pdf_image_doc_block_exact_shape() -> None:
"""The full block in the built prompt: the ``From`` line leads the
body (provenance first) before the description — the pre-phase
block shape with the line spliced in; the page-text body stays out
of the whole prompt."""
doc = _image_doc(image_context=PDF_CTX_SINGLE)
prompt = build_high_prompt([doc])
assert (
'<document source="Homelab" path="cluster-diagram.png" '
'title="cluster-diagram" date="2024-06-15">\n'
"From Homelab/docs/guide.pdf (page 2)\n"
"\n"
"A diagram of the cluster.\n"
"</document>"
) in prompt
line = "From Homelab/docs/guide.pdf (page 2)"
block_start = prompt.index('<document source="Homelab"')
assert prompt.index("<documents>") < prompt.index(line)
assert prompt.index(line) < prompt.index("A diagram of the cluster.")
assert block_start < prompt.index(line) < prompt.index("</document>")
assert "The second page of the guide" not in prompt