All gates re-verified against a stable tree. Report: **Verified (all 6 tasks in `complete/`; this was the final pass)** - `SourceLabel`+alembic `0030` (dev DB `alembic current` = `0030 (head)`, table present, 0 rows); admin-only `GET/PUT/DELETE /api/source-labels` (`require_admin` router-wide, `resolve_source_name`, partial-update caps/404s); Sources-page "Names, labels and tags" card (read-only derived-name `<code>`, per-row `saveInFlight`, 404 note that survives the settle, chips reuse `.git-source-hidden-count`); `question_name_hits` D2 rule; ONE `all_labels` read per turn in `chat.py` feeding both the `source_name_hit` flag and the prompt surfaces; `source_name_hit_bonus=0.005` in `_selection_order`; ` name=`/` label=` in `build_high_prompt` + `build_overview_prompt(names=…)`. - **Completion criteria:** E2E `tests/e2e/test_source_name_labels.py` ✅ 7 passed (arms a–e: editor all kinds, control/named question, rename-never-breaks-identity, 404 note, reset) · full suite ✅ · coverage ✅ · README/AGENTS ✅ · phase dir in `complete/` ✅ · commit left to the harness. - **Gates:** `uv run pytest --cov=app --cov-report=term-missing` → **exit 0, TOTAL 7160 stmts / 16 missed = 99%** (>90%) · `uv run pytest tests/e2e/test_source_name_labels.py -v --no-cov` → **7 passed, exit 0** · regression E2E isolated, exit 0: `test_document_viewer`, `test_admin_auth`, `test_api_tokens`, `test_git_sources_admin`, `test_kb_tree`, `test_local_directory_sources` · `uv run ruff check .` → All checks passed · `uv run pyright` → 0 errors. - **Battery (task 04 criterion):** `uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt` at default vs `BOR_SOURCE_NAME_HIT_BONUS=0` → **`diff` empty (line-for-line identical)**, matching the recorded arms (dev KB is empty, so every question deflects; the observable boost is unit/E2E-pinned). - **Documentation:** README — fixed the `/git-sources.html` rows (nav label is **Sources**) and cross-linked the phase-191 card from that section; full audit otherwise clean: all 86 `BOR_*` settings documented with defaults matching `app/config.py`, 0 broken file/dir references and 0 broken anchors in README+AGENTS, AGENTS rule 17 re-checked line-by-line against the code. - **Notable / deviations:** (1) a **concurrent pipeline process** (`phased-execution/progress.mjs` for `191_…_00_phase`) was mutating this same working tree mid-pass — it rewrote `tests/e2e/test_source_name_labels.py` (11→7 tests), `frontend/assets/git-sources.js`, `tests/unit/test_source_labels_editor_ui.py` and `AGENTS.md` at 05:27; every gate above was (re-)run after those edits with the tree no source file newer than the green full run. (2) The task-06 report/phase text claims regression suites `test_chat_token_gate.py`, `test_git_sources_page.py`, `test_rag_browser.py` are green — **those files have never existed in git history** (those runs exit 4); I ran the real equivalents listed above instead (phase files are not mine to edit). (3) **No live changes**: the VM DB already has `source_labels` (0 rows) but the deployed app predates this uncommitted code, so deploying `/api/source-labels` to the live service is the owner's post-commit step, not part of this phase's criteria. - **Next pending phase:** none — `191_source_name_labels_tags` is the only entry in `.agents/phases/todo/`.
2156 lines
99 KiB
Python
2156 lines
99 KiB
Python
"""Unit: locked persona prompt builder (PLAN §6 verbatim + both modes).
|
||
|
||
Also covers the phase-31 ``<knowledge_base>`` section (the stored,
|
||
lite-generated KB outline): its own builder contract (empty → ``""``,
|
||
char budget + ``[…truncated…]`` marker, pathological budgets), its
|
||
placement between ``<relevance>`` and ``<tuning>`` in both modes, and
|
||
the byte-identical-when-absent convention (phase 15 precedent).
|
||
(Phase 142, task 03: the section's intro is the honest-copy rewrite —
|
||
a compact, possibly partial map that favors recent documents, with the
|
||
verify-with-``ls``/``grep`` teaching for "latest / complete set"
|
||
questions — pinned verbatim; the false "(generated at import time)"
|
||
phrase is gone.)
|
||
|
||
And the phase-71 deflection plain-text line (owner-permitted
|
||
2026-09-03): the LOW prompt = pre-phase text + exactly the one new
|
||
line; the ``DEFLECT_MODE`` marker-keying contract is unchanged and
|
||
the line never leaks into the HIGH prompt.
|
||
|
||
And the ``<tools>`` copy (phase 72: the document-identity contract
|
||
stated up front — the combined ``source/path`` identity for
|
||
``read``/``grep``; phase 94, task 03: the ``ls`` clause rewritten to
|
||
the drill-down tree contract — one level per call, sources at the
|
||
top, folders + files below, ``grep`` as the without-listing locator —
|
||
while the ``read``/``grep`` clauses and the discipline rules are
|
||
byte-identical; phase 118, task 04: the ``read`` clause rewritten
|
||
for the summary-seed mode — the ``<documents>`` section holds
|
||
SUMMARIES, ``read`` adds the full text, and only an already-read
|
||
document is refused — while the ``ls``/``grep`` clauses and the
|
||
discipline rules stay byte-identical): the teaching refusals in
|
||
:mod:`app.rag.agent` re-state the same contract; the ``<tools>``
|
||
marker keying (HIGH only) is unchanged.
|
||
|
||
And the phase-182 ``show_text`` clause (task 04, LOCKED A1–A3/A6): the
|
||
verbatim-display teaching appended INSIDE ``<tools>`` when
|
||
``BOR_ANSWER_TEXT`` is on — the user copies from what is DISPLAYED, never
|
||
from the model's retype, and the inline ```bor <source/path> [range]```
|
||
fence places a server-resolved block. The clause composes with the
|
||
phase-124 ``show_image`` clause through one mechanism, and every build
|
||
the flags do not select stays byte-identical (the constant's own sha pin
|
||
lives in :mod:`tests.unit.test_prompt_lock`).
|
||
|
||
And the phase-191 source names (task 05, D4): a ``<document>`` block can
|
||
carry the owner's label row as the ADDITIVE ``name`` / ``label``
|
||
attributes (order ``source`` → ``name?`` → ``label?`` → ``path`` →
|
||
``title`` → ``date``, XML-escaped values), with no map / no entry for the
|
||
source byte-identical to pre-phase (the kill pin).
|
||
|
||
And the phase-119 cite discipline (task 04, LOCKED A5): the intro's
|
||
final sentence ("cite the document(s) you used, by path", phase 118)
|
||
is REPLACED — the HIGH prompt carries the discipline sentence exactly
|
||
once, inside ``<documents>`` after the intro's start-here framing;
|
||
the LOW prompt and ``PERSONA`` / ``TOOLS_SECTION`` stay byte-identical
|
||
(the full sha pins live in :mod:`tests.unit.test_prompt_lock`).
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import inspect
|
||
import re
|
||
import uuid
|
||
from datetime import UTC, datetime
|
||
|
||
import pytest
|
||
|
||
from app.config import Settings
|
||
from app.models import Document
|
||
from app.rag.prompts import (
|
||
ORIGIN_RULE,
|
||
PERSONA,
|
||
SEARCH_DISCIPLINE_RULE,
|
||
SHOW_IMAGE_CLAUSE,
|
||
SHOW_TEXT_CLAUSE,
|
||
SUGGEST_INTRO,
|
||
TOOLS_SECTION,
|
||
_attr,
|
||
_base,
|
||
_document_body,
|
||
_extract_context_line,
|
||
build_deflect_prompt,
|
||
build_high_prompt,
|
||
build_kb_section,
|
||
build_steering_section,
|
||
)
|
||
from app.rag.retriever import TRUNCATION_MARKER
|
||
from app.rag.source_labels import SourceLabelView
|
||
|
||
#: A small multi-line outline standing in for the lite-generated one.
|
||
OVERVIEW = "- Homelab\n - Kubernetes (k3s)\n- Deployments\n - Borg backups"
|
||
|
||
#: The locked one-line intro of the ``<knowledge_base>`` section
|
||
#: (phase 142, task 03 — the honest-intro rewrite, pinned verbatim in
|
||
#: :func:`test_kb_intro_phase142_honest_copy`) — the intro line WITHOUT
|
||
#: its trailing newline (the section renderers / the f-strings below
|
||
#: supply the line break after it, as the pre-phase-142 constant did).
|
||
KB_INTRO = (
|
||
'A compact picture of the knowledge base\'s categories and topics — '
|
||
'it favors recent documents and may omit older or lesser ones: a map '
|
||
'of what exists, not a complete inventory. For questions about the '
|
||
'latest state or the complete set ("the last N …", "everything '
|
||
'about …", "all the …"), verify with `ls` (drill into the folders) '
|
||
'or `grep` before answering from the outline:'
|
||
)
|
||
|
||
|
||
#: The fixture documents' fixed creation date (phase 106, D5) — the
|
||
#: ``<document>`` block formats its UTC date part (the detached fixture
|
||
#: rows carry it exactly as the NOT NULL DB column guarantees it for
|
||
#: real rows).
|
||
_FIXTURE_CREATED_AT = datetime(2024, 6, 15, 12, 0, 0, tzinfo=UTC)
|
||
|
||
|
||
def _doc(
|
||
path: str,
|
||
content: str,
|
||
title: str,
|
||
summary: str | None = None,
|
||
*,
|
||
is_image: bool = False,
|
||
image_context: str | None = None,
|
||
) -> Document:
|
||
return Document(
|
||
id=uuid.uuid4(),
|
||
source="Homelab",
|
||
path=path,
|
||
full_path=f"/tmp/{path}",
|
||
title=title,
|
||
content=content,
|
||
content_hash="0" * 64,
|
||
created_at=_FIXTURE_CREATED_AT,
|
||
summary=summary,
|
||
is_image=is_image,
|
||
image_path="/tmp/pic.png" if is_image else None,
|
||
image_context=image_context,
|
||
)
|
||
|
||
|
||
def test_persona_rules_present_verbatim() -> None:
|
||
# Aligned to the owner's working-tree persona edits (PLAN §6 revision,
|
||
# 2026-08-22): no "you've got this" tagline, no mandated deflection
|
||
# opening. The honesty gate itself (rule 3) is unchanged.
|
||
for fragment in (
|
||
'You are "Brain of Reese" — the digital brain of Reese, a self-hoster and',
|
||
"optimistic about the user's ability to do things",
|
||
"Answer ONLY from the provided document context. Cite which document(s)",
|
||
"you used, by path.",
|
||
"Be concrete: names, versions, ports, hosts, schedules",
|
||
'HONESTY GATE: if <relevance> is "LOW", you must NOT pretend to know',
|
||
"Offer 2-3 alternative questions about things you DO have notes on.",
|
||
"Never invent facts, hosts, or steps that are not in the context.",
|
||
"Keep answers tight: short paragraphs, bullets where helpful.",
|
||
):
|
||
assert fragment in PERSONA
|
||
|
||
|
||
def test_persona_owner_edits_are_preserved() -> None:
|
||
"""PLAN §6 revision (2026-08-22): the removed elements must stay out."""
|
||
assert 'you\'ve got this' not in PERSONA # tagline removed by the owner
|
||
assert "Start your answer with a variant of" not in PERSONA # no mandated opening
|
||
assert "HONESTY GATE" in PERSONA # the gate itself is intact
|
||
|
||
|
||
def test_high_prompt_carries_relevance_marker_and_summary_block() -> None:
|
||
"""Phase 118 (LOCKED A6): the grounded turn seeds the document's
|
||
SUMMARY — the full content never reaches the prompt (the ``read``
|
||
tool is the only full-text path)."""
|
||
doc = _doc(
|
||
"kubernetes.md",
|
||
"Talos Linux on three nodes. FULL_CONTENT_SENTINEL_987654",
|
||
"Kubernetes Homelab Cluster",
|
||
summary="A Talos Linux cluster on three nodes.",
|
||
)
|
||
prompt = build_high_prompt([doc])
|
||
assert "<relevance>HIGH</relevance>" in prompt
|
||
assert "DEFLECT_MODE" not in prompt
|
||
assert "<documents>" in prompt and "</documents>" in prompt
|
||
assert 'path="kubernetes.md"' in prompt
|
||
assert "A Talos Linux cluster on three nodes." in prompt # the summary
|
||
assert "FULL_CONTENT_SENTINEL_987654" not in prompt # never the content
|
||
assert "Talos Linux on three nodes." not in prompt # nor the full sentence
|
||
assert "HONESTY GATE" in prompt # persona intact
|
||
|
||
|
||
def test_high_prompt_lists_multiple_documents_in_order() -> None:
|
||
a = _doc("a.md", "CONTENT_A", "Title A", summary="Summary A.")
|
||
b = _doc("b.md", "CONTENT_B", "Title B", summary="Summary B.")
|
||
prompt = build_high_prompt([a, b])
|
||
assert prompt.index("Summary A.") < prompt.index("Summary B.")
|
||
assert 'title="Title B"' in prompt
|
||
assert "CONTENT_A" not in prompt and "CONTENT_B" not in prompt
|
||
|
||
|
||
def test_high_prompt_seeds_summaries_of_all_suggested_docs() -> None:
|
||
"""The phase-118 completion pin: a grounded prompt built from five
|
||
summary-bearing documents contains ALL FIVE summaries + the intro,
|
||
and ZERO full-content characters (the full texts are not in the
|
||
prompt at all)."""
|
||
docs = [
|
||
_doc(
|
||
f"doc{i}.md",
|
||
f"FULL_CONTENT_SENTINEL_{i} " + "x" * 100,
|
||
f"Title {i}",
|
||
summary=f"SUMMARY_{i} of the document.",
|
||
)
|
||
for i in range(5)
|
||
]
|
||
prompt = build_high_prompt(docs)
|
||
assert SUGGEST_INTRO in prompt
|
||
for i in range(5):
|
||
assert f"SUMMARY_{i} of the document.\n" in prompt
|
||
assert f"FULL_CONTENT_SENTINEL_{i}" not in prompt
|
||
assert f'path="doc{i}.md"' in prompt
|
||
# One block per document, in the given order.
|
||
assert prompt.count("<document ") == 5
|
||
positions = [prompt.index(f"SUMMARY_{i}") for i in range(5)]
|
||
assert positions == sorted(positions)
|
||
# The intro leads the section: <documents> → intro → first block.
|
||
i_open = prompt.index("<documents>")
|
||
i_block = prompt.index("<document ")
|
||
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n\n"
|
||
|
||
|
||
def test_high_prompt_without_documents_stays_honest() -> None:
|
||
prompt = build_high_prompt([])
|
||
assert "<documents>" in prompt
|
||
assert "do not invent specifics" in prompt
|
||
|
||
|
||
def test_low_prompt_has_deflect_mode_and_titles_only() -> None:
|
||
titles = ["Kubernetes Homelab Cluster", "Backup Strategy"]
|
||
prompt = build_deflect_prompt(titles)
|
||
assert "<relevance>LOW</relevance>" in prompt
|
||
assert "DEFLECT_MODE" in prompt # marker the E2E mock keys on
|
||
assert "- Kubernetes Homelab Cluster" in prompt
|
||
assert "- Backup Strategy" in prompt
|
||
|
||
|
||
def test_low_prompt_never_contains_document_content() -> None:
|
||
secret = "SECRET_DOCUMENT_CONTENT_12345"
|
||
prompt = build_deflect_prompt(["Some Title"])
|
||
assert secret not in prompt
|
||
assert "<documents>" not in prompt
|
||
assert "HONESTY GATE" in prompt # the LOW rule is what the model must follow
|
||
|
||
|
||
def test_low_prompt_with_no_titles() -> None:
|
||
assert "nothing close at all" in build_deflect_prompt([])
|
||
|
||
|
||
def test_zero_note_prompt_is_byte_identical_to_pre_steering() -> None:
|
||
"""Phase 15 contract: with no steering notes the prompt is exactly what
|
||
it was before the <tuning> section existed. (Phase 37: the HIGH prompt
|
||
additionally carries the ``<tools>`` section after the mode body — the
|
||
fixtures account for it; phase 118: the ``<documents>`` section leads
|
||
with the ``SUGGEST_INTRO`` line — the fixture accounts for it; the
|
||
LOW prompt is untouched.)"""
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||
block = (
|
||
'<document source="Homelab" path="kubernetes.md" '
|
||
'title="Kubernetes Homelab Cluster" date="2024-06-15">\n'
|
||
"Talos Linux on three nodes.\n"
|
||
"</document>"
|
||
)
|
||
assert build_high_prompt([doc]) == (
|
||
_base("HIGH")
|
||
+ "\n<documents>\n"
|
||
+ SUGGEST_INTRO
|
||
+ "\n\n"
|
||
+ block
|
||
+ "\n</documents>"
|
||
+ "\n"
|
||
+ TOOLS_SECTION
|
||
)
|
||
# Phase 71: the LOW prompt carries the owner-permitted plain-text
|
||
# line after the DEFLECT_MODE sentence (the marker-keying contract
|
||
# is unchanged — the E2E mock keys on the marker's presence).
|
||
assert build_deflect_prompt(["T1", "T2"]) == (
|
||
_base("LOW")
|
||
+ "\nDEFLECT_MODE: retrieval was weak — the titles below are the closest "
|
||
"your notes come to the question. They are titles only; do not pretend "
|
||
"they answer it. Use them to propose 2-3 alternative questions.\n"
|
||
"Reply in plain text only — you have no tools in this mode.\n"
|
||
+ "- T1\n- T2"
|
||
)
|
||
assert "<tuning>" not in build_high_prompt([doc])
|
||
assert "<tuning>" not in build_deflect_prompt([])
|
||
# Phase 70: the rewritten <tools> copy stays out of the LOW path —
|
||
# the byte-identical equality above already proves it; this names
|
||
# the contract (no <tools>, no new copy) on both empty/non-empty LOW
|
||
# builds.
|
||
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
|
||
assert "<tools>" not in low
|
||
assert TOOLS_SECTION not in low
|
||
|
||
|
||
# ---------- <tools> section copy (phase 70: ls / read / grep) ----------
|
||
|
||
|
||
def test_tools_section_markers_and_new_tool_names() -> None:
|
||
"""Phase 70: the section keeps the ``<tools>``/``</tools>`` markers
|
||
the E2E mock keys on and teaches the harness-aligned tool names
|
||
(backticked, exactly as the ``AGENT_TOOLS`` schemas name them)."""
|
||
assert TOOLS_SECTION.startswith("<tools>\n")
|
||
assert TOOLS_SECTION.rstrip().endswith("</tools>")
|
||
for tool in ("`ls`", "`grep`", "`read`"):
|
||
assert tool in TOOLS_SECTION
|
||
|
||
|
||
def test_tools_section_teaches_the_harness_shapes() -> None:
|
||
"""Copy pins: ``ls``'s phase-63 catalog-line format, ``grep``'s
|
||
case- and accent-insensitive exact-string locator contract (up to 20
|
||
``source/path:line: text`` lines, a locator not a context-adder),
|
||
and ``read``'s combined ``source/path`` + full content. (Phase 72:
|
||
the source-name scope clause and the combined-identity clause are
|
||
pinned byte-for-byte in
|
||
:func:`test_tools_section_phase72_contract_clauses`.)"""
|
||
assert "source: X | path: Y | title: Z" in TOOLS_SECTION
|
||
assert "case- and accent-insensitive" in TOOLS_SECTION # phase 162 fold
|
||
assert "up to 20" in TOOLS_SECTION
|
||
assert "source/path:line: text" in TOOLS_SECTION
|
||
assert "locator, not a context-adder" in TOOLS_SECTION
|
||
assert "combined `source/path`" in TOOLS_SECTION
|
||
assert "full content" in TOOLS_SECTION
|
||
assert "Answer as soon as you have what you need" in TOOLS_SECTION
|
||
|
||
|
||
def test_tools_section_phase72_contract_clauses() -> None:
|
||
"""Phase 72 + phase 94: the contract clauses the teaching refusals
|
||
re-state after the fact, pinned byte-for-byte in the constant —
|
||
the ``ls`` clause (phase 94: the drill-down tree contract — one
|
||
level per call, sources at the top, folders + files below, never
|
||
the whole KB in one call, ``grep`` as the without-listing locator)
|
||
and the ``read``/``grep`` combined-identity clause (the combined
|
||
``source/path`` string exactly as shown in the ``ls`` output,
|
||
*including the source name*; a bare document path will not
|
||
resolve)."""
|
||
# The ls drill-down clauses (phase 94, task 03).
|
||
assert "one level at a time" in TOOLS_SECTION
|
||
assert "lists every synced source with its document count" in TOOLS_SECTION
|
||
assert "that source's top-level folders and files" in TOOLS_SECTION
|
||
assert "never the whole knowledge base in one call" in TOOLS_SECTION
|
||
assert "to find one specific document without listing, use `grep`" in TOOLS_SECTION
|
||
# The read combined-identity clause (byte-identical across phases).
|
||
assert (
|
||
"combined `source/path` string, exactly as shown in the `ls` "
|
||
"output — including the source name"
|
||
) in TOOLS_SECTION
|
||
# The bare-path note: read clause AND grep clause (exactly twice).
|
||
assert TOOLS_SECTION.count(
|
||
"a bare document path (without the source name) will not resolve"
|
||
) == 2
|
||
# The pre-phase-70 scope wording is gone — replaced by the
|
||
# explicit source-name contract (and the phase-72 source-name-only
|
||
# clause by the phase-94 drill-down contract).
|
||
assert "pass a source name as `path`" not in TOOLS_SECTION
|
||
assert "not a directory or file path" not in TOOLS_SECTION
|
||
|
||
|
||
def test_tools_section_phase95_read_truncation_clause() -> None:
|
||
"""Phase 95 (task 01): the ``read`` teaching gains exactly one line —
|
||
very large documents are capped, a cut read returns the first part
|
||
plus the TRUNCATED notice (the document did not end where it
|
||
stopped), and ``grep`` is the follow-up (it searches the whole
|
||
document). The ``ls``/``grep`` teaching is untouched (phase 94 owns
|
||
``ls``) — the clause is pinned byte-for-byte in the constant."""
|
||
assert (
|
||
"Very large documents are capped: a cut read returns the first "
|
||
"part plus a TRUNCATED notice — the document did not end where "
|
||
"it stopped; use `grep` (pattern) to find the rest, it searches "
|
||
"the whole document."
|
||
) in TOOLS_SECTION
|
||
# It rides the HIGH prompt and never the LOW (deflected) prompt.
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes")
|
||
assert "Very large documents are capped" in build_high_prompt([doc])
|
||
|
||
|
||
def test_tools_section_phase118_summary_seed_read_clause() -> None:
|
||
"""Phase 118 (task 04, A6): the ``read`` clause is rewritten for
|
||
the summary-seed mode — the ``<documents>`` section holds
|
||
SUMMARIES (a suggested document's full text is not in the prompt
|
||
until ``read`` adds it); do not re-read a document already read
|
||
IN FULL (answer from the text already in the prompt); a PARTIAL
|
||
document (a range read, or a truncated read — re-pinned phase
|
||
141, task 02: the re-read sentence is the coverage-true form —
|
||
read the other parts with ``offset``/``limit``, the document is
|
||
not complete in the context until read all); if the user asks to
|
||
open or read a suggested document, ``read`` it. The phase-72 "do
|
||
not call ``read`` for a ``<documents>`` document" copy is retired.
|
||
The ``ls`` clause (phase 94 drill-down contract), the ``grep``
|
||
clause, and the discipline rules stay byte-identical — the
|
||
prompt lock's prefix/suffix anchors survive (see
|
||
``test_prompt_lock``)."""
|
||
# The summary-seed copy (pinned byte-for-byte — the phase-141
|
||
# coverage-true re-read sentence re-pins the pre-phase-141 line;
|
||
# the section's other bytes are unchanged).
|
||
assert (
|
||
"The <documents> section holds SUMMARIES — the full text of a "
|
||
"suggested document is not in your prompt until you `read` it. "
|
||
"Do not re-read a document you have already read in full — "
|
||
"its full text is in your prompt, answer from it. If you have "
|
||
"read only part of a document (a range read, or a truncated "
|
||
"read), read the other parts with `offset` and `limit` — the "
|
||
"document is not complete in your context until you have read "
|
||
"it all. If the user asks you to open or read a suggested "
|
||
"document, `read` it — that is the point of the section."
|
||
) in TOOLS_SECTION
|
||
# The retired pre-phase-141 re-read copy is gone (it claimed the
|
||
# FULL text for a document that may be only partially in the
|
||
# context — the owner's stuck-turn bug).
|
||
assert (
|
||
"Do not re-read a document you have already read — its full "
|
||
"text is already in your prompt; answer directly from it"
|
||
) not in TOOLS_SECTION
|
||
# The read identity handoff now also names the <documents> summary
|
||
# blocks (the combined identity is shown there), keeping the
|
||
# "including the source name" contract.
|
||
assert (
|
||
"exactly as shown in the `ls` output — including the source "
|
||
"name — or in the <documents> summary blocks — adding its "
|
||
"full content to your context"
|
||
) in TOOLS_SECTION
|
||
# The retired pre-phase-118 copy is gone.
|
||
assert "Do not call `read` for a document already shown in" not in TOOLS_SECTION
|
||
assert "even when the user asks you to open or read it" not in TOOLS_SECTION
|
||
# The ls clause (phase 94 drill-down contract) — byte-identical.
|
||
assert (
|
||
"`ls` lists the knowledge base as a tree, one level at a "
|
||
"time: with no `path` it lists every synced source with its "
|
||
"document count and a summary of its contents; with a source "
|
||
"name (e.g. 'homelab') it lists that source's top-level "
|
||
"folders and files; with a `source/folder` path it drills one "
|
||
"level deeper. A listing shows only that level's subfolders "
|
||
"and its own files — never the whole knowledge base in one "
|
||
"call — and each folder line's summary says what the folder "
|
||
"contains before you drill into it. File lines are `source: X "
|
||
"| path: Y | title: Z`; to find one specific document without "
|
||
"listing, use `grep`."
|
||
) in TOOLS_SECTION
|
||
# The grep clause — folded contract (phase 162, LOCKED A3), up to the
|
||
# search-discipline sentence, byte-identical.
|
||
assert (
|
||
"`grep` locates an exact string (case- and accent-insensitive — "
|
||
"accents fold: `besta` matches `BESTÅ`) in the "
|
||
"indexed documents and returns up to 20 matching "
|
||
"`source/path:line: text` lines — a locator, not a "
|
||
"context-adder: read the winner with `read`. A grep pattern "
|
||
"is a plain substring, NEVER a regex — '.*' and '\\.' are "
|
||
"literal text there; if such a pattern returns no matches, "
|
||
"retry with the plain text you expect to see. "
|
||
) in TOOLS_SECTION
|
||
# The LOCKED search-discipline sentence sits immediately after the
|
||
# grep sentence, before the path clause (phase 162, LOCKED A4).
|
||
assert (
|
||
"retry with the plain text you expect to see. "
|
||
+ SEARCH_DISCIPLINE_RULE
|
||
+ " For a normal search pass only `pattern` — its optional "
|
||
"`path` argument limits the search to one document you already "
|
||
"know, by the same combined `source/path` string; never a "
|
||
"source name — a bare document path (without the source name) "
|
||
"will not resolve there either."
|
||
) in TOOLS_SECTION
|
||
# The discipline rules — byte-identical.
|
||
assert (
|
||
"Make exactly one tool call per reply — a reply carrying two "
|
||
"tool calls runs only the first, the second is discarded — "
|
||
"and wait for the result before the next call. Never repeat a "
|
||
"call that was refused or already succeeded — the refusal "
|
||
"already told you the correct form. Answer as soon as you "
|
||
"have what you need."
|
||
) in TOOLS_SECTION
|
||
# The new read clause rides the built HIGH prompt and never the
|
||
# LOW (deflected) prompt.
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes", summary="K8S")
|
||
high = build_high_prompt([doc])
|
||
assert "The <documents> section holds SUMMARIES" in high
|
||
assert "`read` it — that is the point of the section." in high
|
||
low = build_deflect_prompt(["T1"])
|
||
assert "The <documents> section holds SUMMARIES" not in low
|
||
|
||
|
||
def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
|
||
"""Phase 72/94: the contract clauses ride the HIGH prompt with the
|
||
rest of the section and never leak into the LOW/deflection prompt
|
||
(whose byte-identity is pinned in
|
||
:func:`test_zero_note_prompt_is_byte_identical_to_pre_steering`)."""
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||
high = build_high_prompt([doc])
|
||
assert "<tools>" in high
|
||
assert "one level at a time" in high # the phase-94 ls clause
|
||
assert "including the source name" in high
|
||
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
|
||
assert "<tools>" not in low
|
||
assert "one level at a time" not in low
|
||
assert "including the source name" not in low
|
||
|
||
|
||
def test_tools_section_phase145_origin_rule() -> None:
|
||
"""Phase 145 (task 02): the ``<tools>`` ``read`` clause carries the
|
||
original-source rule (LOCKED verbatim in :data:`ORIGIN_RULE`) — when
|
||
the answer uses a document whose read result carries an ``origin:``
|
||
line, it includes the original source (a link to the repository
|
||
from the origin line, with the document's path and the pinned branch
|
||
where shown) so the user opens the complete original in one step.
|
||
The rule rides the HIGH prompt (``<tools>`` is HIGH-only) and never
|
||
the LOW/deflection prompt. The citation discipline is UNCHANGED —
|
||
the rule extends the ANSWERS of cited documents; it does not widen
|
||
the cite set (the "cite only what you read" sentence stays
|
||
byte-identical — pinned in ``test_prompt_lock``)."""
|
||
# The rule is present verbatim in the constant, exactly once, and at
|
||
# its exact position (the read clause's end, before the grep clause).
|
||
assert TOOLS_SECTION.count(ORIGIN_RULE) == 1
|
||
assert (
|
||
"it searches the whole document. "
|
||
+ ORIGIN_RULE
|
||
+ " `grep` locates an exact string"
|
||
) in TOOLS_SECTION
|
||
# It rides the built HIGH prompt with the rest of the section.
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes")
|
||
high = build_high_prompt([doc])
|
||
assert ORIGIN_RULE in high
|
||
# ...and never the LOW/deflection prompt (``<tools>`` is HIGH-only).
|
||
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
|
||
assert ORIGIN_RULE not in low
|
||
# The citation discipline is UNCHANGED: the rule does not widen the
|
||
# cite set (the "cite only what you read" sentence is untouched).
|
||
assert (
|
||
"Cite only the document(s) you read — or, if you answered from a "
|
||
"suggested summary without reading it, cite that suggested "
|
||
"document — never a document you neither read nor used."
|
||
) in high
|
||
assert "Cite only the document(s) you read" in SUGGEST_INTRO
|
||
|
||
|
||
def test_documents_section_leads_with_the_suggest_intro() -> None:
|
||
"""Phase 118, task 03: the ``<documents>`` section leads with the
|
||
start-here :data:`SUGGEST_INTRO` line BEFORE the first block (the
|
||
phase-15 ``_STEERING_INTRO`` / phase-31 ``_KB_INTRO`` precedent) —
|
||
only when at least one block is present. This is the summary-as-
|
||
starting-point framing, not the reverted phase-72 do-not-read
|
||
reminder (that copy taught the seed texts as already-read context;
|
||
with A6's summary seeding the blocks are starting points the model
|
||
may ``read`` through to full text)."""
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||
high = build_high_prompt([doc])
|
||
i_open = high.index("<documents>")
|
||
i_block = high.index('<document source="Homelab"')
|
||
assert high[i_open : i_block] == f"<documents>\n{SUGGEST_INTRO}\n\n"
|
||
# The intro occurs exactly once and sits inside the section.
|
||
assert high.count(SUGGEST_INTRO) == 1
|
||
assert high.index(SUGGEST_INTRO) > high.index("<documents>")
|
||
assert high.index(SUGGEST_INTRO) < high.index("</documents>")
|
||
|
||
|
||
def test_documents_section_without_blocks_has_no_intro() -> None:
|
||
"""The intro rides only on present blocks: an empty ``<documents>``
|
||
section is exactly the fallback line again (no intro, no blocks)."""
|
||
high = build_high_prompt([])
|
||
assert SUGGEST_INTRO not in high
|
||
i_open = high.index("<documents>")
|
||
i_close = high.index("</documents>")
|
||
assert (
|
||
high[i_open : i_close]
|
||
== "<documents>\n(no documents matched — do not invent specifics)\n"
|
||
)
|
||
|
||
|
||
# ---------- phase 119 (D3, LOCKED A4): the suggested-folder lines ----------
|
||
|
||
|
||
def test_high_prompt_folder_lines_after_intro_before_first_block() -> None:
|
||
"""The folder lines ride the ``<documents>`` section immediately
|
||
AFTER the ``SUGGEST_INTRO`` line — each on its own line — then a
|
||
blank line, then the first ``<document>`` block. Plain lines: no
|
||
new markup/tag anywhere (the E2E mock keys off the ``<documents>``
|
||
marker and the LAST block's tail)."""
|
||
doc = _doc(
|
||
"deploy/Deployments/reeseapps/gitea/app/gitea-web.env.j2",
|
||
"FULL_CONTENT_SENTINEL_119",
|
||
"Gitea Web Env",
|
||
summary="Gitea web env file.",
|
||
)
|
||
lines = [
|
||
"Homelab/deploy/Deployments/reeseapps/gitea/: app/ (5 docs), README.md",
|
||
"Homelab/deploy/Deployments/reeseapps/: gateway/ (3 docs), gitea/ (8 docs)",
|
||
]
|
||
prompt = build_high_prompt([doc], folder_lines=lines)
|
||
i_open = prompt.index("<documents>")
|
||
i_block = prompt.index("<document ")
|
||
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n{lines[0]}\n{lines[1]}\n\n"
|
||
# Each line occurs exactly once, inside the section, before the
|
||
# first block (never after a summary — the E2E tail echo is safe).
|
||
for line in lines:
|
||
assert prompt.count(line) == 1
|
||
assert prompt.index("<documents>") < prompt.index(line) < prompt.index("<document ")
|
||
assert prompt.index(line) < prompt.index("</documents>")
|
||
# The <document> block markup AND body stay byte-identical: from the
|
||
# first block on, the prompt equals the no-folder-line build.
|
||
plain = build_high_prompt([doc])
|
||
assert prompt[prompt.index("<document ") :] == plain[plain.index("<document ") :]
|
||
# No new markup: the section still opens/closes exactly once (the
|
||
# ``<documents>`` MENTION in the TOOLS_SECTION copy is plain text —
|
||
# count the tag + newline, not the bare substring).
|
||
assert prompt.count("<documents>\n") == 1
|
||
assert prompt.count("</documents>") == 1
|
||
assert prompt.count("<document ") == 1
|
||
|
||
|
||
def test_high_prompt_single_folder_line_exact_shape() -> None:
|
||
"""One folder line: the exact slice between the section open and the
|
||
first block is ``<documents>\n`` + intro + ``\n`` + line +
|
||
``\n\n``."""
|
||
doc = _doc("a.md", "CONTENT", "Title A", summary="Summary A.")
|
||
line = "Homelab/: a.md (2 docs), b/ (1 doc)"
|
||
prompt = build_high_prompt([doc], folder_lines=[line])
|
||
i_open = prompt.index("<documents>")
|
||
i_block = prompt.index("<document ")
|
||
assert prompt[i_open:i_block] == f"<documents>\n{SUGGEST_INTRO}\n{line}\n\n"
|
||
|
||
|
||
def test_high_prompt_empty_folder_lines_byte_identical_to_phase_118() -> None:
|
||
"""LOCKED A4: empty ``folder_lines`` (the default) ⇒ the phase-118
|
||
build, byte-identical — the existing pins keep passing and this
|
||
pins the default explicitly (both an explicit ``()`` and the
|
||
omitted parameter)."""
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||
block = (
|
||
'<document source="Homelab" path="kubernetes.md" '
|
||
'title="Kubernetes Homelab Cluster" date="2024-06-15">\n'
|
||
"Talos Linux on three nodes.\n"
|
||
"</document>"
|
||
)
|
||
expected = (
|
||
_base("HIGH")
|
||
+ "\n<documents>\n"
|
||
+ SUGGEST_INTRO
|
||
+ "\n\n"
|
||
+ block
|
||
+ "\n</documents>"
|
||
+ "\n"
|
||
+ TOOLS_SECTION
|
||
)
|
||
# An explicit empty tuple AND the omitted parameter (the default):
|
||
assert build_high_prompt([doc], folder_lines=()) == expected
|
||
assert build_high_prompt([doc]) == expected
|
||
# The LOW (deflected) prompt has no folder_lines parameter at all —
|
||
# the deflected build stays byte-identical (the phase-118 pin in
|
||
# test_zero_note_prompt_is_byte_identical_to_pre_steering stands).
|
||
|
||
|
||
def test_folder_lines_ride_only_on_present_blocks() -> None:
|
||
"""Like the intro, the folder lines ride on present blocks ONLY: an
|
||
empty ``<documents>`` section is unchanged (no lines, no blocks)."""
|
||
prompt = build_high_prompt([], folder_lines=["X/: y.md"])
|
||
assert prompt == build_high_prompt([])
|
||
assert "X/: y.md" not in prompt
|
||
|
||
|
||
# ---------- phase 119 (D4, LOCKED A5): the cite-discipline sentence ----------
|
||
|
||
#: The LOCKED A5 sentence (phase 119, task 04) — the exact replacement
|
||
#: for the retired phase-118 final sentence of :data:`SUGGEST_INTRO`
|
||
#: ("Cite the document(s) you used, by path."). Closes the live
|
||
#: confabulation: the answer's "Docs used:" line cited a file the
|
||
#: agent never read.
|
||
CITE_DISCIPLINE = (
|
||
"Cite only the document(s) you read — or, if you answered from a "
|
||
"suggested summary without reading it, cite that suggested "
|
||
"document — never a document you neither read nor used."
|
||
)
|
||
|
||
|
||
def test_high_prompt_carries_cite_discipline_exactly_once_in_documents() -> None:
|
||
"""LOCKED A5: the HIGH prompt carries the discipline sentence
|
||
EXACTLY ONCE — it IS the intro's final sentence, so it sits inside
|
||
``<documents>``, AFTER the intro's start-here framing and BEFORE
|
||
the first block (both the plain build and the folder-line build;
|
||
the sentence rides the intro line, which never moves)."""
|
||
doc = _doc(
|
||
"kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster", summary="K8S"
|
||
)
|
||
assert CITE_DISCIPLINE in SUGGEST_INTRO
|
||
assert SUGGEST_INTRO.endswith(CITE_DISCIPLINE) # it is the final sentence
|
||
for high in (build_high_prompt([doc]), build_high_prompt([doc], folder_lines=["X/: y.md"])):
|
||
assert high.count(CITE_DISCIPLINE) == 1
|
||
i_docs = high.index("<documents>")
|
||
i_sentence = high.index(CITE_DISCIPLINE)
|
||
i_block = high.index("<document ")
|
||
i_close = high.index("</documents>")
|
||
assert i_docs < i_sentence < i_close
|
||
assert i_sentence > high.index("start here if one seems right")
|
||
assert i_sentence < i_block # before the first block, never after a summary
|
||
# The retired phase-118 sentence is gone from the prompt (it is
|
||
# gone from the constant — pinned in test_prompt_lock too).
|
||
high = build_high_prompt([doc])
|
||
assert "Cite the document(s) you used, by path." not in high
|
||
# The sentence rides on present blocks ONLY: an empty ``<documents>``
|
||
# section (no intro) carries none of it.
|
||
assert CITE_DISCIPLINE not in build_high_prompt([])
|
||
|
||
|
||
def test_cite_discipline_absent_from_low_prompt() -> None:
|
||
"""LOCKED A5: the discipline sentence belongs to ``SUGGEST_INTRO``
|
||
(the HIGH path) — it never leaks into the LOW (deflected) prompt,
|
||
whose byte-identity is pinned separately (the LOW anchors in
|
||
``test_prompt_lock`` pass unchanged)."""
|
||
for prompt in (
|
||
build_deflect_prompt(["T1", "T2"]),
|
||
build_deflect_prompt(["T1"], notes=["be concise"], kb_overview=OVERVIEW),
|
||
build_deflect_prompt([]),
|
||
):
|
||
assert CITE_DISCIPLINE not in prompt
|
||
assert "Cite only the document(s) you read" not in prompt
|
||
assert "Cite the document(s) you used, by path." not in prompt
|
||
|
||
|
||
def test_tools_section_old_names_and_budget_copy_gone() -> None:
|
||
"""The phase-37/68 tool names and the phase-37 per-tool budget line
|
||
(phase 45: the round cap is the bound — the prompt does not
|
||
re-state it) are out of the copy."""
|
||
for old in ("list_documents", "read_document", "search_documents"):
|
||
assert old not in TOOLS_SECTION
|
||
assert "more than one" not in TOOLS_SECTION
|
||
assert "extra document" not in TOOLS_SECTION
|
||
|
||
|
||
def test_high_prompt_still_ends_with_tools_section() -> None:
|
||
"""Mock keying intact: the HIGH prompt still ends with the
|
||
``<tools>`` section after ``</documents>``, now in the phase-70
|
||
copy — new names in, old names out."""
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||
prompt = build_high_prompt([doc])
|
||
assert TOOLS_SECTION in prompt
|
||
assert prompt.index("</documents>") < prompt.index("<tools>")
|
||
assert prompt.rstrip().endswith("</tools>")
|
||
for tool in ("`ls`", "`grep`", "`read`"):
|
||
assert tool in prompt
|
||
for old in ("list_documents", "read_document", "search_documents"):
|
||
assert old not in prompt
|
||
|
||
|
||
# ---------- phase 118 (task 03): the NULL-summary preview fallback (A5) ----------
|
||
|
||
|
||
def _content_only_doc(content: str) -> Document:
|
||
return _doc("kubernetes.md", content, "Kubernetes Homelab Cluster")
|
||
|
||
|
||
def test_null_summary_falls_back_to_preview_plus_marker() -> None:
|
||
"""A doc whose ``summary`` is None (a fail-soft import miss) seeds the
|
||
first ``suggestion_preview_chars`` (default 400) content characters +
|
||
the shared ``[…truncated…]`` marker on its own line — never the full
|
||
content, never an LLM call."""
|
||
content = "A" * 400 + "B" * 600 # 1000 chars
|
||
prompt = build_high_prompt([_content_only_doc(content)])
|
||
body_start = prompt.index('date="2024-06-15">\n') + len('date="2024-06-15">\n')
|
||
body_end = prompt.index("\n</document>", body_start)
|
||
assert prompt[body_start:body_end] == content[:400] + "\n" + TRUNCATION_MARKER
|
||
assert "B" * 40 not in prompt # beyond the 400-char cut
|
||
assert prompt.count(TRUNCATION_MARKER) == 1 # only the block's marker
|
||
|
||
|
||
def test_whitespace_summary_uses_the_same_preview_fallback() -> None:
|
||
for blank in ("", " ", "\n \t "):
|
||
doc = _doc("kubernetes.md", "A" * 500 + "B" * 500, "T", summary=blank)
|
||
prompt = build_high_prompt([doc])
|
||
assert "A" * 400 in prompt
|
||
assert "B" * 40 not in prompt
|
||
assert TRUNCATION_MARKER in prompt
|
||
|
||
|
||
def test_short_content_preview_is_the_whole_content_unmarked() -> None:
|
||
"""Content at or under the cap rides whole — nothing was cut, so no
|
||
marker (the marker signals truncation, not the fallback)."""
|
||
for content in ("short body", "C" * 399, "D" * 400):
|
||
prompt = build_high_prompt([_content_only_doc(content)])
|
||
assert content + "\n</document>" in prompt
|
||
assert TRUNCATION_MARKER not in prompt
|
||
|
||
|
||
def test_summary_stripped_and_never_truncated_by_the_preview_cap() -> None:
|
||
"""The summary body is the stripped ``doc.summary`` — even when it is
|
||
longer than the preview cap (the cap bounds only the FALLBACK; a
|
||
stored summary is trusted context, LOCKED A5)."""
|
||
summary = "S" * 900
|
||
content = "CONTENT_SENTINEL " + "x" * 50
|
||
prompt = build_high_prompt([_doc("kubernetes.md", content, "T", summary=f" {summary}\n")])
|
||
assert summary in prompt
|
||
assert content not in prompt
|
||
assert TRUNCATION_MARKER not in prompt
|
||
|
||
|
||
def test_preview_cap_setting_is_honored_on_the_fallback_path(
|
||
monkeypatch: pytest.MonkeyPatch,
|
||
) -> None:
|
||
"""A non-default ``suggestion_preview_chars`` (``BOR_SUGGESTION_PREVIEW_CHARS``
|
||
— the env mapping is pinned in :mod:`tests.unit.test_config`) cuts the
|
||
preview at the setting's cap on the fallback path."""
|
||
from app.rag import prompts as prompts_mod
|
||
|
||
monkeypatch.setattr(
|
||
prompts_mod,
|
||
"get_settings",
|
||
lambda: Settings(_env_file=None, suggestion_preview_chars=10), # pyright: ignore[reportCallIssue]
|
||
)
|
||
content = "A" * 100 + "B" * 100
|
||
prompt = build_high_prompt([_content_only_doc(content)])
|
||
assert "A" * 10 in prompt
|
||
assert "A" * 11 not in prompt
|
||
assert TRUNCATION_MARKER in prompt
|
||
|
||
|
||
def test_preview_fallback_never_reads_settings_for_summarized_docs(
|
||
monkeypatch: pytest.MonkeyPatch,
|
||
) -> None:
|
||
"""House pattern (``build_steering_section`` / ``build_kb_section``):
|
||
the fallback cap is read from settings on the fallback path ONLY — a
|
||
prompt built from summary-bearing docs makes no settings call for
|
||
it (a dead ``get_settings`` must not break such a build)."""
|
||
from app.rag import prompts as prompts_mod
|
||
|
||
doc = _doc("kubernetes.md", "CONTENT_SENTINEL", "T", summary="the summary")
|
||
|
||
def _dead() -> None: # pragma: no cover - must never be called
|
||
raise AssertionError("get_settings() called for a summarized doc")
|
||
|
||
monkeypatch.setattr(prompts_mod, "get_settings", _dead)
|
||
prompt = build_high_prompt([doc])
|
||
assert "the summary\n</document>" in prompt
|
||
assert "CONTENT_SENTINEL" not in prompt
|
||
|
||
|
||
# ---------- phase 118: the LOW prompt stays byte-identical ----------
|
||
|
||
|
||
def test_low_prompt_byte_identical_to_pre_task() -> None:
|
||
"""LOCKED A8 surface: the deflection prompt is untouched by the
|
||
summary seeding — byte-identical build on the same inputs (the full
|
||
sha pin lives in :mod:`tests.unit.test_prompt_lock`)."""
|
||
for titles, notes, kb in (
|
||
(["T1", "T2"], None, None),
|
||
(["T1"], ["be concise"], None),
|
||
([], None, OVERVIEW),
|
||
(["T1", "T2"], ["be concise"], OVERVIEW),
|
||
):
|
||
prompt = build_deflect_prompt(titles, notes=notes, kb_overview=kb)
|
||
assert "DEFLECT_MODE" in prompt
|
||
assert "<documents>" not in prompt
|
||
assert SUGGEST_INTRO not in prompt
|
||
assert "<tools>" not in prompt
|
||
|
||
|
||
def test_relevance_placeholder_rejected_for_garbage() -> None:
|
||
with pytest.raises(ValueError, match="HIGH or LOW"):
|
||
_base("MEDIUM")
|
||
|
||
|
||
# ---------- <knowledge_base> section (phase 31) ----------
|
||
|
||
|
||
def test_kb_section_empty_when_no_overview() -> None:
|
||
assert build_kb_section("") == ""
|
||
assert build_kb_section(" \n\t ") == ""
|
||
|
||
|
||
def test_kb_section_format_intro_and_content() -> None:
|
||
assert build_kb_section(OVERVIEW) == (
|
||
f"<knowledge_base>\n{KB_INTRO}\n{OVERVIEW}\n</knowledge_base>"
|
||
)
|
||
|
||
|
||
def test_kb_intro_phase142_honest_copy() -> None:
|
||
"""Phase 142 (task 03): the intro is pinned verbatim — it says what
|
||
the outline actually is (a compact picture of the KB's categories
|
||
and topics that favors recent documents and may omit older or lesser
|
||
ones — a map of what exists, not a complete inventory) and gives the
|
||
one behavior that fixes the "the last N phases" question class: for
|
||
latest-state / complete-set questions, verify with ``ls`` (drill
|
||
into the folders) or ``grep`` BEFORE answering from the outline. The
|
||
"(generated at import time)" phrase is gone — it was false (the sync
|
||
regenerates the row on every KB change) and it taught the model to
|
||
treat a partial outline as the whole frozen KB."""
|
||
from app.rag.prompts import _KB_INTRO
|
||
|
||
# The app's constant, pinned verbatim (the trailing newline included):
|
||
assert _KB_INTRO == (
|
||
'A compact picture of the knowledge base\'s categories and topics — '
|
||
'it favors recent documents and may omit older or lesser ones: a map '
|
||
'of what exists, not a complete inventory. For questions about the '
|
||
'latest state or the complete set ("the last N …", "everything '
|
||
'about …", "all the …"), verify with `ls` (drill into the folders) '
|
||
'or `grep` before answering from the outline:\n'
|
||
)
|
||
# The local pin is the same text minus the trailing newline.
|
||
assert KB_INTRO + "\n" == _KB_INTRO
|
||
# The false claim is gone from the constant (and thus the prompt).
|
||
assert "(generated at import time)" not in KB_INTRO
|
||
assert "generated at import time" not in KB_INTRO
|
||
# The three contracts it must carry: the recency bias (task 02's
|
||
# newest-first input is what makes it true), the map-not-inventory
|
||
# framing, and the verify-first teaching with both tool names.
|
||
assert "favors recent documents" in KB_INTRO
|
||
assert "not a complete inventory" in KB_INTRO
|
||
assert "verify with `ls` (drill into the folders) or `grep`" in KB_INTRO
|
||
assert "before answering from the outline" in KB_INTRO
|
||
# It rides the rendered section exactly where the old intro did:
|
||
# between the open tag and the outline, colon-terminated.
|
||
section = build_kb_section(OVERVIEW)
|
||
assert section.startswith(f"<knowledge_base>\n{KB_INTRO}")
|
||
assert section.index(KB_INTRO) < section.index(OVERVIEW)
|
||
# And it reaches BOTH modes' prompts (the section is mode-agnostic).
|
||
doc = _doc("kubernetes.md", "CONTENT", "T", summary="S.")
|
||
for prompt in (
|
||
build_high_prompt([doc], kb_overview=OVERVIEW),
|
||
build_deflect_prompt(["T1"], kb_overview=OVERVIEW),
|
||
):
|
||
assert KB_INTRO in prompt
|
||
assert "(generated at import time)" not in prompt
|
||
|
||
|
||
def test_kb_section_trims_overview_edges() -> None:
|
||
assert build_kb_section(f" {OVERVIEW} \n") == build_kb_section(OVERVIEW)
|
||
|
||
|
||
def test_kb_section_fits_budget_exactly_no_marker() -> None:
|
||
exact = f"<knowledge_base>\n{KB_INTRO}\n{OVERVIEW}\n</knowledge_base>"
|
||
section = build_kb_section(OVERVIEW, max_chars=len(exact))
|
||
assert TRUNCATION_MARKER not in section
|
||
assert section == exact
|
||
|
||
|
||
def test_kb_section_over_budget_capped_with_marker() -> None:
|
||
text = "- " + "x" * 500
|
||
# The section frame alone is 422 chars (the tag + the phase-142
|
||
# honest intro + the marker + the close tag), so the cap must clear
|
||
# it for any outline prefix to fit (pathological budgets are tested
|
||
# below).
|
||
cap = 600
|
||
section = build_kb_section(text, max_chars=cap)
|
||
assert len(section) <= cap # the budget is never exceeded
|
||
assert TRUNCATION_MARKER in section
|
||
assert section.startswith(f"<knowledge_base>\n{KB_INTRO}\n-")
|
||
assert section.endswith(f"{TRUNCATION_MARKER}\n</knowledge_base>")
|
||
# The body is the kept prefix + the marker on its own line, and the
|
||
# kept part must be a true prefix of the outline (longest-fitting).
|
||
body = section.removeprefix(f"<knowledge_base>\n{KB_INTRO}\n").removesuffix(
|
||
"\n</knowledge_base>"
|
||
)
|
||
kept, marker = body.rsplit("\n", 1)
|
||
assert marker == TRUNCATION_MARKER
|
||
assert kept.startswith("- ")
|
||
assert text.startswith(kept), "the kept part must be a prefix of the outline"
|
||
# And it is the longest such prefix: one more char would not fit.
|
||
assert len(section) > cap - 2, "the cut must sit as close to the cap as possible"
|
||
|
||
|
||
def test_kb_section_default_budget_from_settings(monkeypatch: pytest.MonkeyPatch) -> None:
|
||
from app.rag import prompts as prompts_mod
|
||
|
||
monkeypatch.setattr(
|
||
prompts_mod, "get_settings", lambda: Settings(_env_file=None) # pyright: ignore[reportCallIssue]
|
||
)
|
||
text = "y" * 9_000 # > the 4 000-char default
|
||
section = build_kb_section(text)
|
||
assert TRUNCATION_MARKER in section
|
||
assert len(section) <= 4_000
|
||
|
||
|
||
def test_kb_section_nonpositive_budget_is_empty() -> None:
|
||
assert build_kb_section(OVERVIEW, max_chars=0) == ""
|
||
assert build_kb_section(OVERVIEW, max_chars=-10) == ""
|
||
|
||
|
||
def test_kb_section_tiny_budget_never_exceeds_cap() -> None:
|
||
# Pathological budget (steering precedent, phase 15): the section must
|
||
# never exceed the cap — bare marker when it fits, no section at all
|
||
# when even that doesn't.
|
||
assert build_kb_section("a" * 500, max_chars=10) == "" # marker (13) > 10
|
||
fits_marker = build_kb_section("a" * 500, max_chars=len(TRUNCATION_MARKER))
|
||
assert fits_marker == TRUNCATION_MARKER
|
||
|
||
|
||
# ---------- <knowledge_base> placement (both modes) ----------
|
||
|
||
|
||
def test_no_overview_prompt_is_byte_identical_to_pre_phase() -> None:
|
||
"""Phase 31 contract: with no KB overview (None, empty, or blank)
|
||
every prompt is exactly what it was before the ``<knowledge_base>``
|
||
section existed — with or without steering notes. (Phase 37: the HIGH
|
||
prompt additionally carries the ``<tools>`` section after the mode
|
||
body; phase 118: the ``<documents>`` section leads with the
|
||
``SUGGEST_INTRO`` line — the fixtures account for both; the LOW
|
||
prompt is untouched.)"""
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||
block = (
|
||
'<document source="Homelab" path="kubernetes.md" '
|
||
'title="Kubernetes Homelab Cluster" date="2024-06-15">\n'
|
||
"Talos Linux on three nodes.\n"
|
||
"</document>"
|
||
)
|
||
docs_block = (
|
||
"\n<documents>\n"
|
||
+ SUGGEST_INTRO
|
||
+ "\n\n"
|
||
+ block
|
||
+ "\n</documents>"
|
||
+ "\n"
|
||
+ TOOLS_SECTION
|
||
)
|
||
high_plain = _base("HIGH") + docs_block
|
||
high_steered = _base("HIGH") + "\n" + build_steering_section(["be concise"]) + docs_block
|
||
# Phase 71: the owner-permitted plain-text line is part of the
|
||
# DEFLECT_MODE body in every LOW build (with or without steering).
|
||
low_plain = (
|
||
_base("LOW")
|
||
+ "\nDEFLECT_MODE: retrieval was weak — the titles below are the closest "
|
||
"your notes come to the question. They are titles only; do not pretend "
|
||
"they answer it. Use them to propose 2-3 alternative questions.\n"
|
||
"Reply in plain text only — you have no tools in this mode.\n"
|
||
+ "- T1\n- T2"
|
||
)
|
||
low_steered = (
|
||
_base("LOW")
|
||
+ "\n"
|
||
+ build_steering_section(["be concise"])
|
||
+ "\nDEFLECT_MODE: retrieval was weak — the titles below are the closest "
|
||
"your notes come to the question. They are titles only; do not pretend "
|
||
"they answer it. Use them to propose 2-3 alternative questions.\n"
|
||
"Reply in plain text only — you have no tools in this mode.\n"
|
||
+ "- T1\n- T2"
|
||
)
|
||
for kb in (None, "", " \n\t "):
|
||
assert build_high_prompt([doc], kb_overview=kb) == high_plain
|
||
assert build_high_prompt([doc], notes=["be concise"], kb_overview=kb) == high_steered
|
||
assert build_deflect_prompt(["T1", "T2"], kb_overview=kb) == low_plain
|
||
assert build_deflect_prompt(
|
||
["T1", "T2"], notes=["be concise"], kb_overview=kb
|
||
) == low_steered
|
||
assert "<knowledge_base>" not in build_high_prompt(
|
||
[doc], notes=["be concise"], kb_overview=kb
|
||
)
|
||
assert "<knowledge_base>" not in build_deflect_prompt(
|
||
["T1"], notes=["be concise"], kb_overview=kb
|
||
)
|
||
|
||
|
||
def test_high_prompt_kb_section_ordered_between_relevance_and_tuning() -> None:
|
||
doc = _doc("kubernetes.md", "TALOS_DOC_CONTENT", "Kubernetes Homelab Cluster")
|
||
prompt = build_high_prompt([doc], notes=["be concise"], kb_overview=OVERVIEW)
|
||
i_rel = prompt.index("<relevance>HIGH</relevance>")
|
||
i_kb_open = prompt.index("<knowledge_base>")
|
||
i_kb_close = prompt.index("</knowledge_base>")
|
||
i_tuning = prompt.index("<tuning>")
|
||
i_docs = prompt.index("<documents>")
|
||
assert i_rel < i_kb_open < i_kb_close < i_tuning < i_docs
|
||
assert KB_INTRO in prompt
|
||
assert OVERVIEW in prompt # outline intact within the section
|
||
assert "1. be concise" in prompt # steering still there
|
||
assert "TALOS_DOC_CONTENT" in prompt # documents still full
|
||
|
||
|
||
def test_high_prompt_kb_section_without_steering() -> None:
|
||
doc = _doc("kubernetes.md", "TALOS_DOC_CONTENT", "Kubernetes Homelab Cluster")
|
||
prompt = build_high_prompt([doc], kb_overview=OVERVIEW)
|
||
i_rel = prompt.index("<relevance>HIGH</relevance>")
|
||
i_kb_close = prompt.index("</knowledge_base>")
|
||
i_docs = prompt.index("<documents>")
|
||
assert i_rel < i_kb_close < i_docs
|
||
assert "<tuning>" not in prompt # no notes → no steering section
|
||
assert build_kb_section(OVERVIEW) in prompt
|
||
|
||
|
||
def test_deflect_prompt_kb_section_ordered_between_relevance_and_tuning() -> None:
|
||
prompt = build_deflect_prompt(
|
||
["Title A", "Title B"], notes=["be concise"], kb_overview=OVERVIEW
|
||
)
|
||
i_rel = prompt.index("<relevance>LOW</relevance>")
|
||
i_kb_open = prompt.index("<knowledge_base>")
|
||
i_kb_close = prompt.index("</knowledge_base>")
|
||
i_tuning = prompt.index("<tuning>")
|
||
i_mode = prompt.index("DEFLECT_MODE")
|
||
assert i_rel < i_kb_open < i_kb_close < i_tuning < i_mode
|
||
assert KB_INTRO in prompt
|
||
assert OVERVIEW in prompt
|
||
assert "1. be concise" in prompt
|
||
assert "- Title A" in prompt # weak-hit titles still carried
|
||
|
||
|
||
def test_prompt_kb_section_over_settings_budget_capped_with_marker(
|
||
monkeypatch: pytest.MonkeyPatch,
|
||
) -> None:
|
||
"""Prompt-level budget: an overview longer than
|
||
``kb_overview_max_chars`` is capped with the shared marker — in both
|
||
modes."""
|
||
from app.rag import prompts as prompts_mod
|
||
|
||
monkeypatch.setattr(
|
||
prompts_mod,
|
||
"get_settings",
|
||
# 600 > the 422-char section frame, so a prefix + marker can fit.
|
||
lambda: Settings(_env_file=None, kb_overview_max_chars=600), # pyright: ignore[reportCallIssue]
|
||
)
|
||
text = "- " + "z" * 500
|
||
doc = _doc("kubernetes.md", "TALOS_DOC_CONTENT", "Kubernetes Homelab Cluster")
|
||
for prompt in (
|
||
build_high_prompt([doc], kb_overview=text),
|
||
build_deflect_prompt(["Title A"], kb_overview=text),
|
||
):
|
||
assert TRUNCATION_MARKER in prompt
|
||
assert prompt.index("<knowledge_base>") < prompt.index(TRUNCATION_MARKER)
|
||
# The capped section (open tag through close tag) fits the budget.
|
||
section = prompt[prompt.index("<knowledge_base>") :]
|
||
close = section.index("</knowledge_base>")
|
||
section = section[: close + len("</knowledge_base>")]
|
||
assert len(section) <= 600
|
||
|
||
|
||
# ---------- phase 71: the deflection plain-text line (prevention) ----------
|
||
|
||
#: The owner-permitted (2026-09-03) line appended to the ``DEFLECT_MODE``
|
||
#: body — the LOW prompt's only phase-71 change. The E2E mock keys on
|
||
#: the ``DEFLECT_MODE`` marker's *presence*, not the wording, so the
|
||
#: marker-keying contract is unchanged by the appended line.
|
||
PLAIN_TEXT_ONLY_LINE = "Reply in plain text only — you have no tools in this mode."
|
||
|
||
|
||
def _pre_phase71_low_body() -> str:
|
||
"""The ``DEFLECT_MODE`` body exactly as it was before phase 71."""
|
||
return (
|
||
"DEFLECT_MODE: retrieval was weak — the titles below are the closest "
|
||
"your notes come to the question. They are titles only; do not pretend "
|
||
"they answer it. Use them to propose 2-3 alternative questions.\n"
|
||
)
|
||
|
||
|
||
def test_low_prompt_is_pre_phase_plus_exactly_the_plain_text_line() -> None:
|
||
"""Diff pin: the LOW prompt = pre-phase text + exactly the one new
|
||
line, appended to the ``DEFLECT_MODE`` body; the weak-hit title list
|
||
follows exactly as before (and the line occurs exactly once)."""
|
||
prompt = build_deflect_prompt(["T1", "T2"])
|
||
assert prompt == (
|
||
_base("LOW")
|
||
+ "\n"
|
||
+ _pre_phase71_low_body()
|
||
+ PLAIN_TEXT_ONLY_LINE
|
||
+ "\n"
|
||
+ "- T1\n- T2"
|
||
)
|
||
assert prompt.count(PLAIN_TEXT_ONLY_LINE) == 1
|
||
assert prompt.endswith("- T1\n- T2") # the title list is untouched
|
||
|
||
|
||
def test_low_prompt_carries_the_line_and_keeps_the_mock_marker() -> None:
|
||
"""The new line is present in the LOW prompt (inside the
|
||
``DEFLECT_MODE`` body, after the marker) and the ``DEFLECT_MODE``
|
||
marker the E2E mock keys on stays put."""
|
||
for titles, tail in ((["T1"], "- T1"), ([], "(nothing close at all)")):
|
||
prompt = build_deflect_prompt(titles)
|
||
assert "DEFLECT_MODE" in prompt
|
||
assert PLAIN_TEXT_ONLY_LINE in prompt
|
||
assert prompt.index("DEFLECT_MODE") < prompt.index(PLAIN_TEXT_ONLY_LINE)
|
||
# The title list (or the no-titles fallback) follows the line
|
||
# exactly as before.
|
||
assert prompt.endswith(tail)
|
||
|
||
|
||
def test_plain_text_line_never_leaks_into_high_prompt() -> None:
|
||
"""The line is the LOW prompt's: every HIGH build (with/without
|
||
steering/overview) is unchanged and carries none of it."""
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||
for notes, kb in (
|
||
(None, None),
|
||
(["be concise"], None),
|
||
(None, OVERVIEW),
|
||
(["be concise"], OVERVIEW),
|
||
):
|
||
high = build_high_prompt([doc], notes=notes, kb_overview=kb)
|
||
assert PLAIN_TEXT_ONLY_LINE not in high
|
||
assert "you have no tools" not in high
|
||
|
||
|
||
# ---------- phase 124 (task 03, LOCKED A5): the show_image prompt clause ----------
|
||
|
||
|
||
def test_show_image_clause_states_the_four_gate_elements() -> None:
|
||
"""LOCKED A5: the clause states ALL FOUR gate elements, pinned on
|
||
stable phrases, and references the tool by its backticked name (the
|
||
``TOOLS_SECTION`` voice). (1) the image-doc nature — its text is a
|
||
description GENERATED from the image; (2) the "contains the answer"
|
||
bar — never a merely related image, never padding; (3) at most one
|
||
image per answer; (4) the text answer is still provided in full."""
|
||
assert "`show_image`" in SHOW_IMAGE_CLAUSE # backticked, like the other tools
|
||
assert "description GENERATED from the image" in SHOW_IMAGE_CLAUSE # (1)
|
||
assert "what the user actually wants to see" in SHOW_IMAGE_CLAUSE # (2)
|
||
assert "never for a merely related image" in SHOW_IMAGE_CLAUSE # (2)
|
||
assert "never to pad an answer" in SHOW_IMAGE_CLAUSE # (2)
|
||
assert "at most one image per answer" in SHOW_IMAGE_CLAUSE # (3)
|
||
assert "answer fully in text" in SHOW_IMAGE_CLAUSE # (4)
|
||
# It is a clause, not a section: no ``<tools>`` markup of its own
|
||
# (the ``<documents>`` mention is plain text, as in the section copy).
|
||
assert "<tools>" not in SHOW_IMAGE_CLAUSE
|
||
assert "</tools>" not in SHOW_IMAGE_CLAUSE
|
||
|
||
|
||
def test_high_prompt_default_stays_byte_identical_to_pre_task() -> None:
|
||
"""Completion pin: ``build_high_prompt(docs)`` (no new arg) and the
|
||
explicit ``show_image=False`` are byte-identical to each other and
|
||
end with the untouched ``TOOLS_SECTION`` — the pre-task pins above
|
||
(and the sha pins in :mod:`tests.unit.test_prompt_lock`) pass
|
||
UNMODIFIED, with or without documents."""
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||
plain = build_high_prompt([doc])
|
||
assert build_high_prompt([doc], show_image=False) == plain
|
||
assert plain.endswith(TOOLS_SECTION)
|
||
assert SHOW_IMAGE_CLAUSE not in plain
|
||
empty = build_high_prompt([])
|
||
assert build_high_prompt([], show_image=False) == empty
|
||
assert empty.endswith(TOOLS_SECTION)
|
||
assert SHOW_IMAGE_CLAUSE not in empty
|
||
|
||
|
||
def test_high_prompt_show_image_clause_lands_inside_tools_after_the_copy() -> None:
|
||
"""Phase 124 (task 03): ``show_image=True`` builds EXACTLY the
|
||
toggle-off prompt with its final ``</tools>`` replaced by the clause
|
||
(on its own line) + the closing tag — the prefix up to the clause
|
||
is byte-identical to the toggle-off output (the three-tool copy
|
||
untouched), the clause lands AFTER the existing copy and BEFORE the
|
||
closing tag, and appears exactly once."""
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||
off = build_high_prompt([doc])
|
||
on = build_high_prompt([doc], show_image=True)
|
||
# The full structural equation (the clause is the ONLY difference).
|
||
assert on == off[: -len("</tools>")] + SHOW_IMAGE_CLAUSE + "\n</tools>"
|
||
clause_start = on.index(SHOW_IMAGE_CLAUSE)
|
||
# The prefix up to the clause is byte-identical to the toggle-off
|
||
# build minus its closing tag.
|
||
assert on[:clause_start] == off[: -len("</tools>")]
|
||
# From the <tools> opener to the clause is TOOLS_SECTION's own text
|
||
# minus its final </tools> — the constant itself is unedited (sha-
|
||
# pinned in test_prompt_lock) and the three-tool copy is intact.
|
||
assert on[on.index("<tools>") : clause_start] == TOOLS_SECTION[: -len("</tools>")]
|
||
assert "Answer as soon as you have what you need." in on[:clause_start]
|
||
# The clause (then the closing tag) is the tail of the prompt, after
|
||
# the existing copy and before the closing tag — exactly once.
|
||
assert on.endswith(SHOW_IMAGE_CLAUSE + "\n</tools>")
|
||
assert on.index("Answer as soon as you have what you need.") < clause_start
|
||
assert clause_start < on.index("</tools>")
|
||
assert on.count(SHOW_IMAGE_CLAUSE) == 1
|
||
assert on.count("<tools>") == 1 and on.count("</tools>") == 1
|
||
|
||
|
||
def test_show_image_clause_rides_only_the_high_prompt() -> None:
|
||
"""The clause appears ONLY in the HIGH prompt: the deflection prompt
|
||
takes NO ``show_image`` flag at all and its output never carries the
|
||
clause (byte-identical builds with/without notes/overview — a
|
||
deflected turn has no tools and never shows an image, the phase's
|
||
locked design)."""
|
||
assert "show_image" not in inspect.signature(build_deflect_prompt).parameters
|
||
for prompt in (
|
||
build_deflect_prompt(["T1", "T2"]),
|
||
build_deflect_prompt(["T1"], notes=["be concise"], kb_overview=OVERVIEW),
|
||
build_deflect_prompt([]),
|
||
):
|
||
assert SHOW_IMAGE_CLAUSE not in prompt
|
||
assert "`show_image`" not in prompt
|
||
assert "at most one image per answer" not in prompt
|
||
|
||
|
||
def test_high_prompt_empty_documents_show_image_clause_still_in_tools() -> None:
|
||
"""The "(no documents matched…)" body + ``show_image=True``: the
|
||
clause still lands inside ``<tools>`` — the grounded turn may still
|
||
``ls``/``grep`` its way to an image doc and show it."""
|
||
off = build_high_prompt([])
|
||
on = build_high_prompt([], show_image=True)
|
||
assert on == off[: -len("</tools>")] + SHOW_IMAGE_CLAUSE + "\n</tools>"
|
||
assert "(no documents matched — do not invent specifics)" in on
|
||
assert on.index("<tools>") < on.index(SHOW_IMAGE_CLAUSE) < on.index("</tools>")
|
||
assert on.endswith(SHOW_IMAGE_CLAUSE + "\n</tools>")
|
||
|
||
|
||
def test_show_image_clause_is_last_for_every_high_shape() -> None:
|
||
"""Steering + KB overview + folder lines + documents all present:
|
||
the section order is UNCHANGED (``<relevance>`` →
|
||
``<knowledge_base>`` → ``<tuning>`` → ``<documents>`` → ``<tools>``)
|
||
and the clause is still the last thing the model reads, inside
|
||
``<tools>`` after the three-tool copy."""
|
||
doc = _doc("deploy/k8s.md", "CONTENT_SENTINEL", "K8s Deploy", summary="K8s deploy notes.")
|
||
lines = ["Homelab/deploy/: k8s.md (1 doc)"]
|
||
off = build_high_prompt([doc], notes=["be concise"], kb_overview=OVERVIEW, folder_lines=lines)
|
||
on = build_high_prompt(
|
||
[doc],
|
||
notes=["be concise"],
|
||
kb_overview=OVERVIEW,
|
||
folder_lines=lines,
|
||
show_image=True,
|
||
)
|
||
assert on == off[: -len("</tools>")] + SHOW_IMAGE_CLAUSE + "\n</tools>"
|
||
clause_start = on.index(SHOW_IMAGE_CLAUSE)
|
||
assert on[on.index("<tools>") : clause_start] == TOOLS_SECTION[: -len("</tools>")]
|
||
for section in ("<relevance>", "<knowledge_base>", "<tuning>", "<documents>"):
|
||
assert on.index(section) < on.index("<tools>")
|
||
assert on.index("K8s deploy notes.") < on.index("<tools>") # the summary still seeds
|
||
assert SHOW_IMAGE_CLAUSE not in off # the toggle-off twin carries none of it
|
||
|
||
|
||
# ---------- phase 182 (task 04, LOCKED A1–A3/A6): the show_text clause ----------
|
||
#
|
||
# The clause is the fix for the retype bug: nothing in the pre-phase prompt
|
||
# ever said the user copies from what is DISPLAYED, so the model retyped
|
||
# files from context and every retype differed from the stored bytes by a
|
||
# character. Pinned three ways: the locked sentence order (the behaviour
|
||
# the E2E and the persona depend on), the byte-identity of every build the
|
||
# flag does NOT select, and the composition of the two display clauses
|
||
# through the one "section minus its final </tools>" mechanism.
|
||
|
||
#: The clause's sentence anchors, in the locked order (task 04's work
|
||
#: list): display contract → when to call → do-NOT-retype → the ```bor```
|
||
#: placement syntax → the numbers belong to the server → the failure rule
|
||
#: → the image pointer.
|
||
SHOW_TEXT_ANCHORS: tuple[str, ...] = (
|
||
"displays part of a document to the user EXACTLY as stored",
|
||
"the user copies from what is displayed, not from your prose",
|
||
"Call it whenever the user wants to SEE, COPY or RUN a file",
|
||
"a quadlet, a config, a script, a unit file",
|
||
"do NOT repeat the displayed lines in your answer",
|
||
"may differ from the file by a character, and the user cannot tell",
|
||
"write a locator line on its own: three backticks followed by `bor`",
|
||
"the combined `source/path`, and optionally the line range (` 12-40`)",
|
||
"Leave its body EMPTY — the app replaces it with the exact stored text",
|
||
"Never type the file's content inside a `bor` block",
|
||
"Never guess line numbers — use the numbers `grep` and `read` reported",
|
||
"OMIT the range and display the whole document",
|
||
"the numbers you display are the server's, never yours",
|
||
"If `show_text` refuses, say what you could not display",
|
||
"do not reconstruct it from memory",
|
||
"For an image use `show_image` instead",
|
||
)
|
||
|
||
|
||
def test_show_text_clause_carries_every_locked_rule_in_order() -> None:
|
||
"""Every sentence of the contract is present, in the locked order, and
|
||
the clause is a clause — one paragraph, no ``<tools>`` markup of its
|
||
own (the ``SHOW_IMAGE_CLAUSE`` precedent)."""
|
||
positions = [SHOW_TEXT_CLAUSE.index(anchor) for anchor in SHOW_TEXT_ANCHORS]
|
||
assert positions == sorted(positions), SHOW_TEXT_ANCHORS
|
||
assert "\n" not in SHOW_TEXT_CLAUSE
|
||
assert SHOW_TEXT_CLAUSE.startswith("`show_text`")
|
||
assert SHOW_TEXT_CLAUSE.endswith("instead.")
|
||
assert "<tools>" not in SHOW_TEXT_CLAUSE and "</tools>" not in SHOW_TEXT_CLAUSE
|
||
# The fence language is named as a token, never as a stray substring.
|
||
assert re.search(r"(?<![A-Za-z0-9_])bor(?![A-Za-z0-9_])", SHOW_TEXT_CLAUSE)
|
||
|
||
|
||
def test_high_prompt_default_stays_byte_identical_without_show_text() -> None:
|
||
"""Completion pin: ``show_text`` defaults to ``False`` — the default
|
||
build and the explicit ``show_text=False`` are byte-identical to each
|
||
other and end with the untouched ``TOOLS_SECTION``, with no trace of
|
||
the tool or the fence language (the sha pins in
|
||
:mod:`tests.unit.test_prompt_lock` pass UNMODIFIED)."""
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||
plain = build_high_prompt([doc])
|
||
assert build_high_prompt([doc], show_text=False) == plain
|
||
assert plain.endswith(TOOLS_SECTION)
|
||
assert SHOW_TEXT_CLAUSE not in plain
|
||
assert "show_text" not in plain
|
||
assert not re.search(r"(?<![A-Za-z0-9_])bor(?![A-Za-z0-9_])", plain)
|
||
empty = build_high_prompt([])
|
||
assert build_high_prompt([], show_text=False) == empty
|
||
assert empty.endswith(TOOLS_SECTION)
|
||
assert SHOW_TEXT_CLAUSE not in empty
|
||
|
||
|
||
def test_high_prompt_show_text_clause_lands_inside_tools_after_the_copy() -> None:
|
||
"""``show_text=True`` builds EXACTLY the toggle-off prompt with its
|
||
final ``</tools>`` replaced by the clause (on its own line, after the
|
||
existing three-tool copy) + the closing tag — the three-tool copy is
|
||
byte-identical, ``TOOLS_SECTION`` is unedited, and the clause is the
|
||
last thing the model reads."""
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||
off = build_high_prompt([doc])
|
||
on = build_high_prompt([doc], show_text=True)
|
||
assert on == off[: -len("</tools>")] + SHOW_TEXT_CLAUSE + "\n</tools>"
|
||
clause_start = on.index(SHOW_TEXT_CLAUSE)
|
||
assert on[:clause_start] == off[: -len("</tools>")]
|
||
assert on[on.index("<tools>") : clause_start] == TOOLS_SECTION[: -len("</tools>")]
|
||
assert "Answer as soon as you have what you need." in on[:clause_start]
|
||
assert on.endswith(SHOW_TEXT_CLAUSE + "\n</tools>")
|
||
assert on.index("Answer as soon as you have what you need.") < clause_start
|
||
assert clause_start < on.index("</tools>")
|
||
assert on.count(SHOW_TEXT_CLAUSE) == 1
|
||
assert on.count("<tools>") == 1 and on.count("</tools>") == 1
|
||
|
||
|
||
def test_high_prompt_display_clauses_compose_independently() -> None:
|
||
"""The two display flags are independent (LOCKED A1): ``show_image``
|
||
only reproduces the phase-124 bytes exactly, ``show_text`` only is the
|
||
three-tool copy plus the text clause, both puts the image clause
|
||
FIRST and the text clause last, and neither is byte-identical to
|
||
pre-phase — no combination leaks the other's clause."""
|
||
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes Homelab Cluster")
|
||
neither = build_high_prompt([doc])
|
||
head = neither[: -len("</tools>")]
|
||
image = build_high_prompt([doc], show_image=True)
|
||
text = build_high_prompt([doc], show_text=True)
|
||
both = build_high_prompt([doc], show_image=True, show_text=True)
|
||
assert neither.endswith(TOOLS_SECTION)
|
||
assert image == head + SHOW_IMAGE_CLAUSE + "\n</tools>"
|
||
assert text == head + SHOW_TEXT_CLAUSE + "\n</tools>"
|
||
assert both == head + SHOW_IMAGE_CLAUSE + "\n\n" + SHOW_TEXT_CLAUSE + "\n</tools>"
|
||
assert SHOW_TEXT_CLAUSE not in image and SHOW_IMAGE_CLAUSE not in text
|
||
assert both.index(SHOW_IMAGE_CLAUSE) < both.index(SHOW_TEXT_CLAUSE)
|
||
for on in (image, text, both):
|
||
assert on[: on.index("<tools>")] == neither[: on.index("<tools>")]
|
||
assert on.count("<tools>") == 1 and on.count("</tools>") == 1
|
||
|
||
|
||
def test_high_prompt_empty_documents_show_text_clause_still_in_tools() -> None:
|
||
"""The "(no documents matched…)" body + ``show_text=True``: the clause
|
||
still lands inside ``<tools>`` — the grounded turn may still ``ls`` /
|
||
``grep`` its way to a file and display it."""
|
||
off = build_high_prompt([])
|
||
on = build_high_prompt([], show_text=True)
|
||
assert on == off[: -len("</tools>")] + SHOW_TEXT_CLAUSE + "\n</tools>"
|
||
assert "(no documents matched — do not invent specifics)" in on
|
||
assert on.index("<tools>") < on.index(SHOW_TEXT_CLAUSE) < on.index("</tools>")
|
||
assert on.endswith(SHOW_TEXT_CLAUSE + "\n</tools>")
|
||
|
||
|
||
def test_show_text_clause_is_last_for_every_high_shape() -> None:
|
||
"""Steering + KB overview + folder lines + documents all present: the
|
||
section order is UNCHANGED (``<relevance>`` → ``<knowledge_base>`` →
|
||
``<tuning>`` → ``<documents>`` → ``<tools>``) and the text clause is
|
||
still the last thing inside ``<tools>`` before the closing tag."""
|
||
doc = _doc("deploy/k8s.md", "CONTENT_SENTINEL", "K8s Deploy", summary="K8s deploy notes.")
|
||
lines = ["Homelab/deploy/: k8s.md (1 doc)"]
|
||
off = build_high_prompt([doc], notes=["be concise"], kb_overview=OVERVIEW, folder_lines=lines)
|
||
on = build_high_prompt(
|
||
[doc],
|
||
notes=["be concise"],
|
||
kb_overview=OVERVIEW,
|
||
folder_lines=lines,
|
||
show_text=True,
|
||
)
|
||
assert on == off[: -len("</tools>")] + SHOW_TEXT_CLAUSE + "\n</tools>"
|
||
clause_start = on.index(SHOW_TEXT_CLAUSE)
|
||
assert on[on.index("<tools>") : clause_start] == TOOLS_SECTION[: -len("</tools>")]
|
||
for section in ("<relevance>", "<knowledge_base>", "<tuning>", "<documents>"):
|
||
assert on.index(section) < on.index("<tools>")
|
||
assert on.index("K8s deploy notes.") < on.index("<tools>")
|
||
assert on.endswith(SHOW_TEXT_CLAUSE + "\n</tools>")
|
||
assert SHOW_TEXT_CLAUSE not in off
|
||
|
||
|
||
def test_show_text_clause_rides_only_the_high_prompt() -> None:
|
||
"""Neither the tool name nor the fence language ever reaches the
|
||
deflection prompt (phase 182's D9: a deflected turn has no tools, and
|
||
resolving a locator there would leak KB text through a turn that just
|
||
disclaimed it) — and ``build_deflect_prompt`` takes no such flag."""
|
||
assert "show_text" not in inspect.signature(build_deflect_prompt).parameters
|
||
for prompt in (
|
||
build_deflect_prompt(["T1", "T2"]),
|
||
build_deflect_prompt(["T1"], notes=["be concise"], kb_overview=OVERVIEW),
|
||
build_deflect_prompt([]),
|
||
):
|
||
assert SHOW_TEXT_CLAUSE not in prompt
|
||
assert "show_text" not in prompt
|
||
assert not re.search(r"(?<![A-Za-z0-9_])bor(?![A-Za-z0-9_])", prompt)
|
||
|
||
|
||
# ---------- phase 125 (task 05, LOCKED A4): the image context line ----------
|
||
# An enriched image doc (tasks 03/04: non-NULL ``image_context`` in the
|
||
# "Referenced in …" block shape) leads its ``<document>`` suggestion
|
||
# block with ONE short context line — the referring markdown's identity
|
||
# + the reference's alt text — before the description, so the LLM knows
|
||
# WHY the image exists, not only what it shows. Every other doc (every
|
||
# text doc, every non-enriched image doc) is byte-identical to the
|
||
# pre-phase body.
|
||
|
||
#: The exact stored block shape (tasks 03/04): one "Referenced in"
|
||
#: block per referring markdown file, blank-line joined; the
|
||
#: ``Context:`` body may span many lines (the surrounding paragraphs).
|
||
CTX_SINGLE = (
|
||
"Referenced in Homelab/homelab/kubernetes.md:\n"
|
||
'- Alt text: "Cluster layout"\n'
|
||
" Context: The diagram shows the three Talos nodes and Cilium.\n"
|
||
"It is referenced again from the runbook."
|
||
)
|
||
CTX_TWO_BLOCKS = CTX_SINGLE + "\n\nReferenced in Homelab/ops/runbook.md:\n- Alt text: \"\""
|
||
CTX_ALT_EMPTY = "Referenced in Homelab/ops/runbook.md:\n- Alt text: \"\""
|
||
|
||
#: The task-04 pending record — stored on MARKDOWN (text) docs, with no
|
||
#: ``Referenced in`` header at all.
|
||
MD_PENDING_RECORD = (
|
||
'IMAGE_CONTEXT:\n- [image.png] (line 42): "Alt text here" — 2 paragraphs of context'
|
||
)
|
||
|
||
|
||
def _image_doc(
|
||
image_context: str | None = None, summary: str | None = "A diagram of the cluster."
|
||
) -> Document:
|
||
return _doc(
|
||
"cluster-diagram.png",
|
||
"A diagram of the cluster.",
|
||
"cluster-diagram",
|
||
summary=summary,
|
||
is_image=True,
|
||
image_context=image_context,
|
||
)
|
||
|
||
|
||
# ---------- _extract_context_line: the parser contract ----------
|
||
|
||
|
||
def test_extract_context_line_single_block_with_alt() -> None:
|
||
"""One block with an alt → the short one-line locator: the stored
|
||
identity (``<source>/<path>`` exactly as the header carries it) +
|
||
the reference's alt between quotes."""
|
||
assert _extract_context_line(CTX_SINGLE) == (
|
||
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"'
|
||
)
|
||
|
||
|
||
def test_extract_context_line_context_body_never_renders() -> None:
|
||
"""The ``Context:`` paragraphs (LOCKED A3: they already reached the
|
||
vision model at enrichment time) stay out of the line — the line is
|
||
the locator + purpose, short enough not to bloat the prompt."""
|
||
line = _extract_context_line(CTX_SINGLE)
|
||
assert line is not None
|
||
assert "The diagram shows" not in line
|
||
assert "referenced again" not in line
|
||
assert "Context:" not in line
|
||
assert line.count("\n") == 0 # genuinely ONE line
|
||
|
||
|
||
def test_extract_context_line_empty_alt_renders_the_bare_identity() -> None:
|
||
"""A reference without an alt (````) → the clause is the
|
||
bare identity — nothing to quote after the colon."""
|
||
assert _extract_context_line(CTX_ALT_EMPTY) == "Referenced in Homelab/ops/runbook.md"
|
||
|
||
|
||
def test_extract_context_line_two_blocks_join_with_semicolon() -> None:
|
||
"""One clause per referring file after a single prefix — the second
|
||
block's empty alt renders bare."""
|
||
assert _extract_context_line(CTX_TWO_BLOCKS) == (
|
||
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"; '
|
||
"Homelab/ops/runbook.md"
|
||
)
|
||
|
||
|
||
def test_extract_context_line_uses_the_first_reference_only() -> None:
|
||
"""A block with two references (two ``- Alt text:`` bullets) → the
|
||
FIRST reference's alt is the purpose; the second stays out."""
|
||
text = (
|
||
"Referenced in Homelab/a.md:\n"
|
||
'- Alt text: "first purpose"\n'
|
||
" Context: some paragraph.\n"
|
||
'- Alt text: "second purpose"'
|
||
)
|
||
assert _extract_context_line(text) == 'Referenced in Homelab/a.md: "first purpose"'
|
||
assert "second purpose" not in (_extract_context_line(text) or "")
|
||
|
||
|
||
def test_extract_context_line_multiline_context_body_is_ignored() -> None:
|
||
"""The ``Context:`` value is paragraph-joined (blank lines inside)
|
||
— the parser walks it without breaking the block."""
|
||
text = (
|
||
"Referenced in Homelab/a.md:\n"
|
||
'- Alt text: "purpose"\n'
|
||
" Context: para one.\n"
|
||
"\n"
|
||
"para two, and it contains a line that\n"
|
||
"looks like context continuation."
|
||
)
|
||
assert _extract_context_line(text) == 'Referenced in Homelab/a.md: "purpose"'
|
||
|
||
|
||
def test_extract_context_line_alt_with_inner_quotes_survives() -> None:
|
||
"""The block writer does not escape quotes: the greedy alt capture
|
||
(between the opening quote and the line's final quote) keeps an
|
||
alt with inner ``"`` intact."""
|
||
text = 'Referenced in Homelab/x.md:\n- Alt text: "He said "hi""'
|
||
assert _extract_context_line(text) == 'Referenced in Homelab/x.md: "He said "hi""'
|
||
|
||
|
||
def test_extract_context_line_spurious_header_is_dropped() -> None:
|
||
"""A look-alike ``Referenced in …:`` line WITHOUT its ``- Alt text:``
|
||
bullet (context text that merely looks like a header) parses to
|
||
``None`` — never a clause from the stray line."""
|
||
assert _extract_context_line(
|
||
"Referenced in Homelab/fake.md:\nsome context line that is not a bullet"
|
||
) is None
|
||
# A header with NO following line at all (end of value) is spurious too.
|
||
assert _extract_context_line("Referenced in Homelab/fake.md:") is None
|
||
# Spurious then REAL: only the real block's clause survives.
|
||
mixed = (
|
||
"Referenced in Homelab/fake.md:\nsome context line\n\n"
|
||
+ CTX_SINGLE
|
||
)
|
||
assert _extract_context_line(mixed) == (
|
||
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"'
|
||
)
|
||
|
||
|
||
def test_extract_context_line_header_after_header_drops_the_bulletless_one() -> None:
|
||
"""Two headers in a row: the first (never followed by its bullet) is
|
||
dropped and the second block parses normally."""
|
||
text = "Referenced in Homelab/fake.md:\nReferenced in Homelab/real.md:\n- Alt text: \"alt\""
|
||
assert _extract_context_line(text) == 'Referenced in Homelab/real.md: "alt"'
|
||
|
||
|
||
def test_extract_context_line_markdown_pending_record_is_none() -> None:
|
||
"""The task-04 ``IMAGE_CONTEXT:`` record (the MARKDOWN doc's stored
|
||
``image_context``) has no ``Referenced in`` header — it parses to
|
||
``None`` (and it rides a text doc, which the ``is_image`` guard
|
||
excludes from the line anyway)."""
|
||
assert _extract_context_line(MD_PENDING_RECORD) is None
|
||
|
||
|
||
def test_extract_context_line_blank_and_empty_are_none() -> None:
|
||
assert _extract_context_line("") is None
|
||
assert _extract_context_line(" \n \t ") is None
|
||
# Garbage that contains the keyword somewhere is not a block.
|
||
assert _extract_context_line("I was referenced in Homelab/a.md: yesterday") is None
|
||
|
||
|
||
# ---------- _document_body: the block-body contract ----------
|
||
|
||
|
||
def test_document_body_enriched_image_doc_leads_with_the_context_line() -> None:
|
||
"""The completion criterion at the body level: the context line
|
||
leads, a blank line, then the description — the exact two-part
|
||
shape."""
|
||
body = _document_body(_image_doc(image_context=CTX_SINGLE))
|
||
assert body == (
|
||
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"\n\n'
|
||
"A diagram of the cluster."
|
||
)
|
||
|
||
|
||
def test_document_body_image_doc_null_context_is_byte_identical_to_pre_phase() -> None:
|
||
"""LOCKED A4 (the byte-identity criterion): a standalone / pre-phase
|
||
image doc (NULL ``image_context``) → the body is EXACTLY the
|
||
stripped summary — no line, no blank, no drift."""
|
||
doc = _image_doc(image_context=None)
|
||
assert _document_body(doc) == "A diagram of the cluster."
|
||
# And the built prompt matches a text twin (same fields, not an
|
||
# image) to the byte.
|
||
twin = _doc("cluster-diagram.png", "A diagram of the cluster.", "cluster-diagram")
|
||
assert build_high_prompt([doc]) == build_high_prompt([twin])
|
||
|
||
|
||
def test_document_body_text_doc_with_image_context_is_unchanged() -> None:
|
||
"""The guard is ``is_image``: a TEXT doc whose ``image_context``
|
||
carries the task-04 pending record (the markdown case) keeps the
|
||
pre-phase body — the record never renders."""
|
||
doc = _doc("guide.md", "CONTENT", "Guide", summary="Guide summary.",
|
||
image_context=MD_PENDING_RECORD)
|
||
assert _document_body(doc) == "Guide summary."
|
||
assert "IMAGE_CONTEXT" not in build_high_prompt([doc])
|
||
assert "Referenced in" not in build_high_prompt([doc])
|
||
|
||
|
||
def test_document_body_image_doc_unparsable_context_is_unchanged() -> None:
|
||
"""Graceful degradation: an image doc whose ``image_context`` parses
|
||
to nothing (owner-edited / unknown shape) → the description only,
|
||
exactly pre-phase — the prompt build never raises."""
|
||
for junk in ("", " ", "referenced in lowercase.md: no header shape", MD_PENDING_RECORD):
|
||
assert _document_body(_image_doc(image_context=junk)) == "A diagram of the cluster."
|
||
|
||
|
||
def test_document_body_enriched_image_doc_without_summary_precedes_the_preview() -> None:
|
||
"""The line leads the fallback body too (the preview + marker of
|
||
LOCKED A5) — the "context before the description" contract holds on
|
||
both body shapes."""
|
||
doc = _image_doc(image_context=CTX_SINGLE, summary=None)
|
||
doc.content = "A" * 400 + "B" * 600 # past the 400-char preview cap
|
||
body = _document_body(doc)
|
||
line = 'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"'
|
||
assert body.startswith(line + "\n\n")
|
||
assert body[len(line) + 2 :] == "A" * 400 + "\n" + TRUNCATION_MARKER
|
||
assert "B" * 40 not in body # beyond the cut, as pre-phase
|
||
|
||
|
||
def test_enriched_image_doc_with_summary_makes_no_settings_call(
|
||
monkeypatch: pytest.MonkeyPatch,
|
||
) -> None:
|
||
"""House pattern (the LOCKED A5 preview-cap contract): the
|
||
context-line path for a summary-bearing doc is pure — a dead
|
||
``get_settings`` must not break such a build (extraction is pure;
|
||
the cap is read on the fallback path only)."""
|
||
from app.rag import prompts as prompts_mod
|
||
|
||
doc = _image_doc(image_context=CTX_SINGLE)
|
||
|
||
def _dead() -> None: # pragma: no cover - must never be called
|
||
raise AssertionError("get_settings() called for a summarized doc")
|
||
|
||
monkeypatch.setattr(prompts_mod, "get_settings", _dead)
|
||
assert _document_body(doc) == (
|
||
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"\n\n'
|
||
"A diagram of the cluster."
|
||
)
|
||
|
||
|
||
# ---------- build_high_prompt: the line in the built block ----------
|
||
|
||
|
||
def test_high_prompt_enriched_image_doc_block_exact_shape() -> None:
|
||
"""The full block in the built prompt: identity attributes
|
||
unchanged (phase 106 date appended after title), then the context
|
||
line, a blank line, the description — the pre-phase block shape
|
||
with the line spliced in."""
|
||
doc = _image_doc(image_context=CTX_SINGLE)
|
||
prompt = build_high_prompt([doc])
|
||
assert (
|
||
'<document source="Homelab" path="cluster-diagram.png" '
|
||
'title="cluster-diagram" date="2024-06-15">\n'
|
||
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"\n'
|
||
"\n"
|
||
"A diagram of the cluster.\n"
|
||
"</document>"
|
||
) in prompt
|
||
# The line is the FIRST thing in the block body (before the
|
||
# description), inside <documents>.
|
||
line = 'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"'
|
||
block_start = prompt.index('<document source="Homelab"')
|
||
assert prompt.index("<documents>") < prompt.index(line)
|
||
assert prompt.index(line) < prompt.index("A diagram of the cluster.")
|
||
assert block_start < prompt.index(line) < prompt.index("</document>")
|
||
# The Context: body stays out of the whole prompt.
|
||
assert "The diagram shows the three Talos nodes" not in prompt
|
||
|
||
|
||
def test_high_prompt_enriched_image_doc_mixed_with_text_docs() -> None:
|
||
"""A suggested tier with both text docs and one enriched image doc:
|
||
the line rides ONLY the image doc's block; the text blocks are
|
||
byte-identical to their summaries (no line, no drift)."""
|
||
text_doc = _doc("kubernetes.md", "CONTENT", "K8s", summary="K8s summary.")
|
||
img = _image_doc(image_context=CTX_SINGLE)
|
||
prompt = build_high_prompt([text_doc, img])
|
||
assert prompt.count('Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"') == 1
|
||
# The text block is the bare summary, exactly as pre-phase.
|
||
assert (
|
||
'<document source="Homelab" path="kubernetes.md" '
|
||
'title="K8s" date="2024-06-15">\nK8s summary.\n</document>'
|
||
) in prompt
|
||
# The image block carries the line + description.
|
||
assert (
|
||
'title="cluster-diagram" date="2024-06-15">\n'
|
||
'Referenced in Homelab/homelab/kubernetes.md: "Cluster layout"\n\n'
|
||
"A diagram of the cluster.\n</document>"
|
||
) in prompt
|
||
# Ordering unchanged: the text block (first doc) still precedes the
|
||
# image block (second doc) — the line never re-orders the tier.
|
||
assert prompt.index('path="kubernetes.md"') < prompt.index('path="cluster-diagram.png"')
|
||
|
||
|
||
def test_high_prompt_image_doc_without_context_is_byte_identical_to_pre_phase() -> None:
|
||
"""LOCKED A4 at the prompt level: an image doc with NULL
|
||
``image_context`` builds the EXACT pre-phase prompt (the text-twin
|
||
equality — no context line anywhere, not even a stray blank)."""
|
||
img = _image_doc(image_context=None)
|
||
twin = _doc("cluster-diagram.png", "A diagram of the cluster.", "cluster-diagram")
|
||
assert build_high_prompt([img]) == build_high_prompt([twin])
|
||
assert "Referenced in" not in build_high_prompt([img])
|
||
|
||
|
||
# ---------- phase 137 (task 04, ASSUMPTION A6): the provenance clause ----------
|
||
# The stored value of a PDF-embedded image doc (tasks 02/03) can carry
|
||
# the ``Extracted from <source>/<pdf path> (page N):`` provenance block
|
||
# — written FIRST (A2: provenance before purpose) — ahead of the
|
||
# markdown reference blocks. The renderer turns that header line into
|
||
# the one clause ``From <source>/<pdf> (page N)`` (the page-text body
|
||
# is NEVER rendered), joins clauses in stored order, and keeps the
|
||
# markdown-only values byte-identical (the phase-125 pin, unedited).
|
||
|
||
#: The exact stored provenance block shape with page text (tasks
|
||
#: 02/03): the header + the page text (capped, stripped).
|
||
PDF_CTX_SINGLE = (
|
||
"Extracted from Homelab/docs/guide.pdf (page 2):\n"
|
||
"The second page of the guide, with the diagram.\n"
|
||
"More page text on the next line."
|
||
)
|
||
#: The tasks 02/03 COMBINED shape (A2): the document block first, then
|
||
#: the markdown reference blocks, blank-line joined.
|
||
PDF_CTX_BOTH = (
|
||
"Extracted from Homelab/docs/guide.pdf (page 2):\n"
|
||
"The second page of the guide, with the diagram.\n"
|
||
"\n"
|
||
"Referenced in Homelab/ops/runbook.md:\n"
|
||
'- Alt text: "alt"'
|
||
)
|
||
|
||
|
||
def test_extract_context_line_pdf_block_alone() -> None:
|
||
"""One provenance block → the clause EXACTLY: the locator + page
|
||
number — the page-text body never renders."""
|
||
line = _extract_context_line(PDF_CTX_SINGLE)
|
||
assert line == "From Homelab/docs/guide.pdf (page 2)"
|
||
assert "The second page of the guide" not in line
|
||
assert "More page text" not in line
|
||
assert line.count("\n") == 0 # genuinely ONE line
|
||
|
||
|
||
def test_extract_context_line_pdf_block_textless_page_is_header_only() -> None:
|
||
"""A text-less page stores the header line alone (tasks 02/03) →
|
||
the same single clause."""
|
||
assert _extract_context_line(
|
||
"Extracted from Homelab/docs/guide.pdf (page 2):"
|
||
) == "From Homelab/docs/guide.pdf (page 2)"
|
||
|
||
|
||
def test_extract_context_line_pdf_locator_greedy_to_the_last_page_marker() -> None:
|
||
"""The pdf path may itself contain spaces — ``loc`` is greedy to
|
||
the LAST `` (page N):`` on the line."""
|
||
assert _extract_context_line(
|
||
"Extracted from Sources/My Docs/guide v2.pdf (page 10):"
|
||
) == "From Sources/My Docs/guide v2.pdf (page 10)"
|
||
|
||
|
||
def test_extract_context_line_pdf_block_page_text_body_is_skipped() -> None:
|
||
"""The page-text body (multi-line, with a blank line and a
|
||
look-alike ``Referenced in …:`` line WITHOUT the alt bullet) is
|
||
skipped / dropped (the existing spurious-header rule) — the clause
|
||
stays the single one."""
|
||
text = (
|
||
"Extracted from Homelab/docs/guide.pdf (page 2):\n"
|
||
"Page text line one.\n"
|
||
"\n"
|
||
"Referenced in the appendix: it goes on like this.\n"
|
||
"More page text after the look-alike header."
|
||
)
|
||
assert _extract_context_line(text) == "From Homelab/docs/guide.pdf (page 2)"
|
||
|
||
|
||
def test_extract_context_line_spurious_pdf_header_in_page_text_emits_a_clause() -> None:
|
||
"""The accepted asymmetry (documented in the docstring): a
|
||
page-text line that LOOKS like a provenance header is NOT guarded
|
||
(unlike the ``Referenced in`` look-alikes, which demand their
|
||
bullet) — it emits a clause. The stored value is the context the
|
||
importer wrote, not free model text."""
|
||
text = (
|
||
"Extracted from Homelab/guide.pdf (page 1):\n"
|
||
"Page text carrying a look-alike line:\n"
|
||
"Extracted from Somewhere/other.pdf (page 9):\n"
|
||
"more page text"
|
||
)
|
||
assert _extract_context_line(text) == (
|
||
"From Homelab/guide.pdf (page 1); From Somewhere/other.pdf (page 9)"
|
||
)
|
||
|
||
|
||
def test_extract_context_line_markdown_only_two_blocks_pin_unchanged() -> None:
|
||
"""The phase-125 byte-identity pin (the task's shape): a
|
||
MARKDOWN-ONLY value renders exactly as pre-task — ONE
|
||
``Referenced in `` prefix over the joined clauses (the second
|
||
block's empty alt renders bare); a single block is unchanged too."""
|
||
text = (
|
||
'Referenced in docs/a.md:\n- Alt text: "alt a"\n'
|
||
"\n"
|
||
'Referenced in docs/b.md:\n- Alt text: ""'
|
||
)
|
||
assert _extract_context_line(text) == 'Referenced in docs/a.md: "alt a"; docs/b.md'
|
||
assert _extract_context_line('Referenced in docs/a.md:\n- Alt text: "alt a"') == (
|
||
'Referenced in docs/a.md: "alt a"'
|
||
)
|
||
|
||
|
||
def test_extract_context_line_both_blocks_document_first() -> None:
|
||
"""The tasks 02/03 combined shape (A2: the document block first,
|
||
then the markdown blocks) renders the clauses IN STORED ORDER —
|
||
each with its own prefix; the page-text body stays out."""
|
||
line = _extract_context_line(PDF_CTX_BOTH)
|
||
assert line == (
|
||
'From Homelab/docs/guide.pdf (page 2); '
|
||
'Referenced in Homelab/ops/runbook.md: "alt"'
|
||
)
|
||
assert "The second page of the guide" not in line
|
||
|
||
|
||
def test_extract_context_line_markdown_first_then_pdf_keeps_stored_order() -> None:
|
||
"""The reverse stored order (a markdown block BEFORE the document
|
||
block — an owner-edited value) renders in stored order too: the
|
||
complete reference block flushes when the provenance header
|
||
arrives, and the mixed join gives each clause its own prefix."""
|
||
text = (
|
||
"Referenced in Homelab/ops/runbook.md:\n"
|
||
'- Alt text: "alt"\n'
|
||
"\n"
|
||
"Extracted from Homelab/docs/guide.pdf (page 2):\n"
|
||
"The second page of the guide."
|
||
)
|
||
assert _extract_context_line(text) == (
|
||
'Referenced in Homelab/ops/runbook.md: "alt"; '
|
||
"From Homelab/docs/guide.pdf (page 2)"
|
||
)
|
||
|
||
|
||
def test_extract_context_line_pdf_garbage_is_none() -> None:
|
||
"""No parsable header of either kind → ``None`` (unchanged
|
||
degradation): the provenance keyword without the exact header
|
||
shape is not a block."""
|
||
# No trailing colon ...
|
||
assert _extract_context_line("Extracted from Homelab/guide.pdf (page 2)") is None
|
||
# ... the lowercase keyword ...
|
||
assert _extract_context_line("extracted from Homelab/guide.pdf (page 2):") is None
|
||
# ... the missing parentheses.
|
||
assert _extract_context_line("Extracted from Homelab/guide.pdf page 2:") is None
|
||
|
||
|
||
def test_document_body_pdf_image_doc_leads_with_the_from_line() -> None:
|
||
"""End-to-end: a PDF-embedded image doc's ``<document>`` body
|
||
leads with the one-line rendering (provenance first, then the
|
||
reference), a blank line, then the description — the page-text
|
||
body never renders."""
|
||
body = _document_body(_image_doc(image_context=PDF_CTX_BOTH))
|
||
assert body == (
|
||
'From Homelab/docs/guide.pdf (page 2); '
|
||
'Referenced in Homelab/ops/runbook.md: "alt"\n'
|
||
"\n"
|
||
"A diagram of the cluster."
|
||
)
|
||
assert "The second page of the guide" not in body
|
||
|
||
|
||
def test_document_body_text_doc_with_pdf_context_is_unchanged() -> None:
|
||
"""The guard is ``is_image``: a TEXT doc carrying a provenance
|
||
context keeps the byte-identical pre-phase body — the context
|
||
never renders."""
|
||
doc = _doc("guide.pdf", "CONTENT", "Guide", summary="Guide summary.",
|
||
image_context=PDF_CTX_SINGLE)
|
||
assert _document_body(doc) == "Guide summary."
|
||
assert "Extracted from" not in build_high_prompt([doc])
|
||
assert "From Homelab/docs/guide.pdf (page 2)" not in build_high_prompt([doc])
|
||
|
||
|
||
def test_high_prompt_pdf_image_doc_block_exact_shape() -> None:
|
||
"""The full block in the built prompt: the ``From`` line leads the
|
||
body (provenance first) before the description — the pre-phase
|
||
block shape with the line spliced in; the page-text body stays out
|
||
of the whole prompt."""
|
||
doc = _image_doc(image_context=PDF_CTX_SINGLE)
|
||
prompt = build_high_prompt([doc])
|
||
assert (
|
||
'<document source="Homelab" path="cluster-diagram.png" '
|
||
'title="cluster-diagram" date="2024-06-15">\n'
|
||
"From Homelab/docs/guide.pdf (page 2)\n"
|
||
"\n"
|
||
"A diagram of the cluster.\n"
|
||
"</document>"
|
||
) in prompt
|
||
line = "From Homelab/docs/guide.pdf (page 2)"
|
||
block_start = prompt.index('<document source="Homelab"')
|
||
assert prompt.index("<documents>") < prompt.index(line)
|
||
assert prompt.index(line) < prompt.index("A diagram of the cluster.")
|
||
assert block_start < prompt.index(line) < prompt.index("</document>")
|
||
assert "The second page of the guide" not in prompt
|
||
|
||
|
||
# ---------------------------------------------------------------------------
|
||
# Phase 191 (D4, task 05): the ``<document>`` block learns the owner's name
|
||
# ---------------------------------------------------------------------------
|
||
|
||
#: The owner's display name for the fixture source (phase 191 D1 — an
|
||
#: ALIAS beside the derived name, never a replacement for it).
|
||
GIT_SERVER = "the git server"
|
||
|
||
#: The owner's one-line description of the same source.
|
||
FORGE_LABEL = "self-hosted forge + Gitea"
|
||
|
||
|
||
def _label_view(
|
||
source: str = "Homelab",
|
||
display_name: str | None = None,
|
||
label: str | None = None,
|
||
tags: tuple[str, ...] = (),
|
||
) -> SourceLabelView:
|
||
"""One label-row view (task 01's frozen dataclass) for the pins."""
|
||
return SourceLabelView(
|
||
source=source,
|
||
display_name=display_name,
|
||
label=label,
|
||
tags=tags,
|
||
)
|
||
|
||
|
||
def _block_head(prompt: str) -> str:
|
||
"""The opening tag of the FIRST ``<document>`` block (attribute
|
||
order/escaping pins read it whole, so nothing is asserted loosely)."""
|
||
start = prompt.index("<document ")
|
||
return prompt[start : prompt.index(">", start) + 1]
|
||
|
||
|
||
@pytest.mark.parametrize(
|
||
"names",
|
||
[
|
||
None,
|
||
{},
|
||
{"Other": _label_view("Other", display_name=GIT_SERVER, label=FORGE_LABEL)},
|
||
],
|
||
ids=["names=None", "names={}", "no-entry-for-the-source"],
|
||
)
|
||
def test_high_prompt_without_a_label_entry_is_byte_identical_to_pre_phase(
|
||
names: dict[str, SourceLabelView] | None,
|
||
) -> None:
|
||
"""THE kill pin (phase 191 D4): no map, an empty map and a map with no
|
||
entry for the document's source each build a prompt byte-identical to
|
||
the pre-phase build — no ``name``, no ``label``, no stray space."""
|
||
doc = _doc(
|
||
"kubernetes.md",
|
||
"Talos Linux on three nodes.",
|
||
"Kubernetes Homelab Cluster",
|
||
summary="A Talos Linux cluster on three nodes.",
|
||
)
|
||
pre = build_high_prompt([doc])
|
||
assert build_high_prompt([doc], names=names) == pre
|
||
assert ' name="' not in pre and ' label="' not in pre
|
||
|
||
|
||
def test_no_label_block_keeps_the_exact_pre_phase_head() -> None:
|
||
"""The block head for a label-less source, byte for byte (the shape the
|
||
phase-106/118/125 pins rely on: ``source`` → ``path`` → ``title`` →
|
||
``date``, single-spaced)."""
|
||
doc = _doc("kubernetes.md", "CONTENT", "Kubernetes Homelab Cluster", summary="S.")
|
||
assert _block_head(build_high_prompt([doc], names={})) == (
|
||
'<document source="Homelab" path="kubernetes.md" '
|
||
'title="Kubernetes Homelab Cluster" date="2024-06-15">'
|
||
)
|
||
|
||
|
||
def test_document_block_name_attribute_sits_immediately_after_source() -> None:
|
||
"""A ``display_name``-only row: exactly `` name="…"`` added, directly
|
||
after ``source="…"`` and before ``path`` (the attribute-order pin)."""
|
||
doc = _doc("kubernetes.md", "CONTENT", "Kubernetes", summary="S.")
|
||
prompt = build_high_prompt([doc], names={"Homelab": _label_view(display_name=GIT_SERVER)})
|
||
assert _block_head(prompt) == (
|
||
'<document source="Homelab" name="the git server" path="kubernetes.md" '
|
||
'title="Kubernetes" date="2024-06-15">'
|
||
)
|
||
assert ' label="' not in prompt
|
||
|
||
|
||
def test_document_block_label_only_view_adds_exactly_the_label_attribute() -> None:
|
||
"""A ``label``-only row gains ONE attribute — an unset field adds
|
||
NOTHING (never an empty ``name=""``)."""
|
||
doc = _doc("kubernetes.md", "CONTENT", "Kubernetes", summary="S.")
|
||
prompt = build_high_prompt([doc], names={"Homelab": _label_view(label=FORGE_LABEL)})
|
||
assert _block_head(prompt) == (
|
||
'<document source="Homelab" label="self-hosted forge + Gitea" '
|
||
'path="kubernetes.md" title="Kubernetes" date="2024-06-15">'
|
||
)
|
||
assert ' name="' not in prompt
|
||
|
||
|
||
def test_document_block_row_with_neither_field_is_byte_identical() -> None:
|
||
"""A row that exists but carries neither name nor label (the owner
|
||
saved only tags) leaves the block untouched — tags never render in the
|
||
block (they are a retrieval input, D2/D3)."""
|
||
doc = _doc("kubernetes.md", "CONTENT", "Kubernetes", summary="S.")
|
||
pre = build_high_prompt([doc])
|
||
names = {"Homelab": _label_view(tags=("gitea", "forge"))}
|
||
built = build_high_prompt([doc], names=names)
|
||
assert built == pre
|
||
assert "gitea" not in built
|
||
|
||
|
||
def test_document_block_carries_both_attributes_in_the_locked_order() -> None:
|
||
"""Both fields set → both attributes, in the fixed order
|
||
``source`` → ``name`` → ``label`` → ``path`` → ``title`` → ``date``."""
|
||
doc = _doc("kubernetes.md", "CONTENT", "Kubernetes", summary="S.")
|
||
names = {"Homelab": _label_view(display_name=GIT_SERVER, label=FORGE_LABEL)}
|
||
head = _block_head(build_high_prompt([doc], names=names))
|
||
order = ['source="', 'name="', 'label="', 'path="', 'title="', 'date="']
|
||
positions = [head.index(attr) for attr in order]
|
||
assert positions == sorted(positions)
|
||
assert head == (
|
||
'<document source="Homelab" name="the git server" '
|
||
'label="self-hosted forge + Gitea" path="kubernetes.md" '
|
||
'title="Kubernetes" date="2024-06-15">'
|
||
)
|
||
|
||
|
||
def test_document_block_name_and_label_are_xml_escaped() -> None:
|
||
"""The escaping pin (task 05 work item 1): a ``"`` in owner text renders
|
||
as ``"`` so the attribute cannot break; ``&``/``<`` fold too."""
|
||
doc = _doc("kubernetes.md", "CONTENT", "Kubernetes", summary="S.")
|
||
names = {
|
||
"Homelab": _label_view(
|
||
display_name='Reese "forge" & <co>',
|
||
label='say "hi"',
|
||
)
|
||
}
|
||
prompt = build_high_prompt([doc], names=names)
|
||
assert ' name="Reese "forge" & <co>"' in prompt
|
||
assert ' label="say "hi""' in prompt
|
||
# The unescaped form never reaches the wire: the attribute stays closed.
|
||
assert ' name="Reese "forge"' not in prompt
|
||
head = _block_head(prompt)
|
||
assert head.count('"') % 2 == 0, "an unescaped quote would break the attribute"
|
||
|
||
|
||
def test_attribute_escaping_helper_covers_the_xml_five() -> None:
|
||
"""``_attr`` is the ONE escaper for the new attribute values: the XML
|
||
specials go, the single quote stays (the delimiter is the double
|
||
quote, and ``Reese's`` must stay readable to the model)."""
|
||
assert _attr('a & b < c > d " e') == "a & b < c > d " e"
|
||
assert _attr("Reese's forge") == "Reese's forge"
|
||
|
||
|
||
def test_names_apply_per_document_source_and_never_touch_the_body() -> None:
|
||
"""Only the blocks whose source has an entry gain the attribute; the
|
||
summary body (and the other source's block) are untouched."""
|
||
named = _doc("a.md", "CONTENT_A", "Title A", summary="SUMMARY A.")
|
||
other = _doc("b.md", "CONTENT_B", "Title B", summary="SUMMARY B.")
|
||
other.source = "Other"
|
||
prompt = build_high_prompt(
|
||
[named, other],
|
||
names={"Homelab": _label_view(display_name=GIT_SERVER)},
|
||
)
|
||
assert 'source="Homelab" name="the git server" path="a.md"' in prompt
|
||
assert 'source="Other" path="b.md"' in prompt
|
||
assert prompt.count(GIT_SERVER) == 1, "the name rides the attribute, nothing else"
|
||
assert "SUMMARY A." in prompt and "SUMMARY B." in prompt
|
||
|
||
|
||
def test_names_on_an_empty_documents_build_change_nothing() -> None:
|
||
"""No blocks → nothing to annotate: the honest "(no documents
|
||
matched…)" build is byte-identical with a label row in hand."""
|
||
names = {"Homelab": _label_view(display_name=GIT_SERVER, label=FORGE_LABEL)}
|
||
assert build_high_prompt([], names=names) == build_high_prompt([])
|
||
|
||
|
||
def test_names_compose_with_folder_lines_and_display_clauses() -> None:
|
||
"""The additive attributes ride alongside the phase-119 folder lines and
|
||
the phase-124/182 display clauses — the section order and the intro are
|
||
untouched (the folder lines stay plain, no new markup)."""
|
||
doc = _doc("kubernetes.md", "CONTENT", "Kubernetes", summary="S.")
|
||
names = {"Homelab": _label_view(display_name=GIT_SERVER)}
|
||
prompt = build_high_prompt(
|
||
[doc],
|
||
folder_lines=["docs/gitea.md — Gitea install"],
|
||
show_image=True,
|
||
show_text=True,
|
||
names=names,
|
||
)
|
||
assert 'source="Homelab" name="the git server" path="kubernetes.md"' in prompt
|
||
assert prompt.index(SUGGEST_INTRO) < prompt.index("docs/gitea.md — Gitea install")
|
||
assert prompt.index("docs/gitea.md — Gitea install") < prompt.index("<document ")
|
||
assert prompt.endswith(TOOLS_SECTION[: -len("</tools>")]
|
||
+ SHOW_IMAGE_CLAUSE + "\n\n" + SHOW_TEXT_CLAUSE + "\n</tools>")
|
||
|
||
|
||
def test_deflect_prompt_has_no_name_or_label_surface() -> None:
|
||
"""D4 touches the suggested-tier block only: the deflected prompt is
|
||
byte-identical (it carries no document blocks to annotate)."""
|
||
prompt = build_deflect_prompt(["Kubernetes Homelab Cluster"])
|
||
assert ' name="' not in prompt and ' label="' not in prompt
|
||
assert GIT_SERVER not in prompt
|