"""Locked system-prompt builder (PLAN §6). The persona + HONESTY GATE text is **locked verbatim** — change it through the plan, not here. (PLAN §6 revision, 2026-08-22: the owner's working-tree persona edits are preserved — no mandated ``"you've got this"`` tagline and no mandated deflection opening; the honesty gate itself is unchanged.) Two modes: * ``HIGH`` — grounded turn: the top-ranked documents' SUMMARIES under ```` (phase 118, LOCKED A6 re-revising A7 — the "start here" suggestion seeding: each block is the document's stored summary, never the full content; the LLM extends its context by ``read``-ing a document's full text through the capped ``read`` tool, the ONLY full- text path. A NULL/blank summary — a fail-soft import miss — falls back to a ``suggestion_preview_chars`` content preview + the shared ``[…truncated…]`` marker; no LLM call at chat time, LOCKED A5). * ``LOW`` — deflection turn: weak-hit *titles only* plus the ``DEFLECT_MODE`` marker (the E2E mock LLM keys on that marker). Steering (phase 15): when the owner has stored tuning notes, both modes carry a ```` section between ``…`` and the mode body. With zero notes the prompt is byte-identical to the pre-steering text. KB overview (phase 31): when the single ``kb_overview`` row holds a lite-generated outline of the knowledge base, both modes carry a ```` section between ``…`` and the ```` section (order: ```` → ```` → ```` → mode body) — the agent knows roughly what the KB contains before retrieval. With an empty row the prompt is byte-identical to the pre-phase text. Agent tools (phase 37; phase 70: the copy teaches the harness-aligned ``ls`` / ``read`` / ``grep`` shapes; phase 72: the copy states the document-identity contract up front — ``ls``'s ``path`` is a *source name*, not a directory or file path, and ``read``/``grep`` take the combined ``source/path`` string *including the source name* (a bare document path will not resolve) — the same two things the phase-72 teaching refusals in :mod:`app.rag.agent` re-state after the fact, so the model carries the contract before it calls a tool; phase 94, task 03: the ``ls`` clause is rewritten to the drill-down tree contract — no ``path``: the synced sources with counts + summaries, a source name: its top-level folders + files, a ``source/folder`` path: one level deeper — a listing shows only that level's subfolders + own files (never the whole KB in one call), the folder summaries say what's in a folder before drilling, and ``grep`` stays the locator for finding one document without listing; the ``read``/``grep`` clauses and the discipline rules are byte-identical): the **HIGH** prompt only carries a ```` section after the ```` body — the grounded turn may extend its context through the three server-side tools (round-capped, see :mod:`app.rag.agent`; the cap is the bound and this section does not re-state it, phase 45). The LOW/deflection prompt never carries it (phase 71: the LOW prompt's only addition is the plain-text line below — it still has no ```` section). Deflection plain-text line (phase 71, owner-permitted 2026-09-03): the otherwise-locked ``LOW`` prompt gains exactly one instruction line — "Reply in plain text only — you have no tools in this mode." — appended to the ``DEFLECT_MODE`` body: a deflected turn offers no tools, so any tool markup there is always wrong, and the line closes the door at the prompt (the deterministic filter + one bounded recovery in :mod:`app.rag.scaffolding` / :mod:`app.rag.agent` is the backstop). The ``DEFLECT_MODE`` marker and everything else in the prompt stay put — the E2E mock LLM keys on the marker's *presence*, not the wording, so that contract is unchanged. Display-tool clauses (phase 124 ``show_image``; phase 182 ``show_text``): the **HIGH** prompt's ```` section can carry two extra clauses before its closing tag, each appended only when its own setting is on — :data:`SHOW_IMAGE_CLAUSE` when ``BOR_ANSWER_IMAGES`` (default OFF) and :data:`SHOW_TEXT_CLAUSE` when ``BOR_ANSWER_TEXT`` (default ON). The text clause teaches the two things the phase-182 incident needed said: the user copies from what is DISPLAYED (so never retype a file), and the inline `` ```bor [range] ``` `` locator fence whose EMPTY body the app replaces with the server-resolved bytes. Both clauses compose through one mechanism (section minus its final tag + the present clauses + the tag), so a build with neither clause is byte-identical to pre-phase and a ``show_image``-only build reproduces the phase-124 bytes exactly; the LOW prompt carries neither. Disclosed-answer contract (phase 112, task 03; owner decision 2026-09-14, roadmap confirmation — option (iii) of the three the 2026-09-15 interactive deflection test raised): that test found the HONESTY GATE's compliance is **stochastic** across runs when misleading context is injected — the "What is the capital of Mongolia?" question had two *irrelevant* docs promoted into the HIGH prompt by a weak single-token FTS hit (the pre-phase A8 rule: any ``fts_hits > 0`` flipped HIGH), and run 1 answered parametrically — "Ulaanbaatar" — *with an explicit disclosure*, while the identical one-tap re-run produced a clean, textbook deflection. The owner's decision: the disclosed general-knowledge answer is treated as **acceptable and documented** — a small local model cannot be relied on to obey Rules 1/3 100% when handed misleading context, so this prompt text stays byte-identical (LOCKED verbatim; options (i) tighten the copy / (ii) amend this locked prompt via the plan to explicitly permit disclosed general-knowledge answers remain open to a future owner decision). The deterministic lever is the honesty gate itself (A8 revised 2026-09-14: an FTS hit flips HIGH only when ``best_cosine >= lexical_support_floor``, so the misleading docs are no longer injected — see :mod:`app.api.chat`). Source names, labels and tags (phase 191, D4): a ```` block can carry the owner's label row for its source as two ADDITIVE attributes — ``name="…"`` (the source's ``display_name``) immediately after ``source="…"``, then ``label="…"`` (the one-line description) — so the model learns the name→docs mapping ("the git server" is a source with these documents). The attribute order is fixed (``source`` → ``name?`` → ``label?`` → ``path`` → ``title`` → ``date``), an unset field adds NOTHING, and no *names* map / no entry for the source leaves the block byte-identical to pre-phase (the alias contract, D1: the identity attributes stay the DERIVED name — a display name never replaces ``source`` or ``path``). Only the two new values are XML-escaped (they are free-form owner text); the identity attributes keep their pre-phase unescaped bytes. """ from __future__ import annotations import re from collections.abc import Mapping, Sequence from typing import Any from xml.sax.saxutils import escape from app.config import Settings, get_settings from app.models import Document from app.rag.retriever import TRUNCATION_MARKER from app.rag.source_labels import SourceLabelView from app.schemas import HistoryTurn #: PLAN §6 verbatim (line wrapping included); ``{relevance}`` is filled by #: :func:`_base`. PERSONA: str = ( 'You are "Brain of Reese" — the digital brain of Reese, a self-hoster and\n' "homelab tinkerer. Personality: chippy, upbeat, warm, and genuinely\n" "optimistic about the user's ability to do things.\n" "\n" "Rules:\n" "1. Answer ONLY from the provided document context. Cite which document(s)\n" " you used, by path.\n" "2. Be concrete: names, versions, ports, hosts, schedules — the specifics in\n" " the docs are the value.\n" '3. HONESTY GATE: if is "LOW", you must NOT pretend to know.\n' " Offer 2-3 alternative questions about things you DO have notes on.\n" "4. Never invent facts, hosts, or steps that are not in the context.\n" "5. Keep answers tight: short paragraphs, bullets where helpful.\n" "\n" "{relevance}" ) #: One-line intro of the ```` section (phase 15): the owner's notes #: steer the answer and win over the defaults when they conflict. _STEERING_INTRO = ( "The owner of this brain asked you to steer your answers as follows. " "Where these instructions conflict with the defaults above, follow the owner:\n" ) #: One-line intro of the ```` section (phase 31; #: phase 142, task 03 — the honest-intro rewrite, owner-confirmed #: 2026-09-30): the outline is a COMPACT, possibly partial picture #: that favors RECENT documents (task 02 orders the generation input #: newest-first, so recent docs are the ones it covers when the cap #: bites) — a map of what exists, not a complete inventory — and #: "latest / complete set" questions verify with ``ls``/``grep`` #: before answering from it; "(generated at import time)" is GONE #: (false — the sync regenerates the row on every KB change — and it #: taught the model to treat a partial outline as the whole frozen KB). _KB_INTRO = ( 'A compact picture of the knowledge base\'s categories and topics — ' 'it favors recent documents and may omit older or lesser ones: a map ' 'of what exists, not a complete inventory. For questions about the ' 'latest state or the complete set ("the last N …", "everything ' 'about …", "all the …"), verify with `ls` (drill into the folders) ' 'or `grep` before answering from the outline:\n' ) #: The ```` section's start-here intro (phase 118, task 03 — #: the owner directive, TODO L3; phase 119, task 04, LOCKED A5 — the #: final sentence re-cut to the cite discipline): the seeded blocks are #: the SUMMARIES of the top-ranked documents for the question, opt-in #: starting points — start here if one seems right; ``read`` the #: document's combined ``source/path`` to add its FULL content before #: answering specifics (the full text is NOT in the prompt until you #: read it); cite ONLY the documents you read (or the suggested document #: you answered from without reading it) — never one you neither read #: nor used. The discipline sentence closes the phase-119 live #: confabulation — the model's "Docs used:" line cited a file it never #: read, confabulated from the suggestion blocks sitting in context — #: and retires the phase-118 "cite the document(s) you used, by path" #: sentence. Rendered BEFORE the first block, only when at least one #: block is present. The ```` tag and the per-block #: ```` markup stay byte-stable around it (the E2E mock's #: block parser and the ``read``-tool copy key off both). SUGGEST_INTRO = ( "The blocks below are the summaries of the top-ranked documents for " "your question — start here if one seems right to you: call `read` " "with that document's combined `source/path` to add its full content " "before answering specifics (its full text is not in the prompt until " "you read it). Cite only the document(s) you read — or, if you " "answered from a suggested summary without reading it, cite that " "suggested document — never a document you neither read nor used." ) #: The ```` instructions section — **HIGH prompt only** (phase 37, #: task 03; phase 70: the copy is rewritten for the harness-aligned #: ``ls`` / ``read`` / ``grep`` shapes, names/args exactly as the #: ``AGENT_TOOLS`` schemas in :mod:`app.rag.agent`; phase 72: the copy #: states the document-identity contract UP FRONT — ``ls``'s optional #: ``path`` is a *source name* (not a directory or file path) and #: ``read``/``grep`` take the combined ``source/path`` string *including #: the source name* (a bare document path will not resolve) — the same #: two things the phase-72 teaching refusals re-state after the fact; #: phase 94, task 03 — the owner-permitted tool-surface revision, #: recorded in the phase 94 overview: the ``ls`` clause is rewritten #: to the drill-down tree contract — no ``path``: the synced sources #: with counts + summaries, a source name: its top-level folders + #: files, a ``source/folder`` path: one level deeper — a listing shows #: only that level's subfolders + own files, never the whole KB in one #: call, the folder summaries say what's in a folder before drilling, #: and ``grep`` stays the locator for finding one document without #: listing; the ``read``/``grep`` clauses and the discipline rules are #: byte-identical): #: a grounded turn may extend its context through the three server-side #: tools (round cap: ``BOR_AGENT_MAX_ROUNDS`` — the cap is the bound and #: this section does not re-state it, phase 45). Appended after the mode #: body (````), so the instructions are the last thing the #: model reads. The LOW/deflection prompt never carries it — a #: deflection has no grounded context to extend (phase 71: the LOW #: prompt's only addition is the plain-text line in #: :func:`build_deflect_prompt`). The E2E mock keys off the ```` #: marker's *presence*, not this wording. Task 05 (the live gate's #: iteration loop) keeps the baseline and carries the discipline rules #: the live telemetry motivated, refined across the task-05 re-runs of #: 2026-09-03/04 (run 1: 7/16 executed, 44% — the ``lite`` model obeyed #: the user's "open it / read it" and ``read`` seed-context documents, #: then repeated the refused call; run 2: 8/18, 44% — the repeat is #: gone, but a grep turn batched two calls per reply and the harness #: runs only the first of a batch): the ``read`` clause carries the #: phase-118 summary-seed contract (the ```` section holds #: SUMMARIES — a suggested document's full text is not in the prompt #: until ``read`` adds it; do not re-read a document you have already #: read IN FULL — answer from the text already in the prompt — a #: PARTIAL document (a range read, or a truncated read) is not #: complete in the context until read all: read the other parts with #: ``offset`` / ``limit`` (phase 141, task 02 — the re-read sentence #: re-cut to the coverage-true form; the mock keys off the ```` #: marker's presence, not the wording, so it is untouched); if the #: user asks to open or read a suggested document, ``read`` it — that #: is the point of the section — the phase-72 do-not-read rule and #: the reverted ```` anchor are retired with the #: full-text seeds); #: the one-call rule names the consequence (a batched second call is #: discarded — runs only the first); the never-repeat rule says why #: (the refusal already told you the correct form); the ``grep`` clause #: leads with "for a normal search pass only ``pattern``" (the gate's #: live runs showed the model scoping ``grep`` with an ``ls``-style #: source name — the incident shape, but on grep). The behavioral #: contract lives in the ``AGENT_TOOLS`` descriptions as well (the most #: local text at call time): ``read`` adds a document's FULL text by #: its combined ``source/path`` identity — the ```` #: summaries are the starting points, not the content; ``grep`` with #: only ``pattern`` searches the whole knowledge base, and a source #: name is not a document. #: Phase 145 (task 02) — the original-source rule the ``read`` teaching #: carries (LOCKED verbatim — pinned in :mod:`tests.unit.test_prompt_lock` #: and :mod:`tests.unit.test_prompts`). :data:`TOOLS_SECTION` embeds this #: exact sentence-group at the ``read`` clause's end (the truncation #: workflow sentence) and BEFORE the ``grep`` clause begins; the #: cross-pins (``TOOLS_SECTION`` minus this rule reproduces the #: pre-phase-145 bytes; ``count == 1`` + the boundary substring) keep #: the embedded copy and this constant byte-identical. A read result for #: a git-sourced document carries an ``origin:`` line (and a ``branch:`` #: line when a branch is pinned) — the read contract's fact — so when #: the answer uses such a document it includes the original source: a #: link to the repository (the ``origin`` line) with the document's path #: (the ``Document …`` header) named and the pinned branch when one is #: shown, as the repository's file-view URL where the model can tell it #: (``/blob//``), or the repository URL plus the #: path as plain text otherwise — the user opens the complete original #: from the answer in one step, not a second question. The rule is #: CONDITIONAL on the ``origin`` line's presence — a local-sourced #: document's read result carries none, so the rule adds nothing to #: those answers (the model has no URL to cite; it must not invent one — #: the sentence's conditioning does that job). ORIGIN_RULE: str = ( "A read result for a document from a git source carries an origin line " "(and a branch line when a branch is pinned): when your answer uses " "such a document, include the original source in the answer — a link " "to the repository from the origin line, with the document's path " "(from the Document header) named and the pinned branch when one is " "shown — as the repository's file-view URL where you can tell it " "(for example /blob//), or the repository " "URL plus the path as plain text otherwise. The user should be able " "to open the complete original from your answer in one step, not ask " "again for the source." ) #: The search-discipline rule (phase 162, task 03, LOCKED A4) — appended #: to the ``TOOLS_SECTION`` tools paragraph immediately after the grep #: sentence (i.e. after the ``…retry with the plain text you expect to #: see.`` clause, the placement from the 2026-10-07 owner conversation #: where the agent answered from the lossy ```` outline #: instead of running ``ls``, and where the exact-substring grep silently #: missed the accented ``BESTÅ`` in the IKEA PDF's text layer). Adapted #: verbatim from the conversation's proposed system-prompt patch for the #: folded grep (LOCKED A3). The three lessons: (1) an outline/summary is #: a hint, never an inventory — ``ls`` (no ``path``) before claiming the #: KB lacks something; (2) grep folds accents but one pattern is one #: string — split filename tokens and the part number need separate #: greps; (3) a gitignored note describes repo status, not index status. #: The mock E2E is unaffected (it keys on markers, not this sentence). SEARCH_DISCIPLINE_RULE: str = ( "A summary or outline is a hint, never an inventory: before claiming " "the knowledge base lacks something, run `ls` with no `path` to list " "every source, and `grep` the artifact's part number or a filename " "token. `grep` folds accents — `besta` matches `BESTÅ` — but one " "pattern is one string: a name's split tokens (`frame`, `white`) and " "its part number (`AA-1272068`) need separate greps. A note saying a " "file is gitignored describes its repo status, not whether it is " "indexed as a source." ) TOOLS_SECTION: str = ( "\n" "You may extend your context with three tools. `ls` lists the " "knowledge base as a tree, one level at a time: with no `path` it " "lists every synced source with its document count and a summary " "of its contents; with a source name (e.g. 'homelab') it lists " "that source's top-level folders and files; with a `source/folder` " "path it drills one level deeper. A listing shows only that " "level's subfolders and its own files — never the whole knowledge " "base in one call — and each folder line's summary says what the " "folder contains before you drill into it. File lines are " "`source: X | path: Y | title: Z`; to find one specific document " "without listing, use `grep`. `read` pulls in one document by its " "combined `source/path` string, exactly as shown in the `ls` " "output — including the source name — or in the " "summary blocks — adding its full content to your context. The " " section holds SUMMARIES — the full text of a " "suggested document is not in your prompt until you `read` it. Do " "not re-read a document you have already read in full — its full " "text is in your prompt, answer from it. If you have read only " "part of a document (a range read, or a truncated read), read the " "other parts with `offset` and `limit` — the document is not " "complete in your context until you have read it all. If the user " "asks you to open or read a suggested document, `read` it — that " "is the point of the section. For `read`, a bare document path " "(without the source name) will not resolve. Very large documents " "are capped: a " "cut read returns the first part plus a TRUNCATED notice — the " "document did not end where it stopped; use `grep` (pattern) to " "find the rest, it searches the whole document. A read result for a " "document from a git source carries an origin line (and a branch line " "when a branch is pinned): when your answer uses such a document, " "include the original source in the answer — a link to the " "repository from the origin line, with the document's path (from the " "Document header) named and the pinned branch when one is shown — as " "the repository's file-view URL where you can tell it (for example " "/blob//), or the repository URL plus the " "path as plain text otherwise. The user should be able to open the " "complete original from your answer in one step, not ask again for " "the source. `grep` locates an exact string " "(case- and accent-insensitive — accents fold: `besta` matches " "`BESTÅ`) in the indexed documents and returns up to 20 " "matching `source/path:line: text` lines — a locator, not a " "context-adder: read the winner with `read`. A grep pattern is a " "plain substring, NEVER a regex — '.*' and '\\.' are literal text " "there; if such a pattern returns no matches, retry with the plain " "text you expect to see. " + SEARCH_DISCIPLINE_RULE + " For a normal search pass only `pattern` — " "its optional `path` argument " "limits the search to one document you already know, by the same " "combined `source/path` string; never a source name — a bare " "document path (without the source name) will not resolve there " "either. Make exactly one tool call per reply — a reply carrying " "two tool calls runs only the first, the second is discarded — and " "wait for the result before the next call. Never repeat a call that " "was refused or already succeeded — the refusal already told you " "the correct form. Answer as soon as you have what you need.\n" "" ) #: The ``show_image`` contract clause (phase 124, task 03, LOCKED A5) — #: appended INSIDE the ```` section of the HIGH prompt, after the #: existing three-tool copy and before the closing tag, ONLY when #: :func:`build_high_prompt` is called with ``show_image=True`` (the #: ``BOR_ANSWER_IMAGES`` toggle, via ``app.api.chat``). It teaches the #: same relevance gate as the ``show_image`` tool description in #: :mod:`app.rag.agent` (LOCKED A5: the gate is the LLM's decision — no #: server-side image scoring), plus the image-doc nature the tool #: description cannot fully carry: the text of an image document (in a #: ```` block or a ``read`` result) is a description #: GENERATED from the image, not the image's own words. The four gate #: elements, all stated here: (1) the image-doc nature; (2) the #: "contains the answer" bar — call it only when THAT image is what the #: user actually wants to see, never a merely related image, never to #: pad an answer; (3) at most ONE image per answer (also enforced #: server-side, task 02's ``ALREADY_SHOWING`` refusal); (4) the text #: answer is still provided in full — the image supplements, never #: replaces. ``TOOLS_SECTION`` itself stays BYTE-IDENTICAL (the #: pre-phase pins in :mod:`tests.unit.test_prompt_lock` hold); a #: ``show_image=False`` build (the default — every pre-phase call site) #: is byte-identical to the pre-task output. The ```` marker the #: E2E mock keys on is untouched either way — the clause rides on its #: presence, not the wording. The LOW/deflection prompt never carries #: the clause (a deflected turn has no tools and never shows an image — #: the phase's locked design). SHOW_IMAGE_CLAUSE: str = ( "`show_image` displays a standalone image document as part of your " "answer. The text of an image document — in a block or " "a `read` result — is a description GENERATED from the image, not " "the image's own words. Call `show_image` only when that image is " "what the user actually wants to see: a diagram, screenshot, or " "photo of the very thing you are answering about — never for a " "merely related image, and never to pad an answer. Show at most one " "image per answer. The image supplements your text — answer fully " "in text either way." ) #: The ``show_text`` contract clause (phase 182, task 04, LOCKED A1–A3/A6) — #: appended INSIDE the ```` section of the HIGH prompt, after the #: existing three-tool copy (and after :data:`SHOW_IMAGE_CLAUSE` when that #: rides too), ONLY when :func:`build_high_prompt` is called with #: ``show_text=True`` (the ``BOR_ANSWER_TEXT`` master switch, via #: ``app.api.chat``'s grounded plan — the only call site that passes it). #: It exists because the incident this phase fixes is a PROMPT gap, not a #: tool gap: nothing ever told the model that the user copies from what is #: DISPLAYED, so it retyped files out of context and every retype differed #: from the stored bytes by a character. What each sentence pins, in order: #: (1) the display contract — the tool shows the document EXACTLY as stored #: and the copy source is the display, not the prose; (2) when to call it — #: the user wants to SEE, COPY or RUN a file (the owner's quadlet / config / #: script shapes) rather than read content out of the answer; (3) the do- #: NOT-retype rule with its reason (anything the model types may differ by #: one character and the user cannot tell — the phase's reason for #: existing); (4) the placement syntax — the inline `` ```bor #: [range] ``` `` locator fence with an EMPTY body (D1/D7: a fence is a #: resolution request, the app supplies the text, and D3 keeps the model's #: body out of the DOM); (5) the honesty-about-numbers rule — line numbers #: come from ``grep``/``read``, and when unsure the range is OMITTED #: because the displayed extent is the server's (A2); (6) the failure rule #: — refuse out loud, never reconstruct from memory (the persona's radical #: honesty, D3's posture on the model side); (7) the image pointer #: (LOCKED A6: images belong to ``show_image``, never silently converted). #: ``TOOLS_SECTION`` itself stays BYTE-IDENTICAL (the sha pins in #: :mod:`tests.unit.test_prompt_lock` hold); a ``show_text=False`` build #: (the default — every pre-phase call site) is byte-identical to the #: pre-task output, and a ``show_image=True, show_text=False`` build #: reproduces the phase-124 bytes EXACTLY. The ```` marker the E2E #: mock keys on is untouched either way — both clauses ride on its #: presence, not its wording. The LOW/deflection prompt never carries the #: clause: a deflected turn has no tools, and phase 182's D9 forbids #: resolving a locator on a turn that just disclaimed the knowledge base. SHOW_TEXT_CLAUSE: str = ( "`show_text` displays part of a document to the user EXACTLY as stored " "— the user copies from what is displayed, not from your prose. Call " "it whenever the user wants to SEE, COPY or RUN a file — a quadlet, a " "config, a script, a unit file — instead of writing the content into " "your answer yourself. After calling it, do NOT repeat the displayed " "lines in your answer — describe them, explain them, point at them. " "Anything you type yourself may differ from the file by a character, " "and the user cannot tell. To put a displayed block exactly where you " "want it in your answer, write a locator line on its own: three " "backticks followed by `bor`, a space, the combined `source/path`, and " "optionally the line range (` 12-40`). Leave its body EMPTY — the app " "replaces it with the exact stored text. Never type the file's content " "inside a `bor` block. Never guess line numbers — use the numbers " "`grep` and `read` reported. When you are not sure which lines, OMIT " "the range and display the whole document: the numbers you display are " "the server's, never yours. If `show_text` refuses, say what you could " "not display — do not reconstruct it from memory. For an image use " "`show_image` instead." ) def history_to_messages( history: Sequence[HistoryTurn], settings: Settings, ) -> list[dict[str, Any]]: """Client-provided chat history → model messages (phase 74, TODO L4). The ``POST /api/chat`` ``history`` (the client's prior turns, oldest first) becomes the message block that sits between the system prompt and the current user message — so a follow-up question reaches the model together with the exchange so far, on BOTH turn branches (the deflected path and the grounded agent). Trimming (owner-locked A3, 2026-09-08): the turns are walked **newest-first** and kept while BOTH budgets hold — the turn count stays ≤ ``settings.history_max_turns`` and the cumulative chars (``len(text) + len(thinking or "")`` per turn) stay ≤ ``settings.history_max_chars``. A turn that would overflow either remaining budget is DROPPED WHOLE — never cut mid-answer — and the walk stops there, so the kept history is always the contiguous newest window (the oldest turns are the ones dropped; ``0`` on either budget yields ``[]`` — the pre-phase-74 behavior). The kept turns are returned in chronological (oldest → newest) order. Mapping (owner-locked A4, 2026-09-08): ``who="user"`` → ``{"role": "user", "content": text}``; ``who="brain"`` → ``{"role": "assistant", "content": text}`` plus ``"reasoning_content": thinking`` ONLY when *thinking* is non-empty — the preserve-thinking wire convention :mod:`app.rag.llm` already reads on the response side (``delta.reasoning_content``), which is what keeps the owner's preserve-thinking models carrying the reasoning chain forward. Pure and side-effect free (no I/O) — unit-testable in isolation. """ kept: list[HistoryTurn] = [] chars = 0 for turn in reversed(history): if len(kept) >= settings.history_max_turns: break size = len(turn.text) + len(turn.thinking or "") if chars + size > settings.history_max_chars: break kept.append(turn) chars += size messages: list[dict[str, Any]] = [] for turn in reversed(kept): if turn.who == "user": messages.append({"role": "user", "content": turn.text}) continue message: dict[str, Any] = {"role": "assistant", "content": turn.text} if turn.thinking: message["reasoning_content"] = turn.thinking messages.append(message) return messages def _base(relevance: str) -> str: if relevance not in ("HIGH", "LOW"): raise ValueError(f"relevance must be HIGH or LOW, got {relevance!r}") return PERSONA.replace("{relevance}", relevance) def build_steering_section(notes: Sequence[str], max_chars: int | None = None) -> str: """The ```` section of the system prompt (phase 15). * No notes (or only blank ones) → ``""`` — callers then build the prompt exactly as before, so a zero-note prompt is byte-identical to the pre-steering text. * Otherwise: numbered notes (in the given order — the chat turn passes them oldest-first, so #1 is the oldest note) capped at *max_chars* (default ``BOR_STEERING_MAX_CHARS``). When the budget cannot hold every note, the oldest-fitting prefix is kept and the overflow is replaced by the shared ``[…truncated…]`` marker. """ cleaned = [str(n).strip() for n in notes] cleaned = [n for n in cleaned if n] if not cleaned: return "" limit = max_chars if max_chars is not None else get_settings().steering_max_chars if limit <= 0: return "" def render(count: int) -> str: lines = [f"{i}. {note}" for i, note in enumerate(cleaned[:count], start=1)] if count < len(cleaned): lines.append(TRUNCATION_MARKER) return f"\n{_STEERING_INTRO}" + "\n".join(lines) + "\n" for count in range(len(cleaned), 0, -1): rendered = render(count) if len(rendered) <= limit: return rendered # Pathological budget: not even the empty note list fits. The section # must still respect the cap — the bare marker when it fits, else none. if len(TRUNCATION_MARKER) <= limit: return TRUNCATION_MARKER return "" def build_kb_section(overview: str, max_chars: int | None = None) -> str: """The ```` section of the system prompt (phase 31). * No outline (or only whitespace) → ``""`` — callers then build the prompt exactly as before, so a no-overview prompt is byte-identical to the pre-phase text (phase 15 convention). * Otherwise: the intro line + the stored outline, capped at *max_chars* (default ``BOR_KB_OVERVIEW_MAX_CHARS``). When the budget cannot hold the whole outline, the longest-fitting prefix is kept and the overflow is replaced by the shared ``[…truncated…]`` marker on its own line — the exact :func:`build_steering_section` pattern, including its pathological-budget handling (never exceed the cap; bare marker when even one outline character does not fit). """ text = str(overview or "").strip() if not text: return "" limit = max_chars if max_chars is not None else get_settings().kb_overview_max_chars if limit <= 0: return "" def render(cut: int) -> str: lines = [text[:cut]] if cut < len(text): lines.append(TRUNCATION_MARKER) return f"\n{_KB_INTRO}" + "\n".join(lines) + "\n" for cut in range(len(text), 0, -1): rendered = render(cut) if len(rendered) <= limit: return rendered # Pathological budget: not even one outline character fits. The # section must still respect the cap — the bare marker when it fits, # else none (steering precedent, phase 15). if len(TRUNCATION_MARKER) <= limit: return TRUNCATION_MARKER return "" #: Phase 125 (task 05, LOCKED A4) — the header line of one block of an #: enriched image doc's stored ``image_context`` (the exact shape the #: importer's reference-block writer emits, tasks 03/04): #: ``Referenced in /:`` — a block is one #: referring markdown file, and a block's line FOLLOWING its header is #: always its first ``- Alt text: "..."`` bullet. A look-alike line in #: the context body without that bullet is spurious and dropped #: (graceful degradation, A4). _IMAGE_CTX_HEADER_RE = re.compile(r"^Referenced in (?P.+):$") #: Phase 137 (task 04, ASSUMPTION A6) — the header line of the PDF #: provenance block of a PDF-embedded image doc's stored #: ``image_context`` (the exact shape the importer's document-block #: writer emits, phase 137 tasks 02/03 — written FIRST, before any #: markdown reference blocks, ASSUMPTION A2): #: ``Extracted from / (page ):`` — ``loc`` is the #: ``/`` locator (``.+`` greedy to the LAST #: `` (page N):`` — the pdf path may itself contain spaces); the line #: FOLLOWING the header is page-text body (or the next block's header) #: and is never part of the clause — the header line alone IS the #: clause (no bullet is required or expected). A look-alike line inside #: the page-text body emits a clause the same way — accepted: the #: stored value is the context the importer wrote, not free model text #: (the ``Referenced in`` twin IS guarded by its bullet requirement). _PDF_CTX_HEADER_RE = re.compile(r"^Extracted from (?P.+) \(page (?P\d+)\):$") #: The ``- Alt text: "..."`` bullet of the same block. The greedy alt #: field (the block writer does not escape quotes) is captured between #: the opening quote (the pattern's literal) and the CLOSING quote #: (the pattern's trailing ``"$``) — an alt with inner ``"`` survives #: intact (the greedy walk keeps the inner quote, the final one is the #: delimiter; an empty alt matches as the empty group). _IMAGE_CTX_ALT_RE = re.compile(r'^- Alt text: "(?P.*)"$') def _image_ctx_clause(ref: str, alt: str | None) -> str: """One clause of the retrieval context line (phase 125, task 05): the referring markdown's stored identity (``/`` exactly as the block header carries it) plus the reference's alt text quoted after a colon — the reference's purpose in one phrase. A reference with no alt (``![](path)``) renders the bare identity (nothing to quote). """ if alt: return f'{ref}: "{alt}"' return ref def _extract_context_line(image_context: str) -> str | None: """The ONE-LINE rendering of an enriched image doc's stored ``image_context`` (phase 125, task 05, LOCKED A4; phase 137, task 04, ASSUMPTION A6) — or ``None`` when nothing in the value parses as a reference block OR a provenance block (graceful degradation: the ```` block then shows only the description, exactly pre-phase — the value never raises into the prompt build). The stored value is blank-line-joined blocks of two kinds: * the phase-137 document (provenance) block (tasks 02/03) — written FIRST for a PDF-embedded image (ASSUMPTION A2: provenance before purpose); ONE header line, no bullet:: Extracted from / (page ): → the clause ``From / (page )`` — the header line alone (the page-text body is skipped as context body; a spurious ``Extracted from … (page N):`` line inside the page text emits a clause the same way — it is the stored context the importer wrote, not free model text, accepted). * the phase-125 reference block (tasks 03/04) — one per referring markdown file; the line FOLLOWING the header is always its first ``- Alt text: "..."`` bullet (a look-alike header without its bullet is spurious and dropped):: Referenced in /: - Alt text: "" Context: - Alt text: "" → the clause ``Referenced in : ""`` (identity + the FIRST reference's alt; the ``Context:`` body is never rendered — LOCKED A3, the paragraphs already reached the vision model at enrichment time, so the line stays short enough not to bloat the prompt). Clauses join with ``"; "`` IN STORED ORDER: * BOTH kinds (the phase-137 combined shape — document block first, then the markdown blocks) render each clause with its own prefix:: From / (page N); Referenced in docs/a.md: "alt" * a MARKDOWN-ONLY value keeps the phase-125 shape byte-identically (the regression pin — ONE ``Referenced in `` prefix for the joined referenced clauses):: Referenced in docs/a.md: "alt a"; docs/b.md A value with NO parsable header of either kind parses to ``None`` (pre-phase rows, unparseable text). The markdown doc's ``IMAGE_CONTEXT:`` pending record (phase 125, task 04) carries no parsable header of either kind and parses to ``None`` — and it rides a TEXT doc, which :func:`_document_body` excludes from the line via the ``is_image`` guard anyway. """ clauses: list[tuple[str, str]] = [] # (kind, text), in stored order: a ``"pdf"`` clause's text already # carries its ``From …`` prefix; a ``"ref"`` clause's text is the # BARE ``: ""`` core — the prefix is applied at the join, # below (a markdown-only value keeps ONE prefix, byte-identically). ref: str | None = None # the current block's "source/markdown_path" alt: str | None = None # the current block's first bullet's alt text expecting_alt = False # the line after a "Referenced in" header must be its bullet for line in image_context.splitlines(): header = _IMAGE_CTX_HEADER_RE.match(line) if header is not None: if ref is not None and not expecting_alt: clauses.append(("ref", _image_ctx_clause(ref, alt))) ref, alt = header.group("ref"), None expecting_alt = True continue pdf_header = _PDF_CTX_HEADER_RE.match(line) if pdf_header is not None: # A provenance header requires NO following bullet — the # header line alone is the clause (the next line is # page-text body or the next header; neither changes it). # A pending reference block's bullet never came → dropped # by the same spurious-header rule; a complete one is # kept, in stored order. if ref is not None and not expecting_alt: clauses.append(("ref", _image_ctx_clause(ref, alt))) clauses.append( ("pdf", f"From {pdf_header.group('loc')} (page {pdf_header.group('page')})") ) ref, alt, expecting_alt = None, None, False continue alt_match = _IMAGE_CTX_ALT_RE.match(line) if ref is not None else None if expecting_alt: if alt_match is None: # Spurious header (a look-alike line in context text): # its bullet never came — drop the block. ref, alt, expecting_alt = None, None, False else: alt = alt_match.group("alt") expecting_alt = False continue # Any other line is context body — a provenance block's page # text, a reference block's Context: paragraphs, or a second # bullet of the same block — and the one-line rendering is # unaffected (no branch needed: with a live block and no # pending bullet, ``alt`` is never ``None``). if ref is not None and not expecting_alt: clauses.append(("ref", _image_ctx_clause(ref, alt))) if not clauses: return None if all(kind == "ref" for kind, _ in clauses): # Markdown-only: the phase-125 shape, byte-identically (the # regression pin) — ONE ``Referenced in `` prefix for the # joined clauses. return "Referenced in " + "; ".join(text for _, text in clauses) # A provenance clause is present: every clause carries its own # prefix (the ``pdf`` ones already do; the referenced clauses gain # theirs) — the stored order (document block first, ASSUMPTION A2) # is the render order. return "; ".join( text if kind == "pdf" else f"Referenced in {text}" for kind, text in clauses ) def _attr(value: str) -> str: """XML-escape one attribute value (phase 191, D4) — ``&``/``<``/``>`` and the double quote (``"``), the delimiter the block uses. Applied ONLY to the two additive label attributes (``name`` / ``label``): they hold free-form OWNER text where a ``"`` is realistic and must not break the attribute. The pre-existing identity attributes (``source`` / ``path`` / ``title`` / ``date``) deliberately keep their unescaped pre-phase bytes — escaping them here would move every label-less block, which the phase's byte-identity criterion forbids. """ return escape(value, {'"': """}) def _source_label_attrs( doc: Document, names: Mapping[str, SourceLabelView] | None ) -> str: """The additive `` name="…"`` / `` label="…"`` fragment of one ```` block (phase 191, D4) — the empty string unless the source has a label row with that field set. ``None`` / an empty map / no entry for ``doc.source`` / a row with neither field set all render ``""`` — the block is then byte-identical to pre-phase, and a label-only row adds exactly the one ``label`` attribute (an unset field adds NOTHING, it is never an empty attribute). """ if not names: return "" view = names.get(doc.source) if view is None: return "" attrs = "" if view.display_name: attrs += f' name="{_attr(view.display_name)}"' if view.label: attrs += f' label="{_attr(view.label)}"' return attrs def _image_context_line(doc: Document) -> str | None: """The phase-125 context line for one suggested document (task 05, LOCKED A4) — ``None`` in the common case (EVERY text doc, and every image doc without a parsable ``image_context`` — standalone / pre-phase rows: the byte-identical path) and the one-line :func:`_extract_context_line` rendering otherwise (``is_image`` docs with a non-NULL, parsable ``image_context`` only).""" if not doc.is_image or not doc.image_context: return None return _extract_context_line(doc.image_context) def _document_body(doc: Document) -> str: """The body of one ```` suggestion block (phase 118, LOCKED A6): the document's stored summary (stripped) — NEVER the full content. Defensive fallback ONLY when the summary is missing (``None`` or whitespace — a fail-soft import miss, LOCKED A5): the first ``suggestion_preview_chars`` characters of the content plus the shared :data:`TRUNCATION_MARKER` on its own line — the settings read happens on this fallback path ONLY (a prompt built from summary-bearing docs makes no settings call for the cap). Content at or under the cap rides whole, unmarked (nothing was cut). No LLM call at chat time — the preview is deterministic. Phase 125 (task 05, LOCKED A4; phase 137, task 04, A6): an ENRICHED image doc (``is_image`` with a non-NULL ``image_context`` that parses — see :func:`_image_context_line`) leads its body with the ONE short context line — ``Referenced in /: ""`` (the phase-125 reference shape) or ``From / (page N)`` (the phase-137 provenance shape) — before the description, so the LLM knows WHY the image exists in the documentation, not only what it shows. Every other doc is byte-identical to the pre-phase body: every text doc (its ``image_context`` may carry the task-04 pending record, which never renders) and every non-enriched image doc (NULL ``image_context`` — the phase's byte-identity criterion). """ summary = (doc.summary or "").strip() if summary: body = summary else: limit = get_settings().suggestion_preview_chars content = doc.content body = ( content[:limit] + "\n" + TRUNCATION_MARKER if len(content) > limit else content ) context_line = _image_context_line(doc) if context_line is not None: body = context_line + "\n\n" + body return body def build_high_prompt( documents: Sequence[Document], notes: Sequence[str] | None = None, kb_overview: str | None = None, folder_lines: Sequence[str] = (), show_image: bool = False, show_text: bool = False, names: Mapping[str, SourceLabelView] | None = None, ) -> str: """Grounded turn: locked persona (+ steering, + KB overview) + the top-ranked documents' SUMMARY blocks + the ```` instructions (phase 37; phase 70: the harness-aligned ``ls`` / ``read`` / ``grep`` shapes; phase 72: the document-identity contract up front; phase 118, LOCKED A6 re-revising A7: the ```` section seeds SUMMARIES, never full texts — full text enters the context only through the capped ``read`` tool; phase 119, LOCKED A4: the suggested-folder context lines, below; phase 124, task 03, LOCKED A5: the optional :data:`SHOW_IMAGE_CLAUSE`, below; phase 182, task 04, LOCKED A1–A3: the optional :data:`SHOW_TEXT_CLAUSE`, below). Section order: ```` → ```` → ```` → ```` → ````; empty steering/overview omit their section. ```` is always present in the HIGH prompt (the round cap — not the prompt — decides whether the tools are actually offered to the model, see :mod:`app.rag.agent`). *show_image* (phase 124, task 03, LOCKED A5): when true (the caller passes the ``BOR_ANSWER_IMAGES`` toggle — ``app.api.chat``'s grounded plan is the only call site that does), the :data:`SHOW_IMAGE_CLAUSE` rides INSIDE the ```` section — appended after the existing three-tool copy and before the closing ```` tag, so the ```` marker the E2E mock keys on is untouched and the clause reaches the model together with the tool teaching. The section order is otherwise UNCHANGED and :data:`TOOLS_SECTION` itself is never edited: the built section is ``TOOLS_SECTION``'s text minus its final ```` + the clause + the closing tag. The default ``False`` keeps EVERY pre-existing call byte-identical (the pre-task pins in :mod:`tests.unit.test_prompts` / :mod:`tests.unit.test_prompt_lock` hold untouched). The deflection prompt (:func:`build_deflect_prompt`) never carries the clause — a deflected turn has no tools and never shows an image. *show_text* (phase 182, task 04, LOCKED A1–A3): when true (the caller passes the ``BOR_ANSWER_TEXT`` master switch — the same single grounded call site), the :data:`SHOW_TEXT_CLAUSE` rides INSIDE the same ```` section, AFTER :data:`SHOW_IMAGE_CLAUSE` when that is present too and always before the closing tag. The two flags compose independently through one mechanism (the built section is ``TOOLS_SECTION`` minus its final ```` + the present clauses, blank-line separated, + the closing tag): ``show_image`` only reproduces the phase-124 bytes exactly, ``show_text`` only is the three-tool copy plus the text clause, both is the image clause then the text clause, neither is byte-identical to pre-phase. The default ``False`` keeps every pre-existing call byte-identical. The deflection prompt NEVER carries it — a deflected turn has no tools, and phase 182's D9 forbids resolving a ``bor`` locator on a turn whose answer just disclaimed the knowledge base. When at least one block is present, the section leads with the start-here :data:`SUGGEST_INTRO` line (before the first block — the phase-15 ``_STEERING_INTRO`` / phase-31 ``_KB_INTRO`` precedent): the blocks are the summaries of the top-ranked documents, ``read`` adds the full text, and the answer cites only the documents it read (or the suggested document it answered from without reading — the phase-119 cite discipline, LOCKED A5; the phase-118 "cite what you used, by path" sentence is retired). This is NOT the reverted phase-72 in-context reminder (that copy taught the seed documents as already-read context; this one teaches the summary-as-starting-point contract the A6 re-revision requires). *folder_lines* (phase 119, D3, LOCKED A4): the suggested-folder context lines (``app.rag.agent.suggested_folder_lines``) — the direct children of each suggested document's parent folder, so the model ``read``s the right file in round 1 instead of walking the ``ls`` drill-downs. When blocks are present AND the lines are non-empty, the ```` body is ``SUGGEST_INTRO``, then the folder lines (each on its own line, immediately after the intro line), a blank line, then the first ```` block. They are PLAIN lines — no new markup/tag (the E2E mock keys off the ```` marker and the LAST block's tail, so the lines must never land after a summary). Empty *folder_lines* ⇒ the phase-118 build is byte-identical (pinned in ``tests/unit/test_prompts.py``). Each ```` block carries the identity attributes ``source`` / ``path`` / ``title`` — and, since phase 106 (D5), ``date`` (the row's ``created_at`` UTC date part, ``YYYY-MM-DD``, APPENDED after ``title`` — the only position; always present, ``created_at`` is NOT NULL) — plus the document's summary (:func:`_document_body`; the NULL/whitespace-summary preview fallback is LOCKED A5; an enriched image doc's summary leads with the phase-125 one-line context line, LOCKED A4). *names* (phase 191, D4): the source-name → :class:`SourceLabelView` map of the owner's label rows (the SAME single ``all_labels`` read that feeds the task-04 selection boost — one query per turn). When a doc's source has a row, the block head gains `` name="{display_name}"`` immediately after ``source="…"`` and `` label="{label}"`` after that (the fixed order ``source`` → ``name?`` → ``label?`` → ``path`` → ``title`` → ``date``); the two values are XML-escaped (:func:`_attr`), the identity attributes are not. An unset field adds NOTHING, so a label-only row gains exactly the one attribute, and ``None`` / ``{}`` / no entry for the source keeps the block byte-identical to pre-phase (the default, and the kill pin in :mod:`tests.unit.test_prompts`). This is surfacing only — the alias contract (D1) keeps ``source`` / ``path`` as the document identity the tools, citations and locators use. """ # Phase 106 (D5): every document the model sees carries its # creation date — the block's ``date`` attribute (the row's # ``created_at`` UTC date part, appended after ``title`` — the # only position; always present, ``created_at`` is NOT NULL). # Phase 118 (A6): the block body is the summary — the full content # stays out of the prompt (the capped ``read`` tool is the only # full-text path). blocks = [ f'\n' f"{_document_body(doc)}\n" "" for doc in documents ] body = "\n\n".join(blocks) if blocks else ( "(no documents matched — do not invent specifics)" ) if blocks: if folder_lines: # Phase 119 (D3, LOCKED A4): the folder lines ride immediately # after the intro line, one per line, a blank line, then the # first block — plain lines, no new markup (the # block markup AND body stay byte-identical around them). body = SUGGEST_INTRO + "\n" + "\n".join(folder_lines) + "\n\n" + body else: body = SUGGEST_INTRO + "\n\n" + body prompt = _base("HIGH") for part in (build_kb_section(kb_overview or ""), build_steering_section(notes or [])): if part: prompt += "\n" + part # Phase 124 (task 03, LOCKED A5) / phase 182 (task 04, LOCKED A1): the # display-tool clauses ride inside — TOOLS_SECTION's text minus # its final , the clauses offered for this build (each on its # own line, blank-line separated, after the existing three-tool copy), # then the closing tag. TOOLS_SECTION itself is unedited, the # no-clause path keeps the exact pre-task concatenation, and a # show_image-only build reproduces the phase-124 bytes exactly. clauses: list[str] = [] if show_image: clauses.append(SHOW_IMAGE_CLAUSE) if show_text: clauses.append(SHOW_TEXT_CLAUSE) if clauses: tools_section = ( TOOLS_SECTION[: -len("")] + "\n\n".join(clauses) + "\n" ) else: tools_section = TOOLS_SECTION return prompt + "\n\n" + body + "\n\n" + tools_section def build_deflect_prompt( titles: Sequence[str], notes: Sequence[str] | None = None, kb_overview: str | None = None, ) -> str: """Deflection turn: weak-hit titles only (no document content). Section order (phase 31): ```` → ```` → ```` → ``DEFLECT_MODE`` body; empty steering/overview omit their section, keeping the prompt byte-identical to the pre-phase text. The body ends with the phase-71 plain-text line (owner- permitted 2026-09-03 — the LOW prompt's only change): a deflected turn offers no tools, so any tool markup there is always wrong. """ weak = "\n".join(f"- {t}" for t in titles) if titles else "(nothing close at all)" mid = "\n".join( part for part in (build_kb_section(kb_overview or ""), build_steering_section(notes or [])) if part ) gap = f"\n{mid}\n" if mid else "\n" return ( _base("LOW") + gap + "DEFLECT_MODE: retrieval was weak — the titles below are the closest " "your notes come to the question. They are titles only; do not pretend " "they answer it. Use them to propose 2-3 alternative questions.\n" # Phase 71 (owner-permitted 2026-09-03): the one plain-text line # — prevention at the prompt. The E2E mock keys on the # DEFLECT_MODE marker's presence, so appending is safe. "Reply in plain text only — you have no tools in this mode.\n" + weak )