brain-of-reese/app/rag/prompts.py

"""Locked system-prompt builder (PLAN §6).

The persona + HONESTY GATE text is **locked verbatim** — change it through
the plan, not here. (PLAN §6 revision, 2026-08-22: the owner's working-tree
persona edits are preserved — no mandated ``"you've got this"`` tagline and
no mandated deflection opening; the honesty gate itself is unchanged.)

Two modes:

* ``HIGH`` — grounded turn: the top-ranked documents' SUMMARIES under
  ``<documents>`` (phase 118, LOCKED A6 re-revising A7 — the "start here"
  suggestion seeding: each block is the document's stored summary, never
  the full content; the LLM extends its context by ``read``-ing a
  document's full text through the capped ``read`` tool, the ONLY full-
  text path. A NULL/blank summary — a fail-soft import miss — falls back
  to a ``suggestion_preview_chars`` content preview + the shared
  ``[…truncated…]`` marker; no LLM call at chat time, LOCKED A5).
* ``LOW``  — deflection turn: weak-hit *titles only* plus the
  ``DEFLECT_MODE`` marker (the E2E mock LLM keys on that marker).

Steering (phase 15): when the owner has stored tuning notes, both modes
carry a ``<tuning>`` section between ``<relevance>…</relevance>`` and the
mode body. With zero notes the prompt is byte-identical to the
pre-steering text.

KB overview (phase 31): when the single ``kb_overview`` row holds a
lite-generated outline of the knowledge base, both modes carry a
``<knowledge_base>`` section between ``<relevance>…</relevance>`` and
the ``<tuning>`` section (order: ``<relevance>`` →
``<knowledge_base>`` → ``<tuning>`` → mode body) — the agent knows
roughly what the KB contains before retrieval. With an empty row the
prompt is byte-identical to the pre-phase text.

Agent tools (phase 37; phase 70: the copy teaches the harness-aligned
``ls`` / ``read`` / ``grep`` shapes; phase 72: the copy states the
document-identity contract up front — ``ls``'s ``path`` is a *source
name*, not a directory or file path, and ``read``/``grep`` take the
combined ``source/path`` string *including the source name* (a bare
document path will not resolve) — the same two things the phase-72
teaching refusals in :mod:`app.rag.agent` re-state after the fact, so
the model carries the contract before it calls a tool; phase 94,
task 03: the ``ls`` clause is rewritten to the drill-down tree
contract — no ``path``: the synced sources with counts + summaries, a
source name: its top-level folders + files, a ``source/folder`` path:
one level deeper — a listing shows only that level's subfolders + own
files (never the whole KB in one call), the folder summaries say
what's in a folder before drilling, and ``grep`` stays the locator for
finding one document without listing; the ``read``/``grep`` clauses
and the discipline rules are byte-identical): the **HIGH**
prompt only carries a ``<tools>`` section after the ``<documents>``
body — the grounded turn may extend its context through the three
server-side tools (round-capped, see :mod:`app.rag.agent`; the cap is
the bound and this section does not re-state it, phase 45). The
LOW/deflection prompt never carries it (phase 71: the LOW prompt's
only addition is the plain-text line below — it still has no
``<tools>`` section).

Deflection plain-text line (phase 71, owner-permitted 2026-09-03):
the otherwise-locked ``LOW`` prompt gains exactly one instruction
line — "Reply in plain text only — you have no tools in this mode."
— appended to the ``DEFLECT_MODE`` body: a deflected turn offers no
tools, so any tool markup there is always wrong, and the line closes
the door at the prompt (the deterministic filter + one bounded
recovery in :mod:`app.rag.scaffolding` / :mod:`app.rag.agent` is the
backstop). The ``DEFLECT_MODE`` marker and everything else in the
prompt stay put — the E2E mock LLM keys on the marker's *presence*,
not the wording, so that contract is unchanged.

Disclosed-answer contract (phase 112, task 03; owner decision
2026-09-14, roadmap confirmation — option (iii) of the three the
2026-09-15 interactive deflection test raised): that test found the
HONESTY GATE's compliance is **stochastic** across runs when
misleading context is injected — the "What is the capital of
Mongolia?" question had two *irrelevant* docs promoted into the HIGH
prompt by a weak single-token FTS hit (the pre-phase A8 rule: any
``fts_hits > 0`` flipped HIGH), and run 1 answered parametrically —
"Ulaanbaatar" — *with an explicit disclosure*, while the identical
one-tap re-run produced a clean, textbook deflection. The owner's
decision: the disclosed general-knowledge answer is treated as
**acceptable and documented** — a small local model cannot be relied
on to obey Rules 1/3 100% when handed misleading context, so this
prompt text stays byte-identical (LOCKED verbatim; options (i) tighten
the copy / (ii) amend this locked prompt via the plan to explicitly
permit disclosed general-knowledge answers remain open to a future
owner decision). The deterministic lever is the honesty gate itself
(A8 revised 2026-09-14: an FTS hit flips HIGH only when
``best_cosine >= lexical_support_floor``, so the misleading docs are
no longer injected — see :mod:`app.api.chat`).
"""
from __future__ import annotations

import re
from collections.abc import Sequence
from typing import Any

from app.config import Settings, get_settings
from app.models import Document
from app.rag.retriever import TRUNCATION_MARKER
from app.schemas import HistoryTurn

#: PLAN §6 verbatim (line wrapping included); ``{relevance}`` is filled by
#: :func:`_base`.
PERSONA: str = (
    'You are "Brain of Reese" — the digital brain of Reese, a self-hoster and\n'
    "homelab tinkerer. Personality: chippy, upbeat, warm, and genuinely\n"
    "optimistic about the user's ability to do things.\n"
    "\n"
    "Rules:\n"
    "1. Answer ONLY from the provided document context. Cite which document(s)\n"
    "   you used, by path.\n"
    "2. Be concrete: names, versions, ports, hosts, schedules — the specifics in\n"
    "   the docs are the value.\n"
    '3. HONESTY GATE: if <relevance> is "LOW", you must NOT pretend to know.\n'
    "   Offer 2-3 alternative questions about things you DO have notes on.\n"
    "4. Never invent facts, hosts, or steps that are not in the context.\n"
    "5. Keep answers tight: short paragraphs, bullets where helpful.\n"
    "\n"
    "<relevance>{relevance}</relevance>"
)

#: One-line intro of the ``<tuning>`` section (phase 15): the owner's notes
#: steer the answer and win over the defaults when they conflict.
_STEERING_INTRO = (
    "The owner of this brain asked you to steer your answers as follows. "
    "Where these instructions conflict with the defaults above, follow the owner:\n"
)

#: One-line intro of the ``<knowledge_base>`` section (phase 31): the
#: lite-generated outline is the agent's a-priori picture of the KB.
_KB_INTRO = (
    "The basic categories of everything in this knowledge base "
    "(generated at import time):\n"
)

#: The ``<documents>`` section's start-here intro (phase 118, task 03 —
#: the owner directive, TODO L3; phase 119, task 04, LOCKED A5 — the
#: final sentence re-cut to the cite discipline): the seeded blocks are
#: the SUMMARIES of the top-ranked documents for the question, opt-in
#: starting points — start here if one seems right; ``read`` the
#: document's combined ``source/path`` to add its FULL content before
#: answering specifics (the full text is NOT in the prompt until you
#: read it); cite ONLY the documents you read (or the suggested document
#: you answered from without reading it) — never one you neither read
#: nor used. The discipline sentence closes the phase-119 live
#: confabulation — the model's "Docs used:" line cited a file it never
#: read, confabulated from the suggestion blocks sitting in context —
#: and retires the phase-118 "cite the document(s) you used, by path"
#: sentence. Rendered BEFORE the first block, only when at least one
#: block is present. The ``<documents>`` tag and the per-block
#: ``<document>`` markup stay byte-stable around it (the E2E mock's
#: block parser and the ``read``-tool copy key off both).
SUGGEST_INTRO = (
    "The blocks below are the summaries of the top-ranked documents for "
    "your question — start here if one seems right to you: call `read` "
    "with that document's combined `source/path` to add its full content "
    "before answering specifics (its full text is not in the prompt until "
    "you read it). Cite only the document(s) you read — or, if you "
    "answered from a suggested summary without reading it, cite that "
    "suggested document — never a document you neither read nor used."
)

#: The ``<tools>`` instructions section — **HIGH prompt only** (phase 37,
#: task 03; phase 70: the copy is rewritten for the harness-aligned
#: ``ls`` / ``read`` / ``grep`` shapes, names/args exactly as the
#: ``AGENT_TOOLS`` schemas in :mod:`app.rag.agent`; phase 72: the copy
#: states the document-identity contract UP FRONT — ``ls``'s optional
#: ``path`` is a *source name* (not a directory or file path) and
#: ``read``/``grep`` take the combined ``source/path`` string *including
#: the source name* (a bare document path will not resolve) — the same
#: two things the phase-72 teaching refusals re-state after the fact;
#: phase 94, task 03 — the owner-permitted tool-surface revision,
#: recorded in the phase 94 overview: the ``ls`` clause is rewritten
#: to the drill-down tree contract — no ``path``: the synced sources
#: with counts + summaries, a source name: its top-level folders +
#: files, a ``source/folder`` path: one level deeper — a listing shows
#: only that level's subfolders + own files, never the whole KB in one
#: call, the folder summaries say what's in a folder before drilling,
#: and ``grep`` stays the locator for finding one document without
#: listing; the ``read``/``grep`` clauses and the discipline rules are
#: byte-identical):
#: a grounded turn may extend its context through the three server-side
#: tools (round cap: ``BOR_AGENT_MAX_ROUNDS`` — the cap is the bound and
#: this section does not re-state it, phase 45). Appended after the mode
#: body (``<documents>``), so the instructions are the last thing the
#: model reads. The LOW/deflection prompt never carries it — a
#: deflection has no grounded context to extend (phase 71: the LOW
#: prompt's only addition is the plain-text line in
#: :func:`build_deflect_prompt`). The E2E mock keys off the ``<tools>``
#: marker's *presence*, not this wording. Task 05 (the live gate's
#: iteration loop) keeps the baseline and carries the discipline rules
#: the live telemetry motivated, refined across the task-05 re-runs of
#: 2026-09-03/04 (run 1: 7/16 executed, 44% — the ``lite`` model obeyed
#: the user's "open it / read it" and ``read`` seed-context documents,
#: then repeated the refused call; run 2: 8/18, 44% — the repeat is
#: gone, but a grep turn batched two calls per reply and the harness
#: runs only the first of a batch): the ``read`` clause carries the
#: phase-118 summary-seed contract (the ``<documents>`` section holds
#: SUMMARIES — a suggested document's full text is not in the prompt
#: until ``read`` adds it; do not re-read a document you have already
#: read — answer from the text already in the prompt; if the user
#: asks to open or read a suggested document, ``read`` it — that is
#: the point of the section — the phase-72 do-not-read rule and the
#: reverted ``<document path="...">`` anchor are retired with the
#: full-text seeds);
#: the one-call rule names the consequence (a batched second call is
#: discarded — runs only the first); the never-repeat rule says why
#: (the refusal already told you the correct form); the ``grep`` clause
#: leads with "for a normal search pass only ``pattern``" (the gate's
#: live runs showed the model scoping ``grep`` with an ``ls``-style
#: source name — the incident shape, but on grep). The behavioral
#: contract lives in the ``AGENT_TOOLS`` descriptions as well (the most
#: local text at call time): ``read`` adds a document's FULL text by
#: its combined ``source/path`` identity — the ``<documents>``
#: summaries are the starting points, not the content; ``grep`` with
#: only ``pattern`` searches the whole knowledge base, and a source
#: name is not a document.
TOOLS_SECTION: str = (
    "<tools>\n"
    "You may extend your context with three tools. `ls` lists the "
    "knowledge base as a tree, one level at a time: with no `path` it "
    "lists every synced source with its document count and a summary "
    "of its contents; with a source name (e.g. 'homelab') it lists "
    "that source's top-level folders and files; with a `source/folder` "
    "path it drills one level deeper. A listing shows only that "
    "level's subfolders and its own files — never the whole knowledge "
    "base in one call — and each folder line's summary says what the "
    "folder contains before you drill into it. File lines are "
    "`source: X | path: Y | title: Z`; to find one specific document "
    "without listing, use `grep`. `read` pulls in one document by its "
    "combined `source/path` string, exactly as shown in the `ls` "
    "output — including the source name — or in the <documents> "
    "summary blocks — adding its full content to your context. The "
    "<documents> section holds SUMMARIES — the full text of a "
    "suggested document is not in your prompt until you `read` it. Do "
    "not re-read a document you have already read — its full text is "
    "already in your prompt; answer directly from it. If the user "
    "asks you to open or read a suggested document, `read` it — that "
    "is the point of the section. For `read`, a bare document path "
    "(without the source name) will not resolve. Very large documents "
    "are capped: a "
    "cut read returns the first part plus a TRUNCATED notice — the "
    "document did not end where it stopped; use `grep` (pattern) to "
    "find the rest, it searches the whole document. `grep` locates an exact string "
    "(case-insensitive) in the indexed documents and returns up to 20 "
    "matching `source/path:line: text` lines — a locator, not a "
    "context-adder: read the winner with `read`. A grep pattern is a "
    "plain substring, NEVER a regex — '.*' and '\\.' are literal text "
    "there; if such a pattern returns no matches, retry with the plain "
    "text you expect to see. For a normal search pass only `pattern` — "
    "its optional `path` argument "
    "limits the search to one document you already know, by the same "
    "combined `source/path` string; never a source name — a bare "
    "document path (without the source name) will not resolve there "
    "either. Make exactly one tool call per reply — a reply carrying "
    "two tool calls runs only the first, the second is discarded — and "
    "wait for the result before the next call. Never repeat a call that "
    "was refused or already succeeded — the refusal already told you "
    "the correct form. Answer as soon as you have what you need.\n"
    "</tools>"
)

#: The ``show_image`` contract clause (phase 124, task 03, LOCKED A5) —
#: appended INSIDE the ``<tools>`` section of the HIGH prompt, after the
#: existing three-tool copy and before the closing tag, ONLY when
#: :func:`build_high_prompt` is called with ``show_image=True`` (the
#: ``BOR_ANSWER_IMAGES`` toggle, via ``app.api.chat``). It teaches the
#: same relevance gate as the ``show_image`` tool description in
#: :mod:`app.rag.agent` (LOCKED A5: the gate is the LLM's decision — no
#: server-side image scoring), plus the image-doc nature the tool
#: description cannot fully carry: the text of an image document (in a
#: ``<documents>`` block or a ``read`` result) is a description
#: GENERATED from the image, not the image's own words. The four gate
#: elements, all stated here: (1) the image-doc nature; (2) the
#: "contains the answer" bar — call it only when THAT image is what the
#: user actually wants to see, never a merely related image, never to
#: pad an answer; (3) at most ONE image per answer (also enforced
#: server-side, task 02's ``ALREADY_SHOWING`` refusal); (4) the text
#: answer is still provided in full — the image supplements, never
#: replaces. ``TOOLS_SECTION`` itself stays BYTE-IDENTICAL (the
#: pre-phase pins in :mod:`tests.unit.test_prompt_lock` hold); a
#: ``show_image=False`` build (the default — every pre-phase call site)
#: is byte-identical to the pre-task output. The ``<tools>`` marker the
#: E2E mock keys on is untouched either way — the clause rides on its
#: presence, not the wording. The LOW/deflection prompt never carries
#: the clause (a deflected turn has no tools and never shows an image —
#: the phase's locked design).
SHOW_IMAGE_CLAUSE: str = (
    "`show_image` displays a standalone image document as part of your "
    "answer. The text of an image document — in a <documents> block or "
    "a `read` result — is a description GENERATED from the image, not "
    "the image's own words. Call `show_image` only when that image is "
    "what the user actually wants to see: a diagram, screenshot, or "
    "photo of the very thing you are answering about — never for a "
    "merely related image, and never to pad an answer. Show at most one "
    "image per answer. The image supplements your text — answer fully "
    "in text either way."
)


def history_to_messages(
    history: Sequence[HistoryTurn],
    settings: Settings,
) -> list[dict[str, Any]]:
    """Client-provided chat history → model messages (phase 74, TODO L4).

    The ``POST /api/chat`` ``history`` (the client's prior turns, oldest
    first) becomes the message block that sits between the system prompt
    and the current user message — so a follow-up question reaches the
    model together with the exchange so far, on BOTH turn branches
    (the deflected path and the grounded agent).

    Trimming (owner-locked A3, 2026-09-08): the turns are walked
    **newest-first** and kept while BOTH budgets hold — the turn count
    stays ≤ ``settings.history_max_turns`` and the cumulative chars
    (``len(text) + len(thinking or "")`` per turn) stay ≤
    ``settings.history_max_chars``. A turn that would overflow either
    remaining budget is DROPPED WHOLE — never cut mid-answer — and the
    walk stops there, so the kept history is always the contiguous
    newest window (the oldest turns are the ones dropped; ``0`` on
    either budget yields ``[]`` — the pre-phase-74 behavior). The kept
    turns are returned in chronological (oldest → newest) order.

    Mapping (owner-locked A4, 2026-09-08): ``who="user"`` →
    ``{"role": "user", "content": text}``; ``who="brain"`` →
    ``{"role": "assistant", "content": text}`` plus
    ``"reasoning_content": thinking`` ONLY when *thinking* is
    non-empty — the preserve-thinking wire convention
    :mod:`app.rag.llm` already reads on the response side
    (``delta.reasoning_content``), which is what keeps the owner's
    preserve-thinking models carrying the reasoning chain forward.

    Pure and side-effect free (no I/O) — unit-testable in isolation.
    """
    kept: list[HistoryTurn] = []
    chars = 0
    for turn in reversed(history):
        if len(kept) >= settings.history_max_turns:
            break
        size = len(turn.text) + len(turn.thinking or "")
        if chars + size > settings.history_max_chars:
            break
        kept.append(turn)
        chars += size
    messages: list[dict[str, Any]] = []
    for turn in reversed(kept):
        if turn.who == "user":
            messages.append({"role": "user", "content": turn.text})
            continue
        message: dict[str, Any] = {"role": "assistant", "content": turn.text}
        if turn.thinking:
            message["reasoning_content"] = turn.thinking
        messages.append(message)
    return messages


def _base(relevance: str) -> str:
    if relevance not in ("HIGH", "LOW"):
        raise ValueError(f"relevance must be HIGH or LOW, got {relevance!r}")
    return PERSONA.replace("{relevance}", relevance)


def build_steering_section(notes: Sequence[str], max_chars: int | None = None) -> str:
    """The ``<tuning>`` section of the system prompt (phase 15).

    * No notes (or only blank ones) → ``""`` — callers then build the
      prompt exactly as before, so a zero-note prompt is byte-identical to
      the pre-steering text.
    * Otherwise: numbered notes (in the given order — the chat turn passes
      them oldest-first, so #1 is the oldest note) capped at *max_chars*
      (default ``BOR_STEERING_MAX_CHARS``). When the budget cannot hold
      every note, the oldest-fitting prefix is kept and the overflow is
      replaced by the shared ``[…truncated…]`` marker.
    """
    cleaned = [str(n).strip() for n in notes]
    cleaned = [n for n in cleaned if n]
    if not cleaned:
        return ""
    limit = max_chars if max_chars is not None else get_settings().steering_max_chars
    if limit <= 0:
        return ""

    def render(count: int) -> str:
        lines = [f"{i}. {note}" for i, note in enumerate(cleaned[:count], start=1)]
        if count < len(cleaned):
            lines.append(TRUNCATION_MARKER)
        return f"<tuning>\n{_STEERING_INTRO}" + "\n".join(lines) + "\n</tuning>"

    for count in range(len(cleaned), 0, -1):
        rendered = render(count)
        if len(rendered) <= limit:
            return rendered
    # Pathological budget: not even the empty note list fits. The section
    # must still respect the cap — the bare marker when it fits, else none.
    if len(TRUNCATION_MARKER) <= limit:
        return TRUNCATION_MARKER
    return ""


def build_kb_section(overview: str, max_chars: int | None = None) -> str:
    """The ``<knowledge_base>`` section of the system prompt (phase 31).

    * No outline (or only whitespace) → ``""`` — callers then build the
      prompt exactly as before, so a no-overview prompt is byte-identical
      to the pre-phase text (phase 15 convention).
    * Otherwise: the intro line + the stored outline, capped at
      *max_chars* (default ``BOR_KB_OVERVIEW_MAX_CHARS``). When the budget
      cannot hold the whole outline, the longest-fitting prefix is kept
      and the overflow is replaced by the shared ``[…truncated…]`` marker
      on its own line — the exact :func:`build_steering_section` pattern,
      including its pathological-budget handling (never exceed the cap;
      bare marker when even one outline character does not fit).
    """
    text = str(overview or "").strip()
    if not text:
        return ""
    limit = max_chars if max_chars is not None else get_settings().kb_overview_max_chars
    if limit <= 0:
        return ""

    def render(cut: int) -> str:
        lines = [text[:cut]]
        if cut < len(text):
            lines.append(TRUNCATION_MARKER)
        return f"<knowledge_base>\n{_KB_INTRO}" + "\n".join(lines) + "\n</knowledge_base>"

    for cut in range(len(text), 0, -1):
        rendered = render(cut)
        if len(rendered) <= limit:
            return rendered
    # Pathological budget: not even one outline character fits. The
    # section must still respect the cap — the bare marker when it fits,
    # else none (steering precedent, phase 15).
    if len(TRUNCATION_MARKER) <= limit:
        return TRUNCATION_MARKER
    return ""


#: Phase 125 (task 05, LOCKED A4) — the header line of one block of an
#: enriched image doc's stored ``image_context`` (the exact shape the
#: importer's reference-block writer emits, tasks 03/04):
#: ``Referenced in <source>/<markdown_path>:`` — a block is one
#: referring markdown file, and a block's line FOLLOWING its header is
#: always its first ``- Alt text: "..."`` bullet. A look-alike line in
#: the context body without that bullet is spurious and dropped
#: (graceful degradation, A4).
_IMAGE_CTX_HEADER_RE = re.compile(r"^Referenced in (?P<ref>.+):$")

#: Phase 137 (task 04, ASSUMPTION A6) — the header line of the PDF
#: provenance block of a PDF-embedded image doc's stored
#: ``image_context`` (the exact shape the importer's document-block
#: writer emits, phase 137 tasks 02/03 — written FIRST, before any
#: markdown reference blocks, ASSUMPTION A2):
#: ``Extracted from <source>/<pdf path> (page <N>):`` — ``loc`` is the
#: ``<source>/<pdf path>`` locator (``.+`` greedy to the LAST
#: `` (page N):`` — the pdf path may itself contain spaces); the line
#: FOLLOWING the header is page-text body (or the next block's header)
#: and is never part of the clause — the header line alone IS the
#: clause (no bullet is required or expected). A look-alike line inside
#: the page-text body emits a clause the same way — accepted: the
#: stored value is the context the importer wrote, not free model text
#: (the ``Referenced in`` twin IS guarded by its bullet requirement).
_PDF_CTX_HEADER_RE = re.compile(r"^Extracted from (?P<loc>.+) \(page (?P<page>\d+)\):$")

#: The ``- Alt text: "..."`` bullet of the same block. The greedy alt
#: field (the block writer does not escape quotes) is captured between
#: the opening quote (the pattern's literal) and the CLOSING quote
#: (the pattern's trailing ``"$``) — an alt with inner ``"`` survives
#: intact (the greedy walk keeps the inner quote, the final one is the
#: delimiter; an empty alt matches as the empty group).
_IMAGE_CTX_ALT_RE = re.compile(r'^- Alt text: "(?P<alt>.*)"$')


def _image_ctx_clause(ref: str, alt: str | None) -> str:
    """One clause of the retrieval context line (phase 125, task 05):
    the referring markdown's stored identity (``<source>/<path>``
    exactly as the block header carries it) plus the reference's alt
    text quoted after a colon — the reference's purpose in one phrase.
    A reference with no alt (``![](path)``) renders the bare identity
    (nothing to quote).
    """
    if alt:
        return f'{ref}: "{alt}"'
    return ref


def _extract_context_line(image_context: str) -> str | None:
    """The ONE-LINE rendering of an enriched image doc's stored
    ``image_context`` (phase 125, task 05, LOCKED A4; phase 137, task
    04, ASSUMPTION A6) — or ``None`` when nothing in the value parses
    as a reference block OR a provenance block (graceful degradation:
    the ``<document>`` block then shows only the description, exactly
    pre-phase — the value never raises into the prompt build).

    The stored value is blank-line-joined blocks of two kinds:

    * the phase-137 document (provenance) block (tasks 02/03) — written
      FIRST for a PDF-embedded image (ASSUMPTION A2: provenance before
      purpose); ONE header line, no bullet::

          Extracted from <source>/<pdf path> (page <N>):
          <page text — never rendered: it already reached the vision
          model at describe time>

      → the clause ``From <source>/<pdf path> (page <N>)`` — the
      header line alone (the page-text body is skipped as context
      body; a spurious ``Extracted from … (page N):`` line inside the
      page text emits a clause the same way — it is the stored context
      the importer wrote, not free model text, accepted).

    * the phase-125 reference block (tasks 03/04) — one per referring
      markdown file; the line FOLLOWING the header is always its first
      ``- Alt text: "..."`` bullet (a look-alike header without its
      bullet is spurious and dropped)::

          Referenced in <source>/<markdown_path>:
          - Alt text: "<alt_text>"
            Context: <context paragraphs — may span many lines>
          - Alt text: "<a second reference's alt>"

      → the clause ``Referenced in <ref>: "<alt>"`` (identity + the
      FIRST reference's alt; the ``Context:`` body is never rendered —
      LOCKED A3, the paragraphs already reached the vision model at
      enrichment time, so the line stays short enough not to bloat
      the prompt).

    Clauses join with ``"; "`` IN STORED ORDER:

    * BOTH kinds (the phase-137 combined shape — document block first,
      then the markdown blocks) render each clause with its own
      prefix::

          From <source>/<pdf> (page N); Referenced in docs/a.md: "alt"

    * a MARKDOWN-ONLY value keeps the phase-125 shape byte-identically
      (the regression pin — ONE ``Referenced in `` prefix for the
      joined referenced clauses)::

          Referenced in docs/a.md: "alt a"; docs/b.md

    A value with NO parsable header of either kind parses to ``None``
    (pre-phase rows, unparseable text). The markdown doc's
    ``IMAGE_CONTEXT:`` pending record (phase 125, task 04) carries no
    parsable header of either kind and parses to ``None`` — and it
    rides a TEXT doc, which :func:`_document_body` excludes from the
    line via the ``is_image`` guard anyway.
    """
    clauses: list[tuple[str, str]] = []
    # (kind, text), in stored order: a ``"pdf"`` clause's text already
    # carries its ``From …`` prefix; a ``"ref"`` clause's text is the
    # BARE ``<ref>: "<alt>"`` core — the prefix is applied at the join,
    # below (a markdown-only value keeps ONE prefix, byte-identically).
    ref: str | None = None  # the current block's "source/markdown_path"
    alt: str | None = None  # the current block's first bullet's alt text
    expecting_alt = False  # the line after a "Referenced in" header must be its bullet
    for line in image_context.splitlines():
        header = _IMAGE_CTX_HEADER_RE.match(line)
        if header is not None:
            if ref is not None and not expecting_alt:
                clauses.append(("ref", _image_ctx_clause(ref, alt)))
            ref, alt = header.group("ref"), None
            expecting_alt = True
            continue
        pdf_header = _PDF_CTX_HEADER_RE.match(line)
        if pdf_header is not None:
            # A provenance header requires NO following bullet — the
            # header line alone is the clause (the next line is
            # page-text body or the next header; neither changes it).
            # A pending reference block's bullet never came → dropped
            # by the same spurious-header rule; a complete one is
            # kept, in stored order.
            if ref is not None and not expecting_alt:
                clauses.append(("ref", _image_ctx_clause(ref, alt)))
            clauses.append(
                ("pdf", f"From {pdf_header.group('loc')} (page {pdf_header.group('page')})")
            )
            ref, alt, expecting_alt = None, None, False
            continue
        alt_match = _IMAGE_CTX_ALT_RE.match(line) if ref is not None else None
        if expecting_alt:
            if alt_match is None:
                # Spurious header (a look-alike line in context text):
                # its bullet never came — drop the block.
                ref, alt, expecting_alt = None, None, False
            else:
                alt = alt_match.group("alt")
                expecting_alt = False
            continue
        # Any other line is context body — a provenance block's page
        # text, a reference block's Context: paragraphs, or a second
        # bullet of the same block — and the one-line rendering is
        # unaffected (no branch needed: with a live block and no
        # pending bullet, ``alt`` is never ``None``).
    if ref is not None and not expecting_alt:
        clauses.append(("ref", _image_ctx_clause(ref, alt)))
    if not clauses:
        return None
    if all(kind == "ref" for kind, _ in clauses):
        # Markdown-only: the phase-125 shape, byte-identically (the
        # regression pin) — ONE ``Referenced in `` prefix for the
        # joined clauses.
        return "Referenced in " + "; ".join(text for _, text in clauses)
    # A provenance clause is present: every clause carries its own
    # prefix (the ``pdf`` ones already do; the referenced clauses gain
    # theirs) — the stored order (document block first, ASSUMPTION A2)
    # is the render order.
    return "; ".join(
        text if kind == "pdf" else f"Referenced in {text}" for kind, text in clauses
    )


def _image_context_line(doc: Document) -> str | None:
    """The phase-125 context line for one suggested document (task 05,
    LOCKED A4) — ``None`` in the common case (EVERY text doc, and every
    image doc without a parsable ``image_context`` — standalone /
    pre-phase rows: the byte-identical path) and the one-line
    :func:`_extract_context_line` rendering otherwise (``is_image``
    docs with a non-NULL, parsable ``image_context`` only)."""
    if not doc.is_image or not doc.image_context:
        return None
    return _extract_context_line(doc.image_context)


def _document_body(doc: Document) -> str:
    """The body of one ``<document>`` suggestion block (phase 118, LOCKED
    A6): the document's stored summary (stripped) — NEVER the full
    content.

    Defensive fallback ONLY when the summary is missing (``None`` or
    whitespace — a fail-soft import miss, LOCKED A5): the first
    ``suggestion_preview_chars`` characters of the content plus the
    shared :data:`TRUNCATION_MARKER` on its own line — the settings read
    happens on this fallback path ONLY (a prompt built from
    summary-bearing docs makes no settings call for the cap). Content at
    or under the cap rides whole, unmarked (nothing was cut). No LLM
    call at chat time — the preview is deterministic.

    Phase 125 (task 05, LOCKED A4; phase 137, task 04, A6): an ENRICHED
    image doc (``is_image`` with a non-NULL ``image_context`` that
    parses — see :func:`_image_context_line`) leads its body with the
    ONE short context line — ``Referenced in <source>/<path>: "<alt>"``
    (the phase-125 reference shape) or ``From <source>/<pdf> (page N)``
    (the phase-137 provenance shape) — before the description, so the
    LLM knows WHY the image exists in the documentation, not only what
    it shows. Every other doc is
    byte-identical to the pre-phase body: every text doc (its
    ``image_context`` may carry the task-04 pending record, which never
    renders) and every non-enriched image doc (NULL ``image_context``
    — the phase's byte-identity criterion).
    """
    summary = (doc.summary or "").strip()
    if summary:
        body = summary
    else:
        limit = get_settings().suggestion_preview_chars
        content = doc.content
        body = (
            content[:limit] + "\n" + TRUNCATION_MARKER
            if len(content) > limit
            else content
        )
    context_line = _image_context_line(doc)
    if context_line is not None:
        body = context_line + "\n\n" + body
    return body


def build_high_prompt(
    documents: Sequence[Document],
    notes: Sequence[str] | None = None,
    kb_overview: str | None = None,
    folder_lines: Sequence[str] = (),
    show_image: bool = False,
) -> str:
    """Grounded turn: locked persona (+ steering, + KB overview) + the
    top-ranked documents' SUMMARY blocks + the ``<tools>`` instructions
    (phase 37; phase 70: the harness-aligned ``ls`` / ``read`` /
    ``grep`` shapes; phase 72: the document-identity contract up front;
    phase 118, LOCKED A6 re-revising A7: the ``<documents>`` section
    seeds SUMMARIES, never full texts — full text enters the context
    only through the capped ``read`` tool; phase 119, LOCKED A4: the
    suggested-folder context lines, below; phase 124, task 03, LOCKED
    A5: the optional :data:`SHOW_IMAGE_CLAUSE`, below).

    Section order: ``<relevance>`` → ``<knowledge_base>`` → ``<tuning>``
    → ``<documents>`` → ``<tools>``; empty steering/overview omit their
    section. ``<tools>`` is always present in the HIGH prompt (the round
    cap — not the prompt — decides whether the tools are actually
    offered to the model, see :mod:`app.rag.agent`).

    *show_image* (phase 124, task 03, LOCKED A5): when true (the
    caller passes the ``BOR_ANSWER_IMAGES`` toggle —
    ``app.api.chat``'s grounded plan is the only call site that does),
    the :data:`SHOW_IMAGE_CLAUSE` rides INSIDE the ``<tools>`` section
    — appended after the existing three-tool copy and before the
    closing ``</tools>`` tag, so the ``<tools>`` marker the E2E mock
    keys on is untouched and the clause reaches the model together with
    the tool teaching. The section order is otherwise UNCHANGED and
    :data:`TOOLS_SECTION` itself is never edited: the built section is
    ``TOOLS_SECTION``'s text minus its final ``</tools>`` + the clause
    + the closing tag. The default ``False`` keeps EVERY pre-existing
    call byte-identical (the pre-task pins in
    :mod:`tests.unit.test_prompts` / :mod:`tests.unit.test_prompt_lock`
    hold untouched). The deflection prompt (:func:`build_deflect_prompt`)
    never carries the clause — a deflected turn has no tools and never
    shows an image.

    When at least one block is present, the section leads with the
    start-here :data:`SUGGEST_INTRO` line (before the first block — the
    phase-15 ``_STEERING_INTRO`` / phase-31 ``_KB_INTRO`` precedent): the
    blocks are the summaries of the top-ranked documents, ``read`` adds
    the full text, and the answer cites only the documents it read (or
    the suggested document it answered from without reading — the
    phase-119 cite discipline, LOCKED A5; the phase-118 "cite what you
    used, by path" sentence is retired). This is NOT the reverted
    phase-72 in-context reminder (that copy
    taught the seed documents as already-read context; this one teaches
    the summary-as-starting-point contract the A6 re-revision requires).

    *folder_lines* (phase 119, D3, LOCKED A4): the suggested-folder
    context lines (``app.rag.agent.suggested_folder_lines``) — the
    direct children of each suggested document's parent folder, so the
    model ``read``s the right file in round 1 instead of walking the
    ``ls`` drill-downs. When blocks are present AND the lines are
    non-empty, the ``<documents>`` body is ``SUGGEST_INTRO``, then the
    folder lines (each on its own line, immediately after the intro
    line), a blank line, then the first ``<document>`` block. They are
    PLAIN lines — no new markup/tag (the E2E mock keys off the
    ``<documents>`` marker and the LAST block's tail, so the lines must
    never land after a summary). Empty *folder_lines* ⇒ the phase-118
    build is byte-identical (pinned in
    ``tests/unit/test_prompts.py``).

    Each ``<document>`` block carries the identity attributes
    ``source`` / ``path`` / ``title`` — and, since phase 106 (D5),
    ``date`` (the row's ``created_at`` UTC date part, ``YYYY-MM-DD``,
    APPENDED after ``title`` — the only position; always present,
    ``created_at`` is NOT NULL) — plus the document's summary
    (:func:`_document_body`; the NULL/whitespace-summary preview
    fallback is LOCKED A5; an enriched image doc's summary leads with
    the phase-125 one-line context line, LOCKED A4).
    """
    # Phase 106 (D5): every document the model sees carries its
    # creation date — the block's ``date`` attribute (the row's
    # ``created_at`` UTC date part, appended after ``title`` — the
    # only position; always present, ``created_at`` is NOT NULL).
    # Phase 118 (A6): the block body is the summary — the full content
    # stays out of the prompt (the capped ``read`` tool is the only
    # full-text path).
    blocks = [
        f'<document source="{doc.source}" path="{doc.path}" title="{doc.title}" '
        f'date="{doc.created_at:%Y-%m-%d}">\n'
        f"{_document_body(doc)}\n"
        "</document>"
        for doc in documents
    ]
    body = "\n\n".join(blocks) if blocks else (
        "(no documents matched — do not invent specifics)"
    )
    if blocks:
        if folder_lines:
            # Phase 119 (D3, LOCKED A4): the folder lines ride immediately
            # after the intro line, one per line, a blank line, then the
            # first block — plain lines, no new markup (the <document>
            # block markup AND body stay byte-identical around them).
            body = SUGGEST_INTRO + "\n" + "\n".join(folder_lines) + "\n\n" + body
        else:
            body = SUGGEST_INTRO + "\n\n" + body
    prompt = _base("HIGH")
    for part in (build_kb_section(kb_overview or ""), build_steering_section(notes or [])):
        if part:
            prompt += "\n" + part
    if show_image:
        # Phase 124 (task 03, LOCKED A5): the clause rides inside
        # <tools> — TOOLS_SECTION's text minus its final </tools>, the
        # clause (on its own line, after the existing three-tool copy),
        # then the closing tag. TOOLS_SECTION itself is unedited and the
        # show_image=False path keeps the exact pre-task concatenation.
        tools_section = TOOLS_SECTION[: -len("</tools>")] + SHOW_IMAGE_CLAUSE + "\n</tools>"
    else:
        tools_section = TOOLS_SECTION
    return prompt + "\n<documents>\n" + body + "\n</documents>\n" + tools_section


def build_deflect_prompt(
    titles: Sequence[str],
    notes: Sequence[str] | None = None,
    kb_overview: str | None = None,
) -> str:
    """Deflection turn: weak-hit titles only (no document content).

    Section order (phase 31): ``<relevance>`` → ``<knowledge_base>`` →
    ``<tuning>`` → ``DEFLECT_MODE`` body; empty steering/overview omit
    their section, keeping the prompt byte-identical to the pre-phase
    text. The body ends with the phase-71 plain-text line (owner-
    permitted 2026-09-03 — the LOW prompt's only change): a deflected
    turn offers no tools, so any tool markup there is always wrong.
    """
    weak = "\n".join(f"- {t}" for t in titles) if titles else "(nothing close at all)"
    mid = "\n".join(
        part
        for part in (build_kb_section(kb_overview or ""), build_steering_section(notes or []))
        if part
    )
    gap = f"\n{mid}\n" if mid else "\n"
    return (
        _base("LOW")
        + gap
        + "DEFLECT_MODE: retrieval was weak — the titles below are the closest "
        "your notes come to the question. They are titles only; do not pretend "
        "they answer it. Use them to propose 2-3 alternative questions.\n"
        # Phase 71 (owner-permitted 2026-09-03): the one plain-text line
        # — prevention at the prompt. The E2E mock keys on the
        # DEFLECT_MODE marker's presence, so appending is safe.
        "Reply in plain text only — you have no tools in this mode.\n"
        + weak
    )