brain-of-reese/app/api/chat.py

"""POST /api/chat — a RAG chat turn streamed over SSE (PLAN §3/§4).

Flow (LOCKED A7/A15): embed the question → hybrid retrieval (cosine
top-N ∪ Postgres FTS top-N, RRF-fused) → the **honesty gate** → locked
persona prompt (PLAN §6) → ``turbo`` streamed as ``delta`` events → final
``done`` event (``deflected``, ``sources``, ``suggestions``) +
``query_log`` row + the per-turn log line (PLAN §9).
Mid-stream failures become a structured ``error`` event; a pre-stream DB
outage is a plain 503 JSON.

Thinking (phase 17, PLAN §4 extension, owner permission 2026-08-23): the
model's reasoning arrives ahead of the answer and is streamed as
``thinking`` events before the ``delta`` events of the same turn. Each
turn's thinking is counted in the per-turn log line
(``thinking_chars=N``); ``BOR_STREAM_THINKING=0`` suppresses the
``thinking`` frames (the pieces are still counted). The per-turn log
line records ``thinking=<level>`` (the effective
``BOR_LLM_CHAT_THINKING_LEVEL``, trimmed, ``-`` when empty) before
``thinking_chars`` (phase 172, A6; phase 178, A5 — the **chat**
level; the one-shot summary level never appears on the line).

Honesty gate (A8, revised 2026-09-14): LOW — deflection — only when
``best_cosine < BOR_RELEVANCE_THRESHOLD`` **and** (``fts_hits == 0``
or ``best_cosine < BOR_LEXICAL_SUPPORT_FLOOR``). A lexical hit alone
(cosine < lexical_support_floor) no longer promotes to HIGH — the vector
signal must corroborate the lexical match (A8 revised 2026-09-14, the
"Mongolia" fix). HIGH fires when ``best_cosine >= threshold`` OR
(fts>0 AND cosine >= lexical_support_floor). Deflection mode carries
weak-hit *titles only* (never document content) plus deterministic
"Maybe try" chips, and the ``done`` event / ``query_log`` row record
``deflected=true``, the weak score and the ``fts_hits`` count. A
deflected turn cites nothing: the ``done`` frame's ``sources`` is
``[]`` (``done.sources`` is the citation surface — the UI chips every
entry as "the answer used this" — and weak hits are scored docs, not
citations, TODO L2), while ``query_log.sources`` and the per-turn log
line keep recording the retrieval for tuning (phase 112, A8 revised).


Steering (phase 15): the owner's stored tuning notes are loaded per turn
(oldest first) and injected into the system prompt as a ``<tuning>``
section — both the HIGH and the LOW prompt carry it. The per-turn log
line records ``tuning=N`` (the number of injected notes).

Summaries (phase 30; phase 118 redefinition): a lite-model summary
chunk's parent *is* the source document, so a summary hit resolves to
the full source document through the unchanged chunk→document mapping
(A7 revised). ``TurnPlan.summary_hits`` counts the hit chunks with
``is_summary`` whose parent document is in the SUGGESTED set (phase 118:
redefined from the phase-113 cited set — the suggestion tier is the
seeded context now), and the per-turn log line records ``summary_hits=N``
after ``fts_hits`` (PLAN §9 line extension; phase 118 adds
``suggested=N`` after ``summary_hits=N``).

Summary seed context (phase 118, TODO L3 — the owner directive that
re-revises A7, LOCKED A6): a grounded turn's ``<documents>`` section
seeds the top-5 suggested documents' SUMMARIES (never their full
texts) — the "start here if these summaries seem right to you" starting
point — and the LLM extends its context by ``read``-ing what it needs
(the capped ``read`` tool is the ONLY full-text path). Both tiers are
computed once per turn, for BOTH branches: ``suggested_docs`` (no
floor, A3) and ``related_docs`` (rank 6+, the done frame's row).
Phase 119 (LOCKED A1, owner decision 2026-09-16 — the phase-118 A4
union RETIRED): ``done.sources`` is the citation surface and carries
the AGENT-READ documents only (``holder.read_docs``, deduped by
``(source, path)``, read order) — a zero-read grounded turn (the
summary-only fast path) chips nothing, an accepted, owner-directed
consequence (the answer prose names the doc it used per the phase-119
cite discipline, A5); ``query_log.sources`` and the per-turn log line
keep recording the full retrieval (suggested + related + read, 118-A3
stands — the durable record records retrieval, not citations). The
deflected branch's prompt is byte-identical (weak-hit titles only —
A8 untouched); its TurnPlan still carries both tiers for the durable
record.

KB overview (phase 31): the lite-generated outline of the knowledge
base (single ``kb_overview`` row) is read per turn (one indexed PK
lookup — no LLM call) and injected into **both** prompts as the
``<knowledge_base>`` section, ordered ``<relevance>`` →
``<knowledge_base>`` → ``<tuning>`` → mode body. With an empty row the
prompts stay byte-identical to the pre-phase text (phase 15
convention); ``TurnPlan.kb_chars`` records the length of the stored
outline (0 when absent) and the per-turn log line records
``kb_chars=N`` after ``tuning=N`` (PLAN §9 line extension).

Agent document tools (phase 37, PLAN §4 extension, owner permission
2026-08-26; phase 45 removed the per-tool budgets — owner permission
2026-08-27; phase 70 aligned the surface to the harness-trained
``ls`` / ``read`` / ``grep`` — owner permission 2026-09-03): a
**grounded** turn (``not plan.deflected``) no longer streams a bare
``chat_stream`` — it runs the agent loop (``app.rag.agent.run_agent``),
which offers the model the three server-side tools
``ls`` / ``read`` / ``grep`` for the whole turn (as many calls as the
model wants, re-lists and re-greps included) until it answers or the
round cap (``BOR_AGENT_MAX_ROUNDS``, default 10) forces one final
no-tools answer. Each model-requested call streams as an SSE ``tool``
event — ``{"type": "tool", "name": …, "argument": … | null}`` — ahead
of the answer's ``delta`` frames: ``argument`` is the single string the
model passed — ``read``'s ``path`` (the combined ``source/path``),
``grep``'s ``pattern``, ``ls``'s ``path`` — or null (a non-string
value — a model error the backend refuses — and an omitted argument
both yield null).

Tool-result frames (phase 95, ``TODO.md`` L5 — A15 extension, owner
permission 2026-09-10; the event-type list grows from six to SEVEN,
``tool_result`` among them; PLAN.md is being redone by the owner): a
``read`` whose document is longer than ``BOR_READ_MAX_CHARS`` streams,
AFTER the matching ``tool`` frame (the line is already on screen — the
marker lands a beat later, the phase-37/48 tool-line timing is
untouched) and BEFORE the next model round, exactly one optional
``tool_result`` event — ``{"type": "tool_result", "name": "read",
"argument": …, "truncated": true, "chars_shown": N, "chars_total": M}``
— the additive truncation notice the UI turns into the
"(truncated — showing N of M chars)" marker on the Reading line. A
non-truncated read streams NO such frame (one frame = one noteworthy
event), deflected turns never stream one (the agent never runs, A8),
and every pre-existing frame is byte-identical — clients that do not
know the type ignore it. ``done.sources``, ``query_log.sources`` and the
per-turn log line all report the same combined source list (retrieval
docs + the agent's read docs, deduped by ``(source, path)``, order
preserved — a grep adds no source; it is a locator, locked A5), and the
log line records ``tool_calls=N`` after ``thinking_chars=N`` (PLAN §9
line extension — ``N`` counts executed tool calls; rejected calls do not
count). **Deflected turns keep the direct ``chat_stream`` —
byte-identical to the pre-phase path (A8):**
the LOW prompt never carries tools, and with ``agent_max_rounds`` at
**0** ``run_agent`` makes exactly one ``tools=None`` request,
reproducing the pre-phase behavior (the kill switch).

No ``tool`` frame without tools (phase 160, task 02): a request made
WITHOUT tools never produces a ``tool`` frame — a structured ``tool_calls``
reply to a ``tools=None`` request (the documented ``lite`` behavior, the
phase-71 incident class) is rejected at the transport: ``_pump``
(``tools_offered=False`` on the deflected turn and its recovery) drops the
``ToolCallPiece`` (counted in a local, named by one debug line) instead of
rendering it as a tool line, and the grounded cap path's rejected calls are
suppressed inside ``run_agent`` (``holder.rejected_calls``). The ``tool``
event shape (A15) is unchanged — a frame is simply no longer emitted for a
call that cannot run.

Budget-exhausted report (phase 160, task 03, TODO L4): a turn that
hits ``BOR_AGENT_MAX_ROUNDS`` reports itself — the done frame carries
one additive key, ``budget_exhausted: true`` (driven by
``AgentHolder.cap_reached``; ABSENT — never ``false`` — on every other
turn, so non-cap frames stay byte-identical to pre-phase; the UI's
"Continue researching" affordance keys on it, task 04). The per-turn
log line records ``rounds=N`` and ``cap_hit=0|1`` after
``tool_calls=N`` (PLAN §9): ``N`` is the rounds CONSUMED (an executed
call or a rejection; a cap-path rejected call is not a round) and
``cap_hit=1`` only on a cap-reached turn — ``rounds=0 cap_hit=0`` on
deflected turns (the agent never runs) and kill-switch turns.

LLM retries (phase 67, ``BOR_LLM_RETRIES`` / ``BOR_LLM_RETRY_DELAY``,
owner-locked 2026-09-01): when the aipi endpoint dies before a request
has streamed its first output frame (locked A2), the request is
restarted — up to ``llm_retries`` times (default 3), a flat
``llm_retry_delay`` (default 5 s) between attempts. Every restart is
announced with an SSE ``retry`` frame (``{"type": "retry",
"attempt": n, "max_attempts": N}`` — the attempt about to be tried,
1-based, ahead of the pre-retry wait) so the UI can show the transient
"Communication interrupted — retrying (n of N)…" status (locked A4);
exhaustion and any failure after the first frame keep the existing
terminal ``error`` frames. The pre-stream question embedding retries on
the same budget, and the deflected answer stream goes through
``chat_stream_retried`` (the agent loop retries per round — task 03).
The per-turn log line records ``retries=N`` after ``total_ms=N`` (0
when nothing was retried — the field is uniform across all turn
shapes).

Tool-scaffolding guardrail (phase 71, deterministic only — owner
permission 2026-09-03: "deterministic guardrails only right now, forget
using a model for that"): the raw chat-template tokens
``<|tool_call_start|>…<|tool_call_end|>`` the ``lite`` model sometimes
emits as plain answer text can never reach the user. The DEFLECTED
path's request runs ``delta.content`` through a caller-owned
``ScaffoldingFilter`` (content only — thinking stays raw); when the
filter wipes the whole reply (visible content 0, ``stripped_chars > 0``)
the turn gets exactly ONE bounded recovery: the same messages with
``agent.CORRECTION_INSTRUCTION`` folded into the single system prompt,
``tools=None``, a fresh filter, the same phase-67 retry budget, streamed
through the same piece loop (extracted as the inner ``_pump`` helper).
A recovery that also comes back empty — and any grounded round where the
recovery policy in ``run_agent`` fails — settles with the dedicated
structured ``error`` frame (``MalformedReplyError`` caught before the
generic ``LLMError`` handler): no ``done``, no ``query_log`` row,
byte-for-byte the existing terminal-error shape. The per-turn log line
records ``scaffold_stripped=N`` after ``retries=N`` — the sum across the
turn's requests (grounded: the agent's rounds + forced final + any
recovery, via the holder; deflected: this turn's filters), 0 on clean
turns (the field is uniform, the phase-67 ``retries=N`` pattern); the
recovery does not bump ``retries=N`` (it is not a phase-67
endpoint-retry).

Chat history (phase 74, TODO L4, owner-locked A2/A3/A4 2026-09-08):
``POST /api/chat`` accepts an optional ``history`` — the client's prior
turns, oldest first (the ``bor.chat.v1`` record minus the current
question; the endpoint stays stateless per A10 — nothing is stored). It
is mapped ONCE per turn by :func:`app.rag.prompts.history_to_messages`
— trimmed newest-first against the settings budgets
(``history_max_turns`` / ``history_max_chars``; a capped-out turn is
dropped whole, never truncated) — and fed to the model on BOTH turn
branches: the deflected path splices it between the system prompt and
the current user message (the phase-71 recovery still rebuilds from
``messages[1:]`` — unchanged), and the grounded agent receives it as
``run_agent(..., history=hist)``. Prior brain turns' thinking travels
as ``reasoning_content`` on the assistant message (the preserve-
thinking wire convention, A4). The per-turn log line records
``history_msgs=N`` after ``kb_chars=N`` (0 when the request carries no
history — the two-message request stays byte-identical).

Question images (phase 123, TODO L6; LOCKED A5/A7): the request may
attach ONE image to the CURRENT question — ``ChatRequest.image`` is the
STORED path from ``POST /api/chat-images`` (``app.api.chat_images``),
never a data URL. The turn validates it BEFORE any model call (the
toggle gate — ``settings.images`` false settles the phase-114 hinted
error frame; the stale-file gate — the stored bytes deleted out-of-
band settles the same frame shape) and, when valid, the user message's
content becomes the multimodal list ``[{type: "text", …}, {type:
"image_url", image_url: {url: <data URL>}}]`` — built by
:func:`build_user_content` at BOTH construction sites (this module's
deflected-branch ``messages`` list and the grounded branch's
``run_agent`` call, which builds its own ``[system, *history, user]``
— the flow is pinned in ``app.rag.agent.run_agent``). The data URL is
the shared :func:`app.rag.summarizer.image_data_url` (one
construction, both call sites — the phase-122 describe path).
``image=None`` keeps the plain-string content byte-identical to
pre-phase. A question image is turn-local: it is NEVER indexed as a
document, and prior turns' images are never replayed into the model's
history (``history_to_messages`` is unchanged — the text of a prior
turn that had an image stands alone, LOCKED A7).

Verbatim display (phase 182, tasks 03/05; LOCKED A1/A4/A7/A9): a turn
that showed file contents through the ``show_text`` tool (task 03, the
bytes the SERVER resolved) — or whose ANSWER carries a `` ```bor ``
locator fence (task 02's resolver, placement sugar that resolves
standalone) — gains one additive ``done`` key, ``answer_text``: the
list of :class:`~app.schemas.VerbatimBlockRef` blocks in DISPLAY order,
each carrying the exact ``documents.content`` slice plus the SERVER's
resolved 1-based, inclusive line extent and its canonical ``locator``.
The model never supplies those bytes: the endpoint accumulates the
turn's CLEAN VISIBLE answer text (the same post-scaffold-filter pieces
that feed ``content_chars`` — never ``thinking``, so a locator can only
ever come from the answer) and resolves it through
:func:`app.rag.verbatim.resolve_answer_blocks` when the ``done`` frame
is built. A deflected turn resolves NOTHING (D9 — a fence hallucinated
into a disclaimer must not leak KB text through the turn that just said
it had nothing); ``answer_text=False``, ``show_text_max_blocks=0`` and
``show_text_max_chars=0`` short-circuit inside the resolver, and a turn
with no blocks serializes WITHOUT the key (A9 — every block-free frame
keeps its exact pre-phase bytes). The per-turn log line records
``verbatim_blocks=N`` after ``scaffold_stripped=N`` — the blocks
ACTUALLY SENT (after dedupe and the ``show_text_max_blocks`` cut), 0 on
every turn without them including deflected ones (the PLAN §9 additive
revision).

Mid-turn verbatim frame (phase 192, task 01, A1; ``TODO.md`` L5): the
same block ALSO reaches the client the moment the ``show_text`` call
succeeds, as the additive ``verbatim_block`` event —
``{"type": "verbatim_block", "source", "path", "title", "start_line",
"end_line", "total_lines", "text", "truncated", "locator"}`` (the exact
:class:`~app.schemas.VerbatimBlockRef` field set, so the one panel client
accepts it as-is). It lands AFTER the round's ``tool`` / ``tool_result`` frames and
BEFORE the next model round — while the answer is still streaming, so the
panel appears where the model introduced the bytes instead of popping in
after it finished talking. Success-only: every refusal (missing ``path``,
no document, an image row, the block cap, the already-shown dedupe) emits
nothing, and a ``BOR_ANSWER_TEXT=0`` turn offers no tool, so its frame
sequence is byte-identical to pre-phase. Everything downstream of the
turn keeps the phase-182 contract EXACTLY: the ``done`` frame's
``answer_text`` list, the saved record key, the restore path and the
shared page are untouched, and ``verbatim_blocks=N`` still counts the
``done``-frame blocks (never the mid-turn frames).
"""
from __future__ import annotations

import asyncio
import json
import logging
import time
from collections.abc import AsyncIterator, Mapping, Sequence
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any

from fastapi import APIRouter, Depends
from fastapi.responses import JSONResponse, StreamingResponse
from sqlalchemy.orm import Session

from app.api.steering import load_steering_notes
from app.config import Settings, get_settings
from app.core.auth import require_user
from app.db import SessionLocal, db_available
from app.models import Document, QueryLog
from app.rag.agent import (
    CORRECTION_INSTRUCTION,  # phase 71: the harness-owned recovery line
    AgentHolder,
    AgentPiece,  # the ONE union run_agent yields (phase 192: + VerbatimBlockPiece)
    MalformedReplyError,  # phase 71: the recovery policy's terminal signal
    VerbatimBlockPiece,  # phase 192: one shown verbatim block (mid-turn frame)
    run_agent,
    suggested_folder_lines,  # phase 119 (D3): the suggested-folder context lines
)
from app.rag.llm import (
    EmbeddingError,
    EmbeddingInputTooLargeError,  # phase 114: the deterministic too-large failure
    LLMClient,
    LLMError,
    RetryPiece,  # phase 67: one LLM request restart (an SSE retry frame)
    ToolCallPiece,  # phase 37: one model-requested tool call
    ToolResultPiece,  # phase 95: one truncated tool result (SSE frame = task 02)
    chat_stream_retried,  # phase 67: the retry-before-first-piece primitive
)
from app.rag.overview import load_kb_overview
from app.rag.prompts import build_deflect_prompt, build_high_prompt, history_to_messages
from app.rag.retriever import (
    RetrievedChunk,
    retrieve,
    select_related,
    select_suggested,
    source_ref_with_image,
    weak_hit_titles,
)
from app.rag.scaffolding import ScaffoldingFilter  # phase 71: the streaming filter
from app.rag.source_labels import (
    SourceLabelView,
    all_labels,
    question_name_hits,
)
from app.rag.suggestions import derive_suggestions
from app.rag.summarizer import (
    IMAGE_FALLBACK_MIME,
    IMAGE_MIMES,
    image_data_url,
)  # phase 123: the phase-122 mime map + the shared data-URL helper
from app.rag.verbatim import (
    resolve_answer_blocks,  # phase 182: fences + shown blocks -> the done frame
)
from app.schemas import (
    ChatDoneEvent,
    ChatErrorEvent,
    ChatRequest,
    ChatRetryEvent,
    ChatThinkingEvent,
    ChatToolEvent,
    ChatToolResultEvent,
    ChatVerbatimBlockEvent,
    SourceRef,
    VerbatimBlockRef,
)

logger = logging.getLogger("app.chat")
router = APIRouter(tags=["chat"])

#: Streaming hints: no proxy buffering, no client caching (PLAN A15).
SSE_HEADERS = {"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}

#: Concurrency cap for /api/chat (SEC-14-04, task 03). A module-level
#: counter for the fast-path pre-check and a Semaphore for the real gate.
_chat_active: int = 0
_chat_semaphore: asyncio.Semaphore | None = None


def _get_chat_semaphore() -> asyncio.Semaphore:
    """Lazy-initialize the chat concurrency semaphore (SEC-14-04).

    The semaphore is initialized on first use so that ``get_settings()``
    is called with the correct environment — never at import time.
    """
    global _chat_semaphore
    if _chat_semaphore is None:
        settings = get_settings()
        _chat_semaphore = asyncio.Semaphore(max(1, settings.chat_max_concurrent))
    return _chat_semaphore


_llm: LLMClient | None = None


def get_llm() -> LLMClient:
    """Shared LLM client (FastAPI dependency so tests can override it)."""
    global _llm
    if _llm is None:
        _llm = LLMClient(get_settings())
    return _llm


def reset_runtime_caches() -> None:
    """Drop the settings-derived singletons (phase 168, task 03).

    Called by ``invalidate_settings()`` after the admin saves. The next
    turn rebuilds the client from the CURRENT ``get_settings()`` — an
    in-flight turn keeps the client it started with (its ``AsyncOpenAI``
    transport is mid-request; swapping it would kill the stream), which
    is the same accepted-semantics note as A12's in-process state.
    ``_chat_active`` is deliberately untouched: it counts live turns, and
    the new ``chat_max_concurrent`` applies as permits are taken.
    """
    global _llm, _chat_semaphore
    _llm = None
    _chat_semaphore = None


def _whole_line_number(value: Any) -> bool:
    """A ``read`` range argument's whole-line-number guard (phase 131,
    task 04).

    The same validation the agent's ``_read_range_args`` applies
    server-side (``app/rag/agent.py``): an ``int`` (bools excluded —
    the house guard) and ``>= 1``. A value that fails is a REFUSED
    call (the ``BAD_RANGE_ARGS`` teaching), and its ``tool`` frame
    carries no range — the refusal surfaces as the round's result, not
    as the call's arguments.
    """
    return isinstance(value, int) and not isinstance(value, bool) and value >= 1


def sse_event(payload: dict[str, Any]) -> str:
    """Serialize one SSE frame: ``data: <json>\\n\\n`` (PLAN §4)."""
    return f"data: {json.dumps(payload, ensure_ascii=False)}\n\n"


def build_user_content(
    message: str,
    image_path: str | None,
    settings: Settings,
) -> str | list[dict[str, Any]]:
    """The current turn's user message content (phase 123, task 01).

    *image_path* ``None`` (every text-only question) → the plain
    question string — byte-identical to pre-phase-123 (the multimodal
    branch is inert). Set → the OpenAI-compatible multimodal content
    list: the text part + the image part, a data URL built server-side
    from the stored bytes (``settings.chat_image_dir`` + the path's
    filename) and the phase-122 ``IMAGE_MIMES`` map (one map, one
    truth) through the shared :func:`app.rag.summarizer.image_data_url`
    helper (one construction, both call sites — the phase-122 describe
    path). The caller has already validated the path shape (the
    ``ChatRequest.image`` schema guard) and the file's existence (the
    pre-stream turn gate). The image applies to the CURRENT turn only
    (LOCKED A7) — prior turns' images are never replayed.
    """
    if image_path is None:
        return message
    filename = image_path.rsplit("/", 1)[-1]
    data = (Path(settings.chat_image_dir).expanduser() / filename).read_bytes()
    mime = IMAGE_MIMES.get(Path(filename).suffix.lower(), IMAGE_FALLBACK_MIME)
    return [
        {"type": "text", "text": message},
        {"type": "image_url", "image_url": {"url": image_data_url(data, mime)}},
    ]


@dataclass
class TurnPlan:
    """What one chat turn sends to the LLM and reports on ``done``."""

    top_score: float  # best cosine across candidates (query_log.top_score)
    fts_hits: int  # lexical (OR-tsquery) candidates matched
    deflected: bool
    system_prompt: str
    # The summary-seeded suggestion tier (phase 118, A6: top-N distinct
    # documents, NO floor, A3 — the HIGH prompt carries their SUMMARIES
    # and they seed run_agent). Phase 119 (LOCKED A1): NOT the done
    # frame's citation surface anymore — done.sources is the
    # agent-read docs only (the phase-118 A4 union retired); this tier
    # still rides the durable record (query_log + the log line).
    suggested_docs: list[Document]
    # Rank 6+ after the suggested set (phase 118: the next ranked docs
    # that are not already suggested, at most related_max_docs — the
    # done frame's de-emphasized "nearby docs" row, never a citation).
    related_docs: list[Document] = field(default_factory=list)
    suggestions: list[str] = field(default_factory=list)  # "Maybe try" chips (deflected turns only)
    tuning_count: int = 0  # steering notes injected into the system prompt
    #: Hit chunks with ``is_summary`` whose parent document is in the
    #: SUGGESTED set (phase 30; redefined from the phase-113 cited set
    #: in phase 118; per-turn log line ``summary_hits=N``).
    summary_hits: int = 0
    #: Length of the stored KB overview injected as the
    #: ``<knowledge_base>`` section (phase 31; per-turn log line
    #: ``kb_chars=N``). 0 when no non-empty row exists.
    kb_chars: int = 0


def plan_turn(
    chunks: Sequence[RetrievedChunk],
    settings: Settings,
    notes: Sequence[str] | None = None,
    kb_overview: str | None = None,
    folder_lines: Sequence[str] = (),
    names: Mapping[str, SourceLabelView] | None = None,
) -> TurnPlan:
    """Apply the honesty gate (A8, revised) and assemble prompt + context.

    * **HIGH (grounded)** when ``best_cosine >= threshold`` **or**
      (``fts_hits > 0`` **and** ``best_cosine >= lexical_support_floor``):
      HIGH prompt seeded with the suggested documents' SUMMARIES (phase
      118, A6 — never their full texts), no suggestions. A cosine
      exactly at the threshold is an answer — the gate is strict
      (``< threshold``). An FTS hit alone, without vector corroboration
      (cosine < lexical_support_floor), stays LOW (A8 revised 2026-09-14).
    * **LOW (deflected)** otherwise — including the fts>0 / cosine < floor
      case (the "Mongolia" case): LOW prompt (``DEFLECT_MODE``) with
      weak-hit titles only — never document content — plus deterministic
      alternative-question chips derived from those titles.

    ``top_score`` (stored in ``query_log``) is the best cosine, so the
    gate input is always a pure vector-similarity number; the lexical
    signal is recorded separately as ``fts_hits``.

    *notes* are the owner's steering notes (phase 15, oldest first):
    when non-empty, both the HIGH and the LOW prompt carry the
    ``<tuning>`` section; with no notes the prompts are unchanged.

    *kb_overview* is the stored KB outline (phase 31, one PK lookup per
    turn): when non-empty, both prompts carry the ``<knowledge_base>``
    section (between ``<relevance>`` and ``<tuning>``) and
    ``kb_chars`` records the outline's length; with no outline the
    prompts are byte-identical to the pre-phase text and ``kb_chars``
    is 0.

    *folder_lines* (phase 119, D3, LOCKED A4): the suggested-folder
    context lines, computed by the chat endpoint (which holds the DB
    session) via ``suggested_folder_lines`` over the SAME deterministic
    suggested walk this function performs internally. The HIGH branch
    passes them to ``build_high_prompt`` (they ride after
    ``SUGGEST_INTRO``, before the first ``<document>`` block); the LOW
    branch IGNORES them — the deflected prompt stays byte-identical.
    Empty *folder_lines* ⇒ the HIGH prompt is byte-identical to the
    phase-118 shape. The :class:`TurnPlan`, the durable record, and the
    per-turn log line are untouched.

    *names* (phase 191, D4): the source-name → ``SourceLabelView`` map of
    the owner's label rows — the SAME single ``all_labels`` read the
    endpoint performs for the task-04 selection boost (ONE query per
    turn). The HIGH branch hands it to ``build_high_prompt`` (the
    additive ``name`` / ``label`` attributes of the ``<document>``
    blocks); the LOW branch IGNORES it — the deflected prompt stays
    byte-identical. ``None`` ⇒ every block is byte-identical to
    pre-phase.

    ``summary_hits`` (phase 30; phase 118 redefinition) counts the hit
    chunks with ``is_summary`` whose parent document is in the
    SUGGESTED set — both the HIGH and the LOW branch record it.

    Phase 118 (the summary seeding, LOCKED A3/A6): retrieval documents
    are tiered once, before either branch — ``suggested_docs`` are the
    top-N distinct parent documents in fused rank order with NO cosine
    floor (``settings.suggested_docs``, default 5 — the "start here"
    seeding: the HIGH prompt carries their SUMMARIES, never their full
    texts; the floor never filters, so a lexical-only hit is suggested
    when it ranks) and ``related_docs`` are the next ranked documents
    that are not already suggested (rank 6+ for the contiguous top-5
    suggestion set, at most ``related_max_docs`` — the done frame's
    de-emphasized "nearby docs" row). On a deflected turn the weak hits
    are suggested too (no floor); the LOW prompt itself is unchanged
    (weak-hit titles only), and both tiers still ride the TurnPlan for
    the durable record (LOCKED A3: query_log records retrieval, not
    citations).

    ``settings.top_n_docs`` and ``settings.source_usefulness_floor`` are
    NOT consulted here (phase 118 retired their seeding role, A6) —
    they remain settings for env back-compat only.
    """
    steering = list(notes or [])
    kb_text = (kb_overview or "").strip()
    kb_chars = len(kb_text)
    best_cosine = max((c.cosine for c in chunks), default=0.0)
    fts_hits = sum(1 for c in chunks if c.fts_hit)
    # Phase 118 (A3/A6): BOTH tiers, computed once, for BOTH branches —
    # the suggested tier (top-N, NO floor) seeds the HIGH prompt and the
    # agent; the related tier (rank 6+ after the suggested set) feeds
    # the done frame's row and the durable record.
    suggested = select_suggested(chunks, n=settings.suggested_docs)
    related_docs = select_related(
        chunks, {d.id for d in suggested}, settings.related_max_docs
    )
    suggested_ids = {d.id for d in suggested}
    summary_hits = sum(1 for c in chunks if c.is_summary and c.document.id in suggested_ids)
    lexical_supported = fts_hits > 0 and best_cosine >= settings.lexical_support_floor
    if best_cosine >= settings.relevance_threshold or lexical_supported:
        return TurnPlan(
            best_cosine,
            fts_hits,
            False,
            build_high_prompt(
                suggested,
                notes=steering,
                kb_overview=kb_text,
                folder_lines=folder_lines,
                # Phase 124 (task 03, LOCKED A5): the grounded HIGH prompt
                # carries the show_image clause exactly when the
                # BOR_ANSWER_IMAGES toggle is on (off = byte-identical).
                show_image=settings.answer_images,
                # Phase 182 (task 04, LOCKED A1/A8): the same rule for the
                # show_text clause — it rides exactly when the BOR_ANSWER_TEXT
                # master switch is on (the default), and an off build is
                # byte-identical to pre-phase (no clause, and agent_tools
                # offers four tools, not five).
                show_text=settings.answer_text,
                # Phase 191 (D4, task 05): the owner's source names ride the
                # same map the boost flags came from — one all_labels read
                # per turn, no second label query here.
                names=names,
            ),
            suggested,
            related_docs,
            [],
            len(steering),
            summary_hits,
            kb_chars,
        )
    titles = weak_hit_titles(chunks)
    return TurnPlan(
        best_cosine,
        fts_hits,
        True,
        build_deflect_prompt(titles, notes=steering, kb_overview=kb_text),
        suggested,
        related_docs,
        derive_suggestions(titles, settings.suggestions),
        len(steering),
        summary_hits,
        kb_chars,
    )


def merge_cited_docs(
    read_docs: list[Document],
    suggested_docs: list[Document],
    deflected: bool,
) -> list[Document]:
    """Compose the citation tier (the done frame's ``sources``, PLAN §9).

    Phase 119 (LOCKED A1): on grounded turns the agent's READ DOCS —
    deduped by ``(source, path)``, read order; on deflected turns
    nothing (a weak-hit answer is not a citation). Phase 124 (TODO L3)
    adds the rule's single documented exception: a suggested-tier
    IMAGE doc's seed block IS its entire content (content == summary
    == the vision description, phase 122 task 03) — a ``read`` would
    add only the marker line, so a real model never performs one and
    the doc's chip never appeared. "Suggested" is therefore equivalent
    to "read" for it, and it is auto-cited — in SUGGESTION order AFTER
    the read docs, deduped against them (an image doc the agent also
    explicitly read appears once, at its read position). Non-image
    suggested docs are never auto-cited: the phase-119 read-docs-only
    rule stays intact for text docs (a zero-read text turn still chips
    nothing).
    """
    if deflected:
        return []
    cited: list[Document] = []
    seen: set[tuple[str, str]] = set()
    for doc in read_docs:
        key = (doc.source, doc.path)
        if key not in seen:
            seen.add(key)
            cited.append(doc)
    # Phase 124 (TODO L3): the suggested-tier IMAGE docs join the
    # citation tier — "suggested" is equivalent to "read" for them
    # (their seed block is their entire content), so a zero-read
    # grounded turn chips them too. Suggestion order, deduped.
    for doc in suggested_docs:
        if not doc.is_image:
            continue
        key = (doc.source, doc.path)
        if key not in seen:
            seen.add(key)
            cited.append(doc)
    return cited


@router.post("/chat")
async def chat(
    request: ChatRequest,
    _user: None = Depends(require_user),  # noqa: B008  # phase 79: admin or live token
    llm: LLMClient = Depends(get_llm),  # noqa: B008
):
    """One chat turn: SSE stream of ``delta`` events + a final ``done``.

    User-gated (phase 79): anonymous callers get 401 ``authentication
    required`` before any streaming — the ONLY anonymous content is the
    shared chats.

    DB sessions (SEC-14-04): no long-lived session is held across the
    SSE stream — every DB step (retrieval, tool execution, query_log
    write) uses its own short-lived session via ``SessionLocal()``.

    Concurrency (SEC-14-04, task 03): a semaphore limits concurrent
    turns; excess requests get a 503 error.
    """
    # SEC-14-04 / task 03: concurrency cap — reject immediately if at
    # capacity (the fast-path pre-check; the semaphore inside stream()
    # is the real gate that prevents races).
    settings = get_settings()
    max_concurrent = settings.chat_max_concurrent
    if _chat_active >= max_concurrent:
        return JSONResponse(
            status_code=503,
            content={"detail": "Too many concurrent chat turns — try again."},
        )

    if not db_available():
        return JSONResponse(
            status_code=503,
            content={
                "detail": (
                    "The knowledge base is offline — start Postgres with "
                    "`podman compose up -d db`, then ask again."
                )
            },
        )

    started = time.monotonic()

    def db_factory() -> Session:
        """SEC-14-04: session factory for short-lived sessions."""
        return SessionLocal()

    async def stream() -> AsyncIterator[str]:
        # SEC-14-04 / task 03: concurrency accounting — increment before
        # the semaphore acquire (the pre-check counter is best-effort;
        # the semaphore is the real gate). The outer finally decrements
        # when the stream ends (normal or error).
        global _chat_active
        _chat_active += 1
        sem = _get_chat_semaphore()
        await sem.acquire()
        try:
            # Phase 48: one terminal flag — ``True`` at every terminal
            # exit (the ``done`` yield; every ``error``-then-``return``).
            # The ``finally`` below logs the cancelled-turn line only when
            # the consumer went away before any terminal frame; it must not
            # yield (GeneratorExit handling).
            settled = False
            retries_used = 0  # phase 67: LLM requests restarted this turn (log line)
            try:
                settings = get_settings()
                # Phase 123 (TODO L6, task 01): the question's attached
                # image — validated BEFORE any model call (the embed is
                # a model call): a rejected turn calls nothing and
                # settles with the phase-114 error frame (the existing
                # error-path convention — no ``done``, no ``query_log``
                # row, no persisted record; the question is not saved).
                # Two gates, in order:
                #   1. the ``images`` toggle (``BOR_IMAGES``) — off with
                #      an image set: the HINTED frame (the client's
                #      banner shows the hint in place of its default
                #      reachability copy, phase 114);
                #   2. the stored file — the schema already pinned the
                #      path shape, but the file may have been deleted
                #      out-of-band (the stale-path edge): the same frame
                #      shape, no hint (the banner's default copy is the
                #      honest fallback — there is nothing to point at).
                if request.image is not None:
                    if not settings.images:
                        logger.warning(
                            "chat: image question rejected (images toggle "
                            "off) question=%r image=%r",
                            request.message,
                            request.image,
                        )
                        settled = True  # terminal: the error frame settles the turn
                        yield sse_event(
                            ChatErrorEvent(
                                detail="Image support is turned off on this server.",
                                hint=(
                                    "Enable BOR_IMAGES in the server's .env "
                                    "(and restart) to ask with an image."
                                ),
                            ).model_dump()
                        )
                        return
                    image_file = (
                        Path(settings.chat_image_dir).expanduser()
                        / request.image.rsplit("/", 1)[-1]
                    )
                    if not image_file.is_file():
                        logger.warning(
                            "chat: image question rejected (stored file "
                            "missing) question=%r image=%r",
                            request.message,
                            request.image,
                        )
                        settled = True  # terminal: the error frame settles the turn
                        yield sse_event(
                            ChatErrorEvent(
                                detail="That image is no longer available."
                            ).model_dump()
                        )
                        return
                # Phase 74 (TODO L4): the client's prior turns, mapped ONCE
                # per turn — trimmed newest-first against the settings
                # budgets, assistant turns carrying their prior thinking as
                # ``reasoning_content`` (A4). BOTH branches below (deflected
                # + grounded agent) reuse the same block; an absent/empty
                # history yields ``[]`` (the byte-identical two-message
                # request, A2).
                hist = history_to_messages(request.history, settings)

                # 1. Embed the question.
                #    Phase 67: a dead embeddings endpoint is retried before any
                #    frame has left the server — up to ``llm_retries`` restarts,
                #    a flat ``llm_retry_delay`` between attempts, one SSE
                #    ``retry`` frame per restart (the UI shows the transient
                #    "retrying" status, not an error — locked A4). The final
                #    failure keeps the EXISTING terminal ``error`` frame (the
                #    copy reads correctly after N tries); ``llm_retries=0`` is
                #    byte-identical to the pre-phase-67 single attempt.
                t0 = time.monotonic()
                max_attempts = settings.llm_retries + 1
                attempt = 1
                while True:
                    try:
                        # Phase 114 (TODO L6): the embed input is bounded to the
                        # model's per-request input cap (the chunker's 1200-char
                        # budget, env-tunable) — the FULL question still reaches
                        # the LLM prompt (prompt build + log line untouched).
                        question_vec = await llm.embed_one(
                            request.message[: settings.embed_question_max_chars]
                        )
                        break
                    except EmbeddingInputTooLargeError as e:
                        # Phase 114 (TODO L6, locked A3): a too-large input is
                        # DETERMINISTIC — retrying the same size is guaranteed
                        # to repeat — so this short-circuits the phase-67 retry
                        # loop: no ``retry`` frame, no restart, one terminal
                        # error frame with the accurate "question too long"
                        # detail + the reachability-fine hint (the
                        # "couldn't reach" copy and the retry budget stay for
                        # reachability failures only — the branch below).
                        embed_ms = int((time.monotonic() - t0) * 1000)
                        total_ms = int((time.monotonic() - started) * 1000)
                        logger.error(
                            "chat: question=%r embed_ms=%d total_ms=%d — "
                            "embedding failed (too-large): %s",
                            request.message,
                            embed_ms,
                            total_ms,
                            e,
                        )
                        settled = True  # terminal: the error frame settles the turn
                        yield sse_event(
                            ChatErrorEvent(
                                detail="Question too long — trim it and re-ask.",
                                hint=(
                                    "The app reached the embedding model fine — "
                                    "only the question length is the problem."
                                ),
                            ).model_dump()
                        )
                        return
                    except EmbeddingError as e:
                        embed_ms = int((time.monotonic() - t0) * 1000)
                        if attempt >= max_attempts:
                            total_ms = int((time.monotonic() - started) * 1000)
                            logger.error(
                                "chat: question=%r embed_ms=%d total_ms=%d — embedding failed: %s",
                                request.message,
                                embed_ms,
                                total_ms,
                                e,
                            )
                            settled = True  # terminal: the error frame settles the turn
                            yield sse_event(
                                ChatErrorEvent(
                                    detail=(
                                        "I couldn't reach the embedding model — "
                                        "please try again."
                                    )
                                ).model_dump()
                            )
                            return
                        logger.warning(
                            "chat: question=%r embedding failed (attempt %d/%d) — "
                            "retrying in %.1fs: %s",
                            request.message,
                            attempt,
                            max_attempts,
                            settings.llm_retry_delay,
                            e,
                        )
                        retries_used += 1
                        yield sse_event(
                            ChatRetryEvent(
                                attempt=attempt + 1, max_attempts=max_attempts
                            ).model_dump()
                        )
                        await asyncio.sleep(settings.llm_retry_delay)
                        attempt += 1
                embed_ms = int((time.monotonic() - t0) * 1000)

                # 2. Retrieve top-K chunks, load the owner's steering notes
                #    (phase 15), then the honesty gate (A8) picks the HIGH
                #    (grounded) or LOW (deflected) prompt + context.
                #    SEC-14-04: each step uses a short-lived session.
                try:
                    with SessionLocal() as step_db:
                        steering_notes = load_steering_notes(step_db)
                    with SessionLocal() as step_db:
                        kb_overview = load_kb_overview(step_db)
                    with SessionLocal() as step_db:
                        chunks = retrieve(step_db, request.message, question_vec)
                        # Phase 191 (D3 task 04 + D4 task 05): ONE
                        # all_labels read per turn feeds BOTH surfaces —
                        # the source-name hit flags here AND the prompt
                        # surfaces (the map rides into plan_turn →
                        # build_high_prompt's name/label attributes). The
                        # hit set is the pure D2 rule per view, then a
                        # loop over the few dozen retrieved chunks against
                        # the handful of hit sources (never a per-chunk
                        # query — task 01's question_name_hit_sources is
                        # this same pair, inlined here so the read is
                        # shared). The flags ride the chunk objects into
                        # EVERY selection call below (plan_turn's walk
                        # included); the source_name_hit_bonus itself is
                        # pulled from settings inside the selection
                        # helpers. No label rows → empty map → no flags
                        # and no name attributes → every surface is the
                        # pre-phase one; the per-turn log line is
                        # untouched (the boost is invisible to it).
                        labels = all_labels(step_db)
                        source_name_hits = {
                            view.source
                            for view in labels.values()
                            if question_name_hits(view, request.message)
                        }
                        for rc in chunks:
                            if rc.document.source in source_name_hits:
                                rc.source_name_hit = True
                        # Phase 119 (D3, LOCKED A4): the suggested-folder
                        # context — the SAME deterministic suggested walk
                        # plan_turn performs internally (one extra cheap
                        # walk, no shared state); each suggested doc's
                        # parent-folder contents pre-seed the HIGH prompt
                        # so the model reads the right file in round 1
                        # instead of walking ls drill-downs. The LOW branch
                        # ignores them (the deflected prompt stays
                        # byte-identical); empty ⇒ the phase-118 prompt is
                        # byte-identical too.
                        folder_lines = suggested_folder_lines(
                            step_db,
                            select_suggested(chunks, n=settings.suggested_docs),
                        )
                    plan = plan_turn(
                        chunks,
                        settings,
                        notes=steering_notes,
                        kb_overview=kb_overview,
                        folder_lines=folder_lines,
                        # Phase 191 (D4, task 05): the SAME single read —
                        # the HIGH prompt learns the name→docs mapping
                        # from these views (no second label query).
                        names=labels,
                    )
                except Exception:  # noqa: BLE001 — DB failure mid-turn
                    logger.exception(
                        "chat: retrieval failed question=%r total_ms=%d",
                        request.message,
                        int((time.monotonic() - started) * 1000),
                    )
                    settled = True  # terminal: the error frame settles the turn
                    yield sse_event(
                        ChatErrorEvent(
                            detail="The knowledge base went offline mid-question — is Postgres up?"
                        ).model_dump()
                    )
                    return
                # Phase 123 (task 01): the user message's content — the
                # plain question string (``image=None`` — byte-identical
                # to pre-phase) or the multimodal content list (text
                # part + image_url data URL; the image was validated
                # above). BOTH construction sites use the same build:
                # this deflected-branch list and the grounded branch's
                # ``run_agent`` call below (run_agent builds its own
                # ``[system, *history, user]`` — pinned there).
                user_content: str | list[dict[str, Any]] = build_user_content(
                    request.message, request.image, settings
                )
                messages: list[dict[str, Any]] = [
                    {"role": "system", "content": plan.system_prompt},
                    *hist,  # phase 74: the trimmed prior turns (empty by default)
                    {"role": "user", "content": user_content},
                ]

                # 3. Stream the answer (grounded, or an honest deflection).
                #    Phase 17: thinking pieces stream as ``thinking`` events
                #    ahead of the ``delta`` events (PLAN §4 extension); the
                #    kill-switch (``BOR_STREAM_THINKING=0``) suppresses the
                #    frames, not the counting.
                #    Phase 37: a grounded turn runs the agent loop instead of
                #    a bare ``chat_stream`` — its ``ToolCallPiece``s stream
                #    as ``tool`` events ahead of the answer. A deflected turn
                #    keeps the direct ``chat_stream`` (byte-identical, A8):
                #    the LOW prompt never carries tools, and with
                #    ``agent_max_rounds=0`` ``run_agent`` is a single
                #    ``tools=None`` request anyway (the kill switch).
                holder = AgentHolder()
                # Phase 192: ``AgentPiece`` is the ONE union ``run_agent``
                # yields (the phase-95 truncation piece plus the new
                # ``VerbatimBlockPiece``); the deflected path's narrower
                # ``chat_stream_retried`` union flows into it unchanged.
                answer_stream: AsyncIterator[AgentPiece]
                deflected_filter: ScaffoldingFilter | None = None
                if plan.deflected:
                    # Phase 67: the deflected stream goes through the retry
                    # primitive — a dead endpoint is restarted (SSE ``retry``
                    # frames) only before its first piece (locked A2); the
                    # grounded path stays a plain ``run_agent`` call (task 03
                    # makes IT retry internally) — its ``RetryPiece``s flow
                    # through the shared piece loop below. Phase 71: the
                    # request's content also runs through a caller-owned
                    # filter (one per request) — a scaffolding-only reply
                    # streams zero ``delta`` frames instead of raw tokens, and
                    # the filter's ``stripped_chars`` drives the recovery
                    # decision after the piece loop.
                    deflected_filter = ScaffoldingFilter()
                    answer_stream = chat_stream_retried(
                        llm,
                        messages,
                        tools=None,
                        retries=settings.llm_retries,
                        delay=settings.llm_retry_delay,
                        scaffolding=deflected_filter,
                    )
                else:
                    answer_stream = run_agent(
                        llm,
                        db_factory,  # SEC-14-04: session factory, not a long-lived session
                        system_prompt=plan.system_prompt,
                        # Phase 123 (task 01): the plain question string
                        # (image=None) or the multimodal content list
                        # (validated above) — run_agent builds its own
                        # user message from this value (see its docstring).
                        user_message=user_content,
                        seed_docs=plan.suggested_docs,  # phase 118 (A4): the suggestion tier
                        settings=settings,
                        holder=holder,
                        history=hist,  # phase 74: the same trimmed prior turns
                    )
                thinking_chars = 0
                content_chars = 0  # phase 71: the turn's visible (clean) content
                scaffold_stripped = 0  # phase 71: sum across the turn's requests
                # Phase 182 (task 05, D3): the turn's clean VISIBLE answer
                # text — the SAME pieces counted into ``content_chars``
                # (post-scaffold-filter, never ``thinking``, so a locator
                # can only ever come from the answer), kept so the ``done``
                # builder can resolve the `` ```bor `` fences the answer
                # wrote against the indexed rows. Bounded by the model's own
                # output-token budget (``max_output_tokens``) — the answer
                # already lives in memory for the duration of the turn, so
                # this adds no new cap.
                visible_parts: list[str] = []
                # Phase 160 (task 02): the transport half of the cap path — a
                # request made WITHOUT tools (the deflected turn, A8, and its
                # recovery) never produces a ``tool`` frame: a structured
                # ``tool_calls`` reply is rejected at ``_pump`` (dropped,
                # counted in a local, named by one debug line). The grounded
                # first pass keeps its current behavior (``run_agent``
                # suppresses the cap-path calls internally, and the kill
                # switch is byte-identical — unchanged, phase 45).
                tools_offered = not plan.deflected

                async def _pump(
                    pieces: AsyncIterator[AgentPiece],
                    tools_offered: bool = True,
                ) -> AsyncIterator[str]:
                    """One request's piece loop (phase 71 extraction): the
                    thinking/tool/retry/tool_result/delta handling shared by
                    the turn's first pass and — deflected path only — the one
                    bounded recovery. Behavior-preserving for the first pass
                    (pinned by the existing integration suite). Phase 95:
                    the ``ToolResultPiece`` branch emits the additive
                    ``tool_result`` SSE frame (the seventh, optional event
                    type — the A15 extension). Phase 160 (task 02):
                    ``tools_offered=False`` (a request made WITHOUT tools —
                    the deflected turn and its recovery) drops any
                    ``ToolCallPiece`` — a ``tools=None`` request cannot run a
                    call, so it is rejected at the transport (no ``tool``
                    frame, no visible markup) and counted in a local so one
                    debug line can name it. The grounded first pass keeps its
                    current behavior (``tools_offered=True``; ``run_agent``
                    suppresses the cap-path calls internally). Phase 192
                    (task 01): the ``VerbatimBlockPiece`` branch emits the
                    additive ``verbatim_block`` frame — the verbatim block a
                    successful ``show_text`` just displayed, so its panel
                    lands mid-stream instead of at ``done``."""
                    nonlocal thinking_chars, content_chars, retries_used
                    dropped_calls = 0  # phase 160: tools=None requests only
                    async for piece in pieces:  # app.rag.agent.AgentPiece
                        if isinstance(piece, ToolCallPiece):
                            if not tools_offered:
                                # Phase 160 (task 02): a tools=None request
                                # cannot run a call — reject it (no ``tool``
                                # frame, no visible markup); count it so one
                                # debug line can name it. Never executed,
                                # never rendered.
                                dropped_calls += 1
                                continue
                            # Phase 37 (PLAN §4 extension; phase 70): one SSE
                            # ``tool`` frame per model-requested call.
                            # ``argument`` is the single string the model
                            # passed — ``read``'s ``path`` (the combined
                            # ``source/path``), ``grep``'s ``pattern``,
                            # ``ls``'s ``path`` — or null (a non-string value
                            # is a model error the backend refuses, as is an
                            # omitted argument).
                            argument = piece.arguments.get(
                                "pattern" if piece.name == "grep" else "path"
                            )
                            argument = argument if isinstance(argument, str) else None
                            # Phase 131 (task 04): a range ``read``'s line
                            # range rides the SAME frame — the additive
                            # ``offset`` / ``limit`` fields (the wire's
                            # assertion surface for the owner's
                            # grep → read(offset/limit) dance). Only a
                            # well-formed range (``_whole_line_number``,
                            # the agent's own guard) lands on the frame;
                            # absent → omitted on serialization, so every
                            # non-range frame stays byte-identical to
                            # pre-phase (the ChatToolEvent serializer).
                            offset = (
                                piece.arguments.get("offset")
                                if piece.name == "read"
                                else None
                            )
                            offset = offset if _whole_line_number(offset) else None
                            limit = (
                                piece.arguments.get("limit")
                                if piece.name == "read"
                                else None
                            )
                            limit = limit if _whole_line_number(limit) else None
                            # Phase 146 (task 01): a SCOPED grep's frame
                            # carries the scope — the re-searched
                            # document's combined ``source/path`` (the
                            # grep's optional one-doc argument, PLAN §6)
                            # — as the additive ``path`` field, so the
                            # UI can show WHERE the search runs (the
                            # "looks like the LLM is repeating itself"
                            # confusion, TODO L3). ONLY ``grep`` reads it
                            # (a non-grep ``path`` argument is that
                            # tool's ``argument``, never a scope); the
                            # same string guard as ``argument`` — a
                            # non-string value is a model error the agent
                            # already refuses; the guard just yields
                            # None → the key is omitted on
                            # serialization, so every non-scoped frame
                            # keeps its exact pre-phase wire bytes (the
                            # ChatToolEvent wrap serializer).
                            scope = (
                                piece.arguments.get("path")
                                if piece.name == "grep"
                                else None
                            )
                            scope = scope if isinstance(scope, str) else None
                            yield sse_event(
                                ChatToolEvent(
                                    name=piece.name,
                                    argument=argument,
                                    offset=offset,
                                    limit=limit,
                                    path=scope,
                                ).model_dump()
                            )
                            continue
                        if isinstance(piece, RetryPiece):
                            # Phase 67: the answer stream was restarted before
                            # its first piece (locked A2) — a transient status
                            # frame, never an error. No other state changes:
                            # the thinking/clock/timeout handling is the
                            # client's job.
                            retries_used += 1
                            yield sse_event(
                                ChatRetryEvent(
                                    attempt=piece.attempt, max_attempts=piece.max_attempts
                                ).model_dump()
                            )
                            continue
                        if isinstance(piece, ToolResultPiece):
                            # Phase 95 (A15 extension, task 02): one optional
                            # ``tool_result`` frame per truncated ``read`` —
                            # emitted HERE, right where the agent loop yielded
                            # the piece: AFTER the matching ``tool`` frame and
                            # BEFORE the next model round. Additive: a
                            # non-truncated read yields no piece at all (no
                            # frame), and the other six event types are
                            # byte-identical.
                            yield sse_event(
                                ChatToolResultEvent(
                                    name=piece.name,
                                    argument=piece.argument,
                                    truncated=piece.truncated,
                                    chars_shown=piece.chars_shown,
                                    chars_total=piece.chars_total,
                                ).model_dump()
                            )
                            continue
                        if isinstance(piece, VerbatimBlockPiece):
                            # Phase 192 (task 01, A1): the mid-turn verbatim
                            # frame — the block the ``show_text`` call just
                            # displayed, sent at the moment it exists (the
                            # panel lands while the answer is still
                            # streaming instead of at ``done``). Additive:
                            # a refused call yields no piece (no frame), a
                            # ``answer_text=False`` turn offers no tool (no
                            # frame) — the frame sequence is byte-identical
                            # to pre-phase in both cases, and the ``done``
                            # frame's ``answer_text`` list and its
                            # ``verbatim_blocks=N`` count are untouched.
                            yield sse_event(
                                ChatVerbatimBlockEvent(
                                    source=piece.block.source,
                                    path=piece.block.path,
                                    title=piece.block.title,
                                    start_line=piece.block.start_line,
                                    end_line=piece.block.end_line,
                                    total_lines=piece.block.total_lines,
                                    text=piece.block.text,
                                    truncated=piece.block.truncated,
                                    locator=piece.block.locator,
                                ).model_dump()
                            )
                            continue
                        if piece.kind == "thinking":
                            thinking_chars += len(piece.text)
                            if settings.stream_thinking:
                                yield sse_event(ChatThinkingEvent(text=piece.text).model_dump())
                        else:
                            content_chars += len(piece.text)
                            # Phase 182 (task 05): the same clean piece kept
                            # as TEXT for the done-time locator pass.
                            visible_parts.append(piece.text)
                            yield sse_event({"type": "delta", "text": piece.text})
                    if dropped_calls:
                        logger.debug(
                            "chat: %d tool call(s) from a tools=None request "
                            "rejected, not shown (phase 160)",
                            dropped_calls,
                        )

                try:
                    async for frame in _pump(answer_stream, tools_offered=tools_offered):
                        yield frame
                    if plan.deflected and deflected_filter is not None:
                        scaffold_stripped = deflected_filter.stripped_chars
                        # Phase 71: the deflected reply's visible content was
                        # wiped by the filter (the scaffolding was the whole
                        # "answer") — the ONE bounded recovery: the same
                        # messages with the correction folded into the single
                        # system prompt, ``tools=None``, a FRESH filter, the
                        # same phase-67 retry budget, streamed through the
                        # same piece loop. A round with real visible content
                        # needs no recovery (the clean content stands).
                        if content_chars == 0 and deflected_filter.stripped_chars > 0:
                            logger.warning(
                                "chat: deflected reply was pure tool-scaffolding "
                                "(%d chars stripped) — running the one bounded "
                                "recovery",
                                deflected_filter.stripped_chars,
                            )
                            recovery_filter = ScaffoldingFilter()
                            recovery_stream = chat_stream_retried(
                                llm,
                                [
                                    {
                                        "role": "system",
                                        "content": (
                                            plan.system_prompt + "\n"
                                            + CORRECTION_INSTRUCTION
                                        ),
                                    },
                                    *messages[1:],
                                ],
                                tools=None,
                                retries=settings.llm_retries,
                                delay=settings.llm_retry_delay,
                                scaffolding=recovery_filter,
                            )
                            async for frame in _pump(recovery_stream, tools_offered=False):
                                yield frame
                            scaffold_stripped += recovery_filter.stripped_chars
                            if content_chars == 0:
                                # The second empty reply is terminal (at most
                                # one recovery per turn) — the dedicated error
                                # frame below (no done, no query_log row).
                                logger.warning(
                                    "chat: the recovery reply was still empty "
                                    "(scaffold_stripped=%d) — settling with a "
                                    "malformed-reply error",
                                    scaffold_stripped,
                                )
                                raise MalformedReplyError(
                                    "the deflected model answered in raw "
                                    "tool-scaffolding twice in a row — no "
                                    "clean answer to stream"
                                )
                    else:
                        # Grounded turns: the agent's rounds + forced final +
                        # any recovery already accumulated the turn total on
                        # the holder (the deflected fallback is 0 — the agent
                        # never runs, so this branch is grounded-only).
                        scaffold_stripped = holder.scaffold_stripped
                except MalformedReplyError as e:
                    # Phase 71: the recovery policy's terminal signal —
                    # caught BEFORE the generic LLMError handler (it
                    # subclasses it), so the dedicated copy reaches the UI;
                    # the generic "dropped the connection" copy stays for
                    # transport failures.
                    logger.error(
                        "chat: malformed reply after the one bounded recovery "
                        "question=%r total_ms=%d — %s",
                        request.message,
                        int((time.monotonic() - started) * 1000),
                        e,
                    )
                    settled = True  # terminal: the error frame settles the turn
                    yield sse_event(
                        ChatErrorEvent(
                            detail="The model returned a malformed reply — please try again."
                        ).model_dump()
                    )
                    return
                except LLMError as e:
                    logger.error(
                        "chat: LLM stream failed question=%r total_ms=%d — %s",
                        request.message,
                        int((time.monotonic() - started) * 1000),
                        e,
                    )
                    settled = True  # terminal: the error frame settles the turn
                    yield sse_event(
                        ChatErrorEvent(
                            detail="The chat model dropped the connection — try again?"
                        ).model_dump()
                    )
                    return
                except Exception:  # noqa: BLE001 — a tool call hit the DB mid-stream
                    # Phase 37: tool execution (the drill-down ``ls`` /
                    # ``read`` / ``grep`` lookups) runs inside the stream
                    # now; a mid-turn DB failure gets the same structured
                    # ``error`` event as the pre-stream retrieval path.
                    logger.exception(
                        "chat: tool execution failed question=%r total_ms=%d",
                        request.message,
                        int((time.monotonic() - started) * 1000),
                    )
                    settled = True  # terminal: the error frame settles the turn
                    yield sse_event(
                        ChatErrorEvent(
                            detail="The knowledge base went offline mid-question — is Postgres up?"
                        ).model_dump()
                    )
                    return

                # 4. Durable record + required per-turn log line (PLAN §9).
                #    Phase 37: the agent's read documents join the
                #    retrieval's — deduped by (source, path), order
                #    preserved (a read doc not already suggested stays
                #    last). Phase 118: the retrieval arrives in two
                #    tiers — the suggested docs
                #    (``plan.suggested_docs``, the summary-seeded
                #    "start here" tier, no floor, A3) and the related
                #    docs (``plan.related_docs``, rank 6+ after the
                #    suggested set). The DURABLE record keeps the full
                #    retrieval (118-A3: query_log records retrieval,
                #    not citations — even on deflected turns, where the
                #    weak hits are suggested). Phase 112/119:
                #    done.sources is the CITATION surface — READ DOCS
                #    ONLY (phase 119, LOCKED A1, owner decision
                #    2026-09-16; the phase-118 A4 suggested+read union
                #    is RETIRED): on grounded turns it carries
                #    ``holder.read_docs`` deduped by (source, path),
                #    read order — a zero-read grounded turn (the
                #    summary-only fast path) chips nothing, an accepted
                #    consequence (the answer prose names the doc per the
                #    phase-119 cite discipline, A5) — and [] on
                #    deflected ones (a deflected answer cites nothing;
                #    the weak hits stay in the durable record).
                #    Phase 124 (TODO L3) gives that rule its single
                #    documented exception — the suggested-tier IMAGE
                #    docs (``merge_cited_docs``, below): a seed block
                #    IS an image doc's entire content, so "suggested"
                #    is equivalent to "read" for it and it is
                #    auto-cited in suggestion order AFTER the read
                #    docs, deduped (a read image doc stays at its read
                #    position, once). Text docs keep the read-docs-only
                #    rule — a zero-read text turn still chips nothing.
                #    A cancelled turn (the generator closed by the
                #    consumer) never reaches this step — no query_log row.
                cited_docs = merge_cited_docs(
                    holder.read_docs, plan.suggested_docs, plan.deflected
                )
                cited_seen = {(d.source, d.path) for d in cited_docs}
                record_docs: list[Document] = []
                seen: set[tuple[str, str]] = set()
                for doc in [*plan.suggested_docs, *plan.related_docs, *holder.read_docs]:
                    key = (doc.source, doc.path)
                    if key not in seen:
                        seen.add(key)
                        record_docs.append(doc)
                source_paths = [f"{d.source}/{d.path}" for d in record_docs]
                total_ms = int((time.monotonic() - started) * 1000)
                try:
                    with SessionLocal() as log_db:
                        log_db.add(
                            QueryLog(
                                question=request.message,
                                top_score=plan.top_score,
                                fts_hits=plan.fts_hits,
                                chunk_hits=len(chunks),
                                deflected=plan.deflected,
                                sources=", ".join(source_paths),
                                latency_ms=total_ms,
                            )
                        )
                        log_db.commit()
                except Exception:  # noqa: BLE001 — the answer already went out
                    logger.exception("chat: failed to write query_log question=%r", request.message)

                # Phase 182 (task 05, D2/D9): resolve the turn's verbatim
                # display blocks — the ``show_text`` tool's blocks
                # (``holder.shown_texts``, read AFTER the stream exactly
                # like ``shown_images``) fused with the blocks the answer's
                # own `` ```bor `` locator fences name, over the CLEAN
                # visible text accumulated above (never ``thinking``).
                # A DEFLECTED turn resolves NOTHING (D9 — a fence
                # hallucinated into a disclaimer must not leak KB text
                # through the turn that just said it had nothing), which is
                # also why this is the only branch with a resolved holder.
                # A DB failure here must not break a stream whose answer
                # already went out (the query_log precedent above): the
                # turn degrades to NO blocks, the frame simply omits the
                # key, and the client renders each fence in its honest
                # unresolved state — never the model's own bytes.
                blocks: list[VerbatimBlockRef] = []
                if not plan.deflected:
                    try:
                        with SessionLocal() as verbatim_db:
                            resolved = resolve_answer_blocks(
                                verbatim_db,
                                "".join(visible_parts),
                                holder.shown_texts,
                                settings,
                            )
                        blocks = [VerbatimBlockRef(**asdict(b)) for b in resolved]
                    except Exception:  # noqa: BLE001 — the answer went out
                        logger.exception(
                            "chat: verbatim block resolution failed question=%r",
                            request.message,
                        )
                        blocks = []

                logger.info(
                    "question=%r embed_ms=%d top_score=%.3f fts_hits=%d summary_hits=%d "
                    "suggested=%d tuning=%d "
                    "kb_chars=%d history_msgs=%d threshold=%.2f deflected=%s sources=%r "
                    "thinking=%s thinking_chars=%d tool_calls=%d rounds=%d "
                    "cap_hit=%d total_ms=%d retries=%d scaffold_stripped=%d "
                    "verbatim_blocks=%d",
                    request.message,
                    embed_ms,
                    plan.top_score,
                    plan.fts_hits,
                    plan.summary_hits,
                    len(plan.suggested_docs),  # phase 118: the seeded suggestion tier size
                    plan.tuning_count,
                    plan.kb_chars,
                    len(hist),
                    settings.relevance_threshold,
                    plan.deflected,
                    source_paths,
                    # phase 178 (A5): the effective chat thinking
                    # level, trimmed, `-` when empty — the level this
                    # turn ran at; the summary level is a one-shot
                    # concern and gains no field here.
                    settings.llm_chat_thinking_level.strip() or "-",
                    thinking_chars,
                    holder.tool_calls,
                    holder.rounds,
                    int(holder.cap_reached),
                    total_ms,
                    retries_used,
                    scaffold_stripped,
                    # phase 182 (task 05): the blocks ACTUALLY SENT (after
                    # the canonical-locator dedupe and the
                    # ``show_text_max_blocks`` cut) — 0 on every turn
                    # without them, deflected ones included (PLAN §9
                    # additive revision).
                    len(blocks),
                )
                settled = True  # terminal: the done frame settles the turn
                # Phase 112 (A8 revised, TODO L2): a deflected turn cites
                # nothing — done.sources is [] (the UI chips every entry
                # as a citation; the weak hits are scored docs, not
                # citations). The retrieval stays durably recorded above
                # (query_log.sources + the log line — observability
                # unchanged).
                # Phase 118 (A3) / phase 119 (A1): done.related carries
                # the related tier — the ranked documents beyond the
                # suggested set (rank 6+ after the contiguous top-N
                # suggestion, capped by related_max_docs in the tiering),
                # deduped against the cited list, the same (source,
                # path) pattern as cited_docs — which is the READ DOCS
                # only now: a read related doc is a chip, never a
                # "nearby doc" (unchanged intent, keyed on read docs
                # since phase 119). The UI renders the row as the
                # de-emphasized related-docs row, never a citation chip;
                # old clients ignore the field.
                # Phase 122 (task 05): both tiers build their refs
                # through the shared helper — a ref for an image doc
                # carries the optional ``image_url`` (the bytes route),
                # a text doc's stays byte-identical to pre-phase (the
                # key is omitted, never null).
                cited_refs: list[SourceRef] = []
                if not plan.deflected:
                    cited_refs = [source_ref_with_image(d) for d in cited_docs]
                related_refs = [
                    source_ref_with_image(d)
                    for d in plan.related_docs
                    if (d.source, d.path) not in cited_seen
                ]
                # Phase 124 (task 04): the answer's first-class image —
                # the shown document's ref (the LLM's ``show_image``
                # pick, task 02's ``holder.shown_images``), built by the
                # SAME shared ref builder (source/path/title +
                # ``image_url``). ``None`` on every other turn — a
                # deflected turn runs no agent (``shown_images`` is
                # always empty) and an image-less grounded turn never
                # called the tool — and ``None`` is DROPPED on
                # serialization, so those frames stay byte-identical to
                # pre-phase (no key, no null).
                answer_ref = (
                    source_ref_with_image(holder.shown_images[0])
                    if holder.shown_images
                    else None
                )
                yield sse_event(
                    ChatDoneEvent(
                        deflected=plan.deflected,
                        sources=cited_refs,
                        related=related_refs,
                        suggestions=plan.suggestions,
                        answer_image=answer_ref,
                        # Phase 160 (task 03): the turn reports the cap
                        # hit — ``True`` only when the agent's round
                        # cap forced the final no-tools answer.
                        # Deflected turns run no agent, so the default
                        # (``False`` → ABSENT on serialization) is
                        # correct by construction.
                        budget_exhausted=holder.cap_reached,
                        # Phase 182 (task 05): the answer's verbatim display
                        # blocks, resolved above in DISPLAY order (answer
                        # order first, then the tool blocks the answer never
                        # located). ``blocks or None`` — an empty list is
                        # ``None`` here and the serializer drops the key
                        # either way, so a block-free frame keeps its exact
                        # pre-phase bytes (LOCKED A9).
                        answer_text=blocks or None,
                    ).model_dump()
                )
            finally:
                # Phase 48 (owner-locked): a cancelled turn — the SSE
                # consumer went away before any terminal frame — settles
                # with one warning line and skips query_log entirely (the
                # write above is simply never reached when the generator is
                # closed). The finally must not yield (GeneratorExit
                # handling).
                if not settled:
                    logger.warning(
                        "chat: turn cancelled question=%r total_ms=%d",
                        request.message,
                        int((time.monotonic() - started) * 1000),
                    )
        finally:
            sem.release()
            _chat_active -= 1

    return StreamingResponse(stream(), media_type="text/event-stream", headers=SSE_HEADERS)