"""POST /api/chat — a RAG chat turn streamed over SSE (PLAN §3/§4). Flow (LOCKED A7/A15): embed the question → hybrid retrieval (cosine top-N ∪ Postgres FTS top-N, RRF-fused) → the **honesty gate** → locked persona prompt (PLAN §6) → ``turbo`` streamed as ``delta`` events → final ``done`` event (``deflected``, ``sources``, ``suggestions``) + ``query_log`` row + the per-turn log line (PLAN §9). Mid-stream failures become a structured ``error`` event; a pre-stream DB outage is a plain 503 JSON. Thinking (phase 17, PLAN §4 extension, owner permission 2026-08-23): the model's reasoning arrives ahead of the answer and is streamed as ``thinking`` events before the ``delta`` events of the same turn. Each turn's thinking is counted in the per-turn log line (``thinking_chars=N``); ``BOR_STREAM_THINKING=0`` suppresses the ``thinking`` frames (the pieces are still counted). Honesty gate (A8, revised 2026-09-14): LOW — deflection — only when ``best_cosine < BOR_RELEVANCE_THRESHOLD`` **and** (``fts_hits == 0`` or ``best_cosine < BOR_LEXICAL_SUPPORT_FLOOR``). A lexical hit alone (cosine < lexical_support_floor) no longer promotes to HIGH — the vector signal must corroborate the lexical match (A8 revised 2026-09-14, the "Mongolia" fix). HIGH fires when ``best_cosine >= threshold`` OR (fts>0 AND cosine >= lexical_support_floor). Deflection mode carries weak-hit *titles only* (never document content) plus deterministic "Maybe try" chips, and the ``done`` event / ``query_log`` row record ``deflected=true``, the weak score and the ``fts_hits`` count. A deflected turn cites nothing: the ``done`` frame's ``sources`` is ``[]`` (``done.sources`` is the citation surface — the UI chips every entry as "the answer used this" — and weak hits are scored docs, not citations, TODO L2), while ``query_log.sources`` and the per-turn log line keep recording the retrieval for tuning (phase 112, A8 revised). Steering (phase 15): the owner's stored tuning notes are loaded per turn (oldest first) and injected into the system prompt as a ```` section — both the HIGH and the LOW prompt carry it. The per-turn log line records ``tuning=N`` (the number of injected notes). Summaries (phase 30; phase 118 redefinition): a lite-model summary chunk's parent *is* the source document, so a summary hit resolves to the full source document through the unchanged chunk→document mapping (A7 revised). ``TurnPlan.summary_hits`` counts the hit chunks with ``is_summary`` whose parent document is in the SUGGESTED set (phase 118: redefined from the phase-113 cited set — the suggestion tier is the seeded context now), and the per-turn log line records ``summary_hits=N`` after ``fts_hits`` (PLAN §9 line extension; phase 118 adds ``suggested=N`` after ``summary_hits=N``). Summary seed context (phase 118, TODO L3 — the owner directive that re-revises A7, LOCKED A6): a grounded turn's ```` section seeds the top-5 suggested documents' SUMMARIES (never their full texts) — the "start here if these summaries seem right to you" starting point — and the LLM extends its context by ``read``-ing what it needs (the capped ``read`` tool is the ONLY full-text path). Both tiers are computed once per turn, for BOTH branches: ``suggested_docs`` (no floor, A3) and ``related_docs`` (rank 6+, the done frame's row). ``done.sources`` is the citation surface — suggested + agent-read, deduped (A4: a grounded turn always shows chips); ``query_log.sources`` and the per-turn log line keep recording the full retrieval (suggested + related + read, LOCKED A3). The deflected branch's prompt is byte-identical (weak-hit titles only — A8 untouched); its TurnPlan still carries both tiers for the durable record. KB overview (phase 31): the lite-generated outline of the knowledge base (single ``kb_overview`` row) is read per turn (one indexed PK lookup — no LLM call) and injected into **both** prompts as the ```` section, ordered ```` → ```` → ```` → mode body. With an empty row the prompts stay byte-identical to the pre-phase text (phase 15 convention); ``TurnPlan.kb_chars`` records the length of the stored outline (0 when absent) and the per-turn log line records ``kb_chars=N`` after ``tuning=N`` (PLAN §9 line extension). Agent document tools (phase 37, PLAN §4 extension, owner permission 2026-08-26; phase 45 removed the per-tool budgets — owner permission 2026-08-27; phase 70 aligned the surface to the harness-trained ``ls`` / ``read`` / ``grep`` — owner permission 2026-09-03): a **grounded** turn (``not plan.deflected``) no longer streams a bare ``chat_stream`` — it runs the agent loop (``app.rag.agent.run_agent``), which offers the model the three server-side tools ``ls`` / ``read`` / ``grep`` for the whole turn (as many calls as the model wants, re-lists and re-greps included) until it answers or the round cap (``BOR_AGENT_MAX_ROUNDS``, default 10) forces one final no-tools answer. Each model-requested call streams as an SSE ``tool`` event — ``{"type": "tool", "name": …, "argument": … | null}`` — ahead of the answer's ``delta`` frames: ``argument`` is the single string the model passed — ``read``'s ``path`` (the combined ``source/path``), ``grep``'s ``pattern``, ``ls``'s ``path`` — or null (a non-string value — a model error the backend refuses — and an omitted argument both yield null). Tool-result frames (phase 95, ``TODO.md`` L5 — A15 extension, owner permission 2026-09-10; the event-type list grows from six to SEVEN, ``tool_result`` among them; PLAN.md is being redone by the owner): a ``read`` whose document is longer than ``BOR_READ_MAX_CHARS`` streams, AFTER the matching ``tool`` frame (the line is already on screen — the marker lands a beat later, the phase-37/48 tool-line timing is untouched) and BEFORE the next model round, exactly one optional ``tool_result`` event — ``{"type": "tool_result", "name": "read", "argument": …, "truncated": true, "chars_shown": N, "chars_total": M}`` — the additive truncation notice the UI turns into the "(truncated — showing N of M chars)" marker on the Reading line. A non-truncated read streams NO such frame (one frame = one noteworthy event), deflected turns never stream one (the agent never runs, A8), and every pre-existing frame is byte-identical — clients that do not know the type ignore it. ``done.sources``, ``query_log.sources`` and the per-turn log line all report the same combined source list (retrieval docs + the agent's read docs, deduped by ``(source, path)``, order preserved — a grep adds no source; it is a locator, locked A5), and the log line records ``tool_calls=N`` after ``thinking_chars=N`` (PLAN §9 line extension — ``N`` counts executed tool calls; rejected calls do not count). **Deflected turns keep the direct ``chat_stream`` — byte-identical to the pre-phase path (A8):** the LOW prompt never carries tools, and with ``agent_max_rounds`` at **0** ``run_agent`` makes exactly one ``tools=None`` request, reproducing the pre-phase behavior (the kill switch). LLM retries (phase 67, ``BOR_LLM_RETRIES`` / ``BOR_LLM_RETRY_DELAY``, owner-locked 2026-09-01): when the aipi endpoint dies before a request has streamed its first output frame (locked A2), the request is restarted — up to ``llm_retries`` times (default 3), a flat ``llm_retry_delay`` (default 5 s) between attempts. Every restart is announced with an SSE ``retry`` frame (``{"type": "retry", "attempt": n, "max_attempts": N}`` — the attempt about to be tried, 1-based, ahead of the pre-retry wait) so the UI can show the transient "Communication interrupted — retrying (n of N)…" status (locked A4); exhaustion and any failure after the first frame keep the existing terminal ``error`` frames. The pre-stream question embedding retries on the same budget, and the deflected answer stream goes through ``chat_stream_retried`` (the agent loop retries per round — task 03). The per-turn log line records ``retries=N`` after ``total_ms=N`` (0 when nothing was retried — the field is uniform across all turn shapes). Tool-scaffolding guardrail (phase 71, deterministic only — owner permission 2026-09-03: "deterministic guardrails only right now, forget using a model for that"): the raw chat-template tokens ``<|tool_call_start|>…<|tool_call_end|>`` the ``lite`` model sometimes emits as plain answer text can never reach the user. The DEFLECTED path's request runs ``delta.content`` through a caller-owned ``ScaffoldingFilter`` (content only — thinking stays raw); when the filter wipes the whole reply (visible content 0, ``stripped_chars > 0``) the turn gets exactly ONE bounded recovery: the same messages with ``agent.CORRECTION_INSTRUCTION`` folded into the single system prompt, ``tools=None``, a fresh filter, the same phase-67 retry budget, streamed through the same piece loop (extracted as the inner ``_pump`` helper). A recovery that also comes back empty — and any grounded round where the recovery policy in ``run_agent`` fails — settles with the dedicated structured ``error`` frame (``MalformedReplyError`` caught before the generic ``LLMError`` handler): no ``done``, no ``query_log`` row, byte-for-byte the existing terminal-error shape. The per-turn log line records ``scaffold_stripped=N`` after ``retries=N`` — the sum across the turn's requests (grounded: the agent's rounds + forced final + any recovery, via the holder; deflected: this turn's filters), 0 on clean turns (the field is uniform, the phase-67 ``retries=N`` pattern); the recovery does not bump ``retries=N`` (it is not a phase-67 endpoint-retry). Chat history (phase 74, TODO L4, owner-locked A2/A3/A4 2026-09-08): ``POST /api/chat`` accepts an optional ``history`` — the client's prior turns, oldest first (the ``bor.chat.v1`` record minus the current question; the endpoint stays stateless per A10 — nothing is stored). It is mapped ONCE per turn by :func:`app.rag.prompts.history_to_messages` — trimmed newest-first against the settings budgets (``history_max_turns`` / ``history_max_chars``; a capped-out turn is dropped whole, never truncated) — and fed to the model on BOTH turn branches: the deflected path splices it between the system prompt and the current user message (the phase-71 recovery still rebuilds from ``messages[1:]`` — unchanged), and the grounded agent receives it as ``run_agent(..., history=hist)``. Prior brain turns' thinking travels as ``reasoning_content`` on the assistant message (the preserve- thinking wire convention, A4). The per-turn log line records ``history_msgs=N`` after ``kb_chars=N`` (0 when the request carries no history — the two-message request stays byte-identical). """ from __future__ import annotations import asyncio import json import logging import time from collections.abc import AsyncIterator, Sequence from dataclasses import dataclass, field from typing import Any from fastapi import APIRouter, Depends from fastapi.responses import JSONResponse, StreamingResponse from sqlalchemy.orm import Session from app.api.steering import load_steering_notes from app.config import Settings, get_settings from app.core.auth import require_user from app.db import SessionLocal, db_available from app.models import Document, QueryLog from app.rag.agent import ( CORRECTION_INSTRUCTION, # phase 71: the harness-owned recovery line AgentHolder, MalformedReplyError, # phase 71: the recovery policy's terminal signal run_agent, ) from app.rag.llm import ( EmbeddingError, EmbeddingInputTooLargeError, # phase 114: the deterministic too-large failure LLMClient, LLMError, RetryPiece, # phase 67: one LLM request restart (an SSE retry frame) StreamPiece, # type of the answer pieces streamed by the agent loop ToolCallPiece, # phase 37: one model-requested tool call ToolResultPiece, # phase 95: one truncated tool result (SSE frame = task 02) chat_stream_retried, # phase 67: the retry-before-first-piece primitive ) from app.rag.overview import load_kb_overview from app.rag.prompts import build_deflect_prompt, build_high_prompt, history_to_messages from app.rag.retriever import ( RetrievedChunk, retrieve, select_related, select_suggested, weak_hit_titles, ) from app.rag.scaffolding import ScaffoldingFilter # phase 71: the streaming filter from app.rag.suggestions import derive_suggestions from app.schemas import ( ChatDoneEvent, ChatErrorEvent, ChatRequest, ChatRetryEvent, ChatThinkingEvent, ChatToolEvent, ChatToolResultEvent, SourceRef, ) logger = logging.getLogger("app.chat") router = APIRouter(tags=["chat"]) #: Streaming hints: no proxy buffering, no client caching (PLAN A15). SSE_HEADERS = {"Cache-Control": "no-cache", "X-Accel-Buffering": "no"} #: Concurrency cap for /api/chat (SEC-14-04, task 03). A module-level #: counter for the fast-path pre-check and a Semaphore for the real gate. _chat_active: int = 0 _chat_semaphore: asyncio.Semaphore | None = None def _get_chat_semaphore() -> asyncio.Semaphore: """Lazy-initialize the chat concurrency semaphore (SEC-14-04). The semaphore is initialized on first use so that ``get_settings()`` is called with the correct environment — never at import time. """ global _chat_semaphore if _chat_semaphore is None: settings = get_settings() _chat_semaphore = asyncio.Semaphore(max(1, settings.chat_max_concurrent)) return _chat_semaphore _llm: LLMClient | None = None def get_llm() -> LLMClient: """Shared LLM client (FastAPI dependency so tests can override it).""" global _llm if _llm is None: _llm = LLMClient(get_settings()) return _llm def sse_event(payload: dict[str, Any]) -> str: """Serialize one SSE frame: ``data: \\n\\n`` (PLAN §4).""" return f"data: {json.dumps(payload, ensure_ascii=False)}\n\n" @dataclass class TurnPlan: """What one chat turn sends to the LLM and reports on ``done``.""" top_score: float # best cosine across candidates (query_log.top_score) fts_hits: int # lexical (OR-tsquery) candidates matched deflected: bool system_prompt: str # The summary-seeded suggestion tier (phase 118, A6: top-N distinct # documents, NO floor, A3 — the HIGH prompt carries their SUMMARIES; # the done frame's citation surface + run_agent's seed_docs, A4). suggested_docs: list[Document] # Rank 6+ after the suggested set (phase 118: the next ranked docs # that are not already suggested, at most related_max_docs — the # done frame's de-emphasized "nearby docs" row, never a citation). related_docs: list[Document] = field(default_factory=list) suggestions: list[str] = field(default_factory=list) # "Maybe try" chips (deflected turns only) tuning_count: int = 0 # steering notes injected into the system prompt #: Hit chunks with ``is_summary`` whose parent document is in the #: SUGGESTED set (phase 30; redefined from the phase-113 cited set #: in phase 118; per-turn log line ``summary_hits=N``). summary_hits: int = 0 #: Length of the stored KB overview injected as the #: ```` section (phase 31; per-turn log line #: ``kb_chars=N``). 0 when no non-empty row exists. kb_chars: int = 0 def plan_turn( chunks: Sequence[RetrievedChunk], settings: Settings, notes: Sequence[str] | None = None, kb_overview: str | None = None, ) -> TurnPlan: """Apply the honesty gate (A8, revised) and assemble prompt + context. * **HIGH (grounded)** when ``best_cosine >= threshold`` **or** (``fts_hits > 0`` **and** ``best_cosine >= lexical_support_floor``): HIGH prompt seeded with the suggested documents' SUMMARIES (phase 118, A6 — never their full texts), no suggestions. A cosine exactly at the threshold is an answer — the gate is strict (``< threshold``). An FTS hit alone, without vector corroboration (cosine < lexical_support_floor), stays LOW (A8 revised 2026-09-14). * **LOW (deflected)** otherwise — including the fts>0 / cosine < floor case (the "Mongolia" case): LOW prompt (``DEFLECT_MODE``) with weak-hit titles only — never document content — plus deterministic alternative-question chips derived from those titles. ``top_score`` (stored in ``query_log``) is the best cosine, so the gate input is always a pure vector-similarity number; the lexical signal is recorded separately as ``fts_hits``. *notes* are the owner's steering notes (phase 15, oldest first): when non-empty, both the HIGH and the LOW prompt carry the ```` section; with no notes the prompts are unchanged. *kb_overview* is the stored KB outline (phase 31, one PK lookup per turn): when non-empty, both prompts carry the ```` section (between ```` and ````) and ``kb_chars`` records the outline's length; with no outline the prompts are byte-identical to the pre-phase text and ``kb_chars`` is 0. ``summary_hits`` (phase 30; phase 118 redefinition) counts the hit chunks with ``is_summary`` whose parent document is in the SUGGESTED set — both the HIGH and the LOW branch record it. Phase 118 (the summary seeding, LOCKED A3/A6): retrieval documents are tiered once, before either branch — ``suggested_docs`` are the top-N distinct parent documents in fused rank order with NO cosine floor (``settings.suggested_docs``, default 5 — the "start here" seeding: the HIGH prompt carries their SUMMARIES, never their full texts; the floor never filters, so a lexical-only hit is suggested when it ranks) and ``related_docs`` are the next ranked documents that are not already suggested (rank 6+ for the contiguous top-5 suggestion set, at most ``related_max_docs`` — the done frame's de-emphasized "nearby docs" row). On a deflected turn the weak hits are suggested too (no floor); the LOW prompt itself is unchanged (weak-hit titles only), and both tiers still ride the TurnPlan for the durable record (LOCKED A3: query_log records retrieval, not citations). ``settings.top_n_docs`` and ``settings.source_usefulness_floor`` are NOT consulted here (phase 118 retired their seeding role, A6) — they remain settings for env back-compat only. """ steering = list(notes or []) kb_text = (kb_overview or "").strip() kb_chars = len(kb_text) best_cosine = max((c.cosine for c in chunks), default=0.0) fts_hits = sum(1 for c in chunks if c.fts_hit) # Phase 118 (A3/A6): BOTH tiers, computed once, for BOTH branches — # the suggested tier (top-N, NO floor) seeds the HIGH prompt and the # agent; the related tier (rank 6+ after the suggested set) feeds # the done frame's row and the durable record. suggested = select_suggested(chunks, n=settings.suggested_docs) related_docs = select_related( chunks, {d.id for d in suggested}, settings.related_max_docs ) suggested_ids = {d.id for d in suggested} summary_hits = sum(1 for c in chunks if c.is_summary and c.document.id in suggested_ids) lexical_supported = fts_hits > 0 and best_cosine >= settings.lexical_support_floor if best_cosine >= settings.relevance_threshold or lexical_supported: return TurnPlan( best_cosine, fts_hits, False, build_high_prompt(suggested, notes=steering, kb_overview=kb_text), suggested, related_docs, [], len(steering), summary_hits, kb_chars, ) titles = weak_hit_titles(chunks) return TurnPlan( best_cosine, fts_hits, True, build_deflect_prompt(titles, notes=steering, kb_overview=kb_text), suggested, related_docs, derive_suggestions(titles, settings.suggestions), len(steering), summary_hits, kb_chars, ) @router.post("/chat") async def chat( request: ChatRequest, _user: None = Depends(require_user), # noqa: B008 # phase 79: admin or live token llm: LLMClient = Depends(get_llm), # noqa: B008 ): """One chat turn: SSE stream of ``delta`` events + a final ``done``. User-gated (phase 79): anonymous callers get 401 ``authentication required`` before any streaming — the ONLY anonymous content is the shared chats. DB sessions (SEC-14-04): no long-lived session is held across the SSE stream — every DB step (retrieval, tool execution, query_log write) uses its own short-lived session via ``SessionLocal()``. Concurrency (SEC-14-04, task 03): a semaphore limits concurrent turns; excess requests get a 503 error. """ # SEC-14-04 / task 03: concurrency cap — reject immediately if at # capacity (the fast-path pre-check; the semaphore inside stream() # is the real gate that prevents races). settings = get_settings() max_concurrent = settings.chat_max_concurrent if _chat_active >= max_concurrent: return JSONResponse( status_code=503, content={"detail": "Too many concurrent chat turns — try again."}, ) if not db_available(): return JSONResponse( status_code=503, content={ "detail": ( "The knowledge base is offline — start Postgres with " "`podman compose up -d db`, then ask again." ) }, ) started = time.monotonic() def db_factory() -> Session: """SEC-14-04: session factory for short-lived sessions.""" return SessionLocal() async def stream() -> AsyncIterator[str]: # SEC-14-04 / task 03: concurrency accounting — increment before # the semaphore acquire (the pre-check counter is best-effort; # the semaphore is the real gate). The outer finally decrements # when the stream ends (normal or error). global _chat_active _chat_active += 1 sem = _get_chat_semaphore() await sem.acquire() try: # Phase 48: one terminal flag — ``True`` at every terminal # exit (the ``done`` yield; every ``error``-then-``return``). # The ``finally`` below logs the cancelled-turn line only when # the consumer went away before any terminal frame; it must not # yield (GeneratorExit handling). settled = False retries_used = 0 # phase 67: LLM requests restarted this turn (log line) try: settings = get_settings() # Phase 74 (TODO L4): the client's prior turns, mapped ONCE # per turn — trimmed newest-first against the settings # budgets, assistant turns carrying their prior thinking as # ``reasoning_content`` (A4). BOTH branches below (deflected # + grounded agent) reuse the same block; an absent/empty # history yields ``[]`` (the byte-identical two-message # request, A2). hist = history_to_messages(request.history, settings) # 1. Embed the question. # Phase 67: a dead embeddings endpoint is retried before any # frame has left the server — up to ``llm_retries`` restarts, # a flat ``llm_retry_delay`` between attempts, one SSE # ``retry`` frame per restart (the UI shows the transient # "retrying" status, not an error — locked A4). The final # failure keeps the EXISTING terminal ``error`` frame (the # copy reads correctly after N tries); ``llm_retries=0`` is # byte-identical to the pre-phase-67 single attempt. t0 = time.monotonic() max_attempts = settings.llm_retries + 1 attempt = 1 while True: try: # Phase 114 (TODO L6): the embed input is bounded to the # model's per-request input cap (the chunker's 1200-char # budget, env-tunable) — the FULL question still reaches # the LLM prompt (prompt build + log line untouched). question_vec = await llm.embed_one( request.message[: settings.embed_question_max_chars] ) break except EmbeddingInputTooLargeError as e: # Phase 114 (TODO L6, locked A3): a too-large input is # DETERMINISTIC — retrying the same size is guaranteed # to repeat — so this short-circuits the phase-67 retry # loop: no ``retry`` frame, no restart, one terminal # error frame with the accurate "question too long" # detail + the reachability-fine hint (the # "couldn't reach" copy and the retry budget stay for # reachability failures only — the branch below). embed_ms = int((time.monotonic() - t0) * 1000) total_ms = int((time.monotonic() - started) * 1000) logger.error( "chat: question=%r embed_ms=%d total_ms=%d — " "embedding failed (too-large): %s", request.message, embed_ms, total_ms, e, ) settled = True # terminal: the error frame settles the turn yield sse_event( ChatErrorEvent( detail="Question too long — trim it and re-ask.", hint=( "The app reached the embedding model fine — " "only the question length is the problem." ), ).model_dump() ) return except EmbeddingError as e: embed_ms = int((time.monotonic() - t0) * 1000) if attempt >= max_attempts: total_ms = int((time.monotonic() - started) * 1000) logger.error( "chat: question=%r embed_ms=%d total_ms=%d — embedding failed: %s", request.message, embed_ms, total_ms, e, ) settled = True # terminal: the error frame settles the turn yield sse_event( ChatErrorEvent( detail=( "I couldn't reach the embedding model — " "please try again." ) ).model_dump() ) return logger.warning( "chat: question=%r embedding failed (attempt %d/%d) — " "retrying in %.1fs: %s", request.message, attempt, max_attempts, settings.llm_retry_delay, e, ) retries_used += 1 yield sse_event( ChatRetryEvent( attempt=attempt + 1, max_attempts=max_attempts ).model_dump() ) await asyncio.sleep(settings.llm_retry_delay) attempt += 1 embed_ms = int((time.monotonic() - t0) * 1000) # 2. Retrieve top-K chunks, load the owner's steering notes # (phase 15), then the honesty gate (A8) picks the HIGH # (grounded) or LOW (deflected) prompt + context. # SEC-14-04: each step uses a short-lived session. try: with SessionLocal() as step_db: steering_notes = load_steering_notes(step_db) with SessionLocal() as step_db: kb_overview = load_kb_overview(step_db) with SessionLocal() as step_db: chunks = retrieve(step_db, request.message, question_vec) plan = plan_turn( chunks, settings, notes=steering_notes, kb_overview=kb_overview ) except Exception: # noqa: BLE001 — DB failure mid-turn logger.exception( "chat: retrieval failed question=%r total_ms=%d", request.message, int((time.monotonic() - started) * 1000), ) settled = True # terminal: the error frame settles the turn yield sse_event( ChatErrorEvent( detail="The knowledge base went offline mid-question — is Postgres up?" ).model_dump() ) return messages: list[dict[str, Any]] = [ {"role": "system", "content": plan.system_prompt}, *hist, # phase 74: the trimmed prior turns (empty by default) {"role": "user", "content": request.message}, ] # 3. Stream the answer (grounded, or an honest deflection). # Phase 17: thinking pieces stream as ``thinking`` events # ahead of the ``delta`` events (PLAN §4 extension); the # kill-switch (``BOR_STREAM_THINKING=0``) suppresses the # frames, not the counting. # Phase 37: a grounded turn runs the agent loop instead of # a bare ``chat_stream`` — its ``ToolCallPiece``s stream # as ``tool`` events ahead of the answer. A deflected turn # keeps the direct ``chat_stream`` (byte-identical, A8): # the LOW prompt never carries tools, and with # ``agent_max_rounds=0`` ``run_agent`` is a single # ``tools=None`` request anyway (the kill switch). holder = AgentHolder() answer_stream: AsyncIterator[ StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece ] deflected_filter: ScaffoldingFilter | None = None if plan.deflected: # Phase 67: the deflected stream goes through the retry # primitive — a dead endpoint is restarted (SSE ``retry`` # frames) only before its first piece (locked A2); the # grounded path stays a plain ``run_agent`` call (task 03 # makes IT retry internally) — its ``RetryPiece``s flow # through the shared piece loop below. Phase 71: the # request's content also runs through a caller-owned # filter (one per request) — a scaffolding-only reply # streams zero ``delta`` frames instead of raw tokens, and # the filter's ``stripped_chars`` drives the recovery # decision after the piece loop. deflected_filter = ScaffoldingFilter() answer_stream = chat_stream_retried( llm, messages, tools=None, retries=settings.llm_retries, delay=settings.llm_retry_delay, scaffolding=deflected_filter, ) else: answer_stream = run_agent( llm, db_factory, # SEC-14-04: session factory, not a long-lived session system_prompt=plan.system_prompt, user_message=request.message, seed_docs=plan.suggested_docs, # phase 118 (A4): the suggestion tier settings=settings, holder=holder, history=hist, # phase 74: the same trimmed prior turns ) thinking_chars = 0 content_chars = 0 # phase 71: the turn's visible (clean) content scaffold_stripped = 0 # phase 71: sum across the turn's requests async def _pump( pieces: AsyncIterator[ StreamPiece | ToolCallPiece | RetryPiece | ToolResultPiece ], ) -> AsyncIterator[str]: """One request's piece loop (phase 71 extraction): the thinking/tool/retry/tool_result/delta handling shared by the turn's first pass and — deflected path only — the one bounded recovery. Behavior-preserving for the first pass (pinned by the existing integration suite). Phase 95: the ``ToolResultPiece`` branch emits the additive ``tool_result`` SSE frame (the seventh, optional event type — the A15 extension).""" nonlocal thinking_chars, content_chars, retries_used async for piece in pieces: # StreamPiece | ToolCallPiece | RetryPiece if isinstance(piece, ToolCallPiece): # Phase 37 (PLAN §4 extension; phase 70): one SSE # ``tool`` frame per model-requested call. # ``argument`` is the single string the model # passed — ``read``'s ``path`` (the combined # ``source/path``), ``grep``'s ``pattern``, # ``ls``'s ``path`` — or null (a non-string value # is a model error the backend refuses, as is an # omitted argument). argument = piece.arguments.get( "pattern" if piece.name == "grep" else "path" ) argument = argument if isinstance(argument, str) else None yield sse_event( ChatToolEvent(name=piece.name, argument=argument).model_dump() ) continue if isinstance(piece, RetryPiece): # Phase 67: the answer stream was restarted before # its first piece (locked A2) — a transient status # frame, never an error. No other state changes: # the thinking/clock/timeout handling is the # client's job. retries_used += 1 yield sse_event( ChatRetryEvent( attempt=piece.attempt, max_attempts=piece.max_attempts ).model_dump() ) continue if isinstance(piece, ToolResultPiece): # Phase 95 (A15 extension, task 02): one optional # ``tool_result`` frame per truncated ``read`` — # emitted HERE, right where the agent loop yielded # the piece: AFTER the matching ``tool`` frame and # BEFORE the next model round. Additive: a # non-truncated read yields no piece at all (no # frame), and the other six event types are # byte-identical. yield sse_event( ChatToolResultEvent( name=piece.name, argument=piece.argument, truncated=piece.truncated, chars_shown=piece.chars_shown, chars_total=piece.chars_total, ).model_dump() ) continue if piece.kind == "thinking": thinking_chars += len(piece.text) if settings.stream_thinking: yield sse_event(ChatThinkingEvent(text=piece.text).model_dump()) else: content_chars += len(piece.text) yield sse_event({"type": "delta", "text": piece.text}) try: async for frame in _pump(answer_stream): yield frame if plan.deflected and deflected_filter is not None: scaffold_stripped = deflected_filter.stripped_chars # Phase 71: the deflected reply's visible content was # wiped by the filter (the scaffolding was the whole # "answer") — the ONE bounded recovery: the same # messages with the correction folded into the single # system prompt, ``tools=None``, a FRESH filter, the # same phase-67 retry budget, streamed through the # same piece loop. A round with real visible content # needs no recovery (the clean content stands). if content_chars == 0 and deflected_filter.stripped_chars > 0: logger.warning( "chat: deflected reply was pure tool-scaffolding " "(%d chars stripped) — running the one bounded " "recovery", deflected_filter.stripped_chars, ) recovery_filter = ScaffoldingFilter() recovery_stream = chat_stream_retried( llm, [ { "role": "system", "content": ( plan.system_prompt + "\n" + CORRECTION_INSTRUCTION ), }, *messages[1:], ], tools=None, retries=settings.llm_retries, delay=settings.llm_retry_delay, scaffolding=recovery_filter, ) async for frame in _pump(recovery_stream): yield frame scaffold_stripped += recovery_filter.stripped_chars if content_chars == 0: # The second empty reply is terminal (at most # one recovery per turn) — the dedicated error # frame below (no done, no query_log row). logger.warning( "chat: the recovery reply was still empty " "(scaffold_stripped=%d) — settling with a " "malformed-reply error", scaffold_stripped, ) raise MalformedReplyError( "the deflected model answered in raw " "tool-scaffolding twice in a row — no " "clean answer to stream" ) else: # Grounded turns: the agent's rounds + forced final + # any recovery already accumulated the turn total on # the holder (the deflected fallback is 0 — the agent # never runs, so this branch is grounded-only). scaffold_stripped = holder.scaffold_stripped except MalformedReplyError as e: # Phase 71: the recovery policy's terminal signal — # caught BEFORE the generic LLMError handler (it # subclasses it), so the dedicated copy reaches the UI; # the generic "dropped the connection" copy stays for # transport failures. logger.error( "chat: malformed reply after the one bounded recovery " "question=%r total_ms=%d — %s", request.message, int((time.monotonic() - started) * 1000), e, ) settled = True # terminal: the error frame settles the turn yield sse_event( ChatErrorEvent( detail="The model returned a malformed reply — please try again." ).model_dump() ) return except LLMError as e: logger.error( "chat: LLM stream failed question=%r total_ms=%d — %s", request.message, int((time.monotonic() - started) * 1000), e, ) settled = True # terminal: the error frame settles the turn yield sse_event( ChatErrorEvent( detail="The chat model dropped the connection — try again?" ).model_dump() ) return except Exception: # noqa: BLE001 — a tool call hit the DB mid-stream # Phase 37: tool execution (the drill-down ``ls`` / # ``read`` / ``grep`` lookups) runs inside the stream # now; a mid-turn DB failure gets the same structured # ``error`` event as the pre-stream retrieval path. logger.exception( "chat: tool execution failed question=%r total_ms=%d", request.message, int((time.monotonic() - started) * 1000), ) settled = True # terminal: the error frame settles the turn yield sse_event( ChatErrorEvent( detail="The knowledge base went offline mid-question — is Postgres up?" ).model_dump() ) return # 4. Durable record + required per-turn log line (PLAN §9). # Phase 37: the agent's read documents join the # retrieval's — deduped by (source, path), order # preserved (a read doc not already suggested stays # last). Phase 118: the retrieval arrives in two # tiers — the suggested docs # (``plan.suggested_docs``, the summary-seeded # "start here" tier, no floor, A3) and the related # docs (``plan.related_docs``, rank 6+ after the # suggested set). The DURABLE record keeps the full # retrieval (LOCKED A3: query_log records retrieval, # not citations — even on deflected turns, where the # weak hits are suggested). Phase 112/118: # done.sources is the CITATION surface — it carries the # suggested docs + the agent-read docs (deduped, # LOCKED A4) on grounded turns and [] on deflected # ones (a deflected answer cites nothing; the weak # hits stay in the durable record). # A cancelled turn (the generator closed by the # consumer) never reaches this step — no query_log row. cited_docs: list[Document] = [] cited_seen: set[tuple[str, str]] = set() for doc in [*plan.suggested_docs, *holder.read_docs]: key = (doc.source, doc.path) if key not in cited_seen: cited_seen.add(key) cited_docs.append(doc) record_docs: list[Document] = [] seen: set[tuple[str, str]] = set() for doc in [*plan.suggested_docs, *plan.related_docs, *holder.read_docs]: key = (doc.source, doc.path) if key not in seen: seen.add(key) record_docs.append(doc) source_paths = [f"{d.source}/{d.path}" for d in record_docs] total_ms = int((time.monotonic() - started) * 1000) try: with SessionLocal() as log_db: log_db.add( QueryLog( question=request.message, top_score=plan.top_score, fts_hits=plan.fts_hits, chunk_hits=len(chunks), deflected=plan.deflected, sources=", ".join(source_paths), latency_ms=total_ms, ) ) log_db.commit() except Exception: # noqa: BLE001 — the answer already went out logger.exception("chat: failed to write query_log question=%r", request.message) logger.info( "question=%r embed_ms=%d top_score=%.3f fts_hits=%d summary_hits=%d " "suggested=%d tuning=%d " "kb_chars=%d history_msgs=%d threshold=%.2f deflected=%s sources=%r " "thinking_chars=%d tool_calls=%d total_ms=%d retries=%d scaffold_stripped=%d", request.message, embed_ms, plan.top_score, plan.fts_hits, plan.summary_hits, len(plan.suggested_docs), # phase 118: the seeded suggestion tier size plan.tuning_count, plan.kb_chars, len(hist), settings.relevance_threshold, plan.deflected, source_paths, thinking_chars, holder.tool_calls, total_ms, retries_used, scaffold_stripped, ) settled = True # terminal: the done frame settles the turn # Phase 112 (A8 revised, TODO L2): a deflected turn cites # nothing — done.sources is [] (the UI chips every entry # as a citation; the weak hits are scored docs, not # citations). The retrieval stays durably recorded above # (query_log.sources + the log line — observability # unchanged). # Phase 118 (A3/A4): done.related carries the related # tier — the ranked documents beyond the suggested set # (rank 6+ after the contiguous top-N suggestion, capped # by related_max_docs in the tiering), deduped against # the cited list, the same (source, path) pattern as # cited_docs: an agent-read related doc is a citation, # never a "nearby doc". The UI renders it as the # de-emphasized related-docs row, never a citation chip; # old clients ignore the field. cited_refs: list[SourceRef] = [] if not plan.deflected: cited_refs = [ SourceRef(source=d.source, path=d.path, title=d.title) for d in cited_docs ] related_refs = [ SourceRef(source=d.source, path=d.path, title=d.title) for d in plan.related_docs if (d.source, d.path) not in cited_seen ] yield sse_event( ChatDoneEvent( deflected=plan.deflected, sources=cited_refs, related=related_refs, suggestions=plan.suggestions, ).model_dump() ) finally: # Phase 48 (owner-locked): a cancelled turn — the SSE # consumer went away before any terminal frame — settles # with one warning line and skips query_log entirely (the # write above is simply never reached when the generator is # closed). The finally must not yield (GeneratorExit # handling). if not settled: logger.warning( "chat: turn cancelled question=%r total_ms=%d", request.message, int((time.monotonic() - started) * 1000), ) finally: sem.release() _chat_active -= 1 return StreamingResponse(stream(), media_type="text/event-stream", headers=SSE_HEADERS)