phase: 182_verbatim_show_text
Build and Push Containers / build-and-push-app (push) Successful in 1m46s
Build and Push Containers / build-and-push-db (push) Successful in 12s

**Phase 182 — final verification pass (all 8 tasks already complete; this was verification + docs + one defect fix).**

- Verified: `app/rag/verbatim.py` (locator grammar, `splitlines()` slice, line-boundary cap, `resolve_answer_blocks`), `SHOW_TEXT_TOOL` + `shown_texts`, `SHOW_TEXT_CLAUSE`, `done.answer_text` (9 fields, key dropped when empty) + `ChatMessageRecord.answer_text` (≤50, 600 000-char cap counted in the 8 M budget), `verbatim.js` + 4 `{verbatim:true}` renderers, `Containerfile` minify step, `.env.example` 3 vars.
- `uv run python -c "…answer_text, show_text_max_chars, show_text_max_blocks"` → `True 12000 5`; each negative refuses startup naming its field; `0` accepted as inert ✅
- `bash .agents/validate.sh` → **validation OK**: 4320 unit+integration tests green, `TOTAL 6849 17 → 99%` (gate app/ >90% OK), `ruff check .` "All checks passed!", `pyright` 0 errors (re-run after my edits, 08:28).
- E2E, each in isolation `--no-cov`: `test_verbatim_show_text.py` 13 passed (panel bytes == `documents.content`, server caption, clipboard == server string, `/document.html?source=…&path=…`, honest unresolved state, reload+share, `show_image` regression, flag-off app) ✅; green: `test_answer_images` `test_harness_aligned_tools` `test_agent_document_tools` `test_markdown_tables` `test_markdown_inline_links_images` `test_svg_document_preview` `test_smoke` `test_document_viewer` `test_markdown_blockquotes` `test_chat_persistence` `test_thinking_timeline` `test_chat_rag` `test_settings_mobile_value_width`.
- **Defect found & fixed (pre-existing, not phase 182):** `tests/e2e/test_admin_settings_panel.py::test_admin_sees_groups_and_source_badges` failed — its whole-page `assert "reese" not in page.content()` tripped on the shell's onboarding chips, which follow the newest saved chats in the one shared E2E DB (a leftover `reeselink.json` question from a tool suite). Scoped the strict sweep to `.settings-shell` and kept a whole-page `reese:reese` credential needle → 12 passed with the leftovers still present.
- Docs: README fixed 2 inconsistencies — Quick Start omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the app fail-loud refuses boot) and the broken anchor `#chat--` → `#chat-`; full re-read of README (1816 lines) + AGENTS confirmed the phase-182 feature/settings/log/UI-contract text matches the code, and every `.env.example` var + every `Settings` field is documented.
- Notable: kept `resolve_answer_blocks` synchronous on the SSE terminal path (bounded indexed SELECTs, matching the existing `query_log` precedent and the phase's documented fence-lookup bound); no live-infra changes made.
- Next pending phase: none — `todo/` holds only 182 (its 8 task files are in `complete/`; the harness commits + moves `00_phase.md`).
This commit is contained in:
2026-10-15 08:33:12 -04:00
parent 814a400e6d
commit b34b8ca672
104 changed files with 10852 additions and 203 deletions
+5 -5
View File
@@ -244,16 +244,16 @@ def _refuse_embedded_credentials(key: str, value: str) -> None:
def get_catalogue(
db: Session = Depends(get_db), # noqa: B008
) -> AdminSettingsOut:
"""The whole catalogue (78 items, ``GROUPS`` order — the stable
order the E2E pins against).
"""The whole catalogue (one item per ``Settings`` field, ``GROUPS``
order — the stable order the E2E pins against).
Reads the table once, resolves the effective ``Settings`` through
``get_settings()`` (the overlay — DB-over-env, task 02), and
emits one item per field. GETs log nothing here: the phase-167
``ResponseBudgetMiddleware`` already writes the
``api method=… bytes=…`` line, and the whole catalogue must stay
under ``BOR_LOG_LARGE_RESPONSE_BYTES``' INFO bar (78 items ≈ 20 KB
— verified in task 06 with the audit script).
under ``BOR_LOG_LARGE_RESPONSE_BYTES``' INFO bar (the whole catalogue
is ≈ 20 KB — verified in task 06 with the audit script).
"""
rows = _rows(db)
effective = get_settings()
@@ -371,7 +371,7 @@ def reset_value(
@router.delete("", response_model=AdminSettingsCleared)
def reset_all(db: Session = Depends(get_db)) -> AdminSettingsCleared: # noqa: B008
"""The panel's "Clear all overrides": delete every row (the env
values win again for all 78 settings), invalidate (task 03) and
values win again for every setting), invalidate (task 03) and
report how many rows went."""
result = db.execute(delete(AppSetting).returning(AppSetting.key))
db.commit()
+97 -2
View File
@@ -227,6 +227,30 @@ pre-phase. A question image is turn-local: it is NEVER indexed as a
document, and prior turns' images are never replayed into the model's
history (``history_to_messages`` is unchanged — the text of a prior
turn that had an image stands alone, LOCKED A7).
Verbatim display (phase 182, tasks 03/05; LOCKED A1/A4/A7/A9): a turn
that showed file contents through the ``show_text`` tool (task 03, the
bytes the SERVER resolved) — or whose ANSWER carries a `` ```bor ``
locator fence (task 02's resolver, placement sugar that resolves
standalone) — gains one additive ``done`` key, ``answer_text``: the
list of :class:`~app.schemas.VerbatimBlockRef` blocks in DISPLAY order,
each carrying the exact ``documents.content`` slice plus the SERVER's
resolved 1-based, inclusive line extent and its canonical ``locator``.
The model never supplies those bytes: the endpoint accumulates the
turn's CLEAN VISIBLE answer text (the same post-scaffold-filter pieces
that feed ``content_chars`` — never ``thinking``, so a locator can only
ever come from the answer) and resolves it through
:func:`app.rag.verbatim.resolve_answer_blocks` when the ``done`` frame
is built. A deflected turn resolves NOTHING (D9 — a fence hallucinated
into a disclaimer must not leak KB text through the turn that just said
it had nothing); ``answer_text=False``, ``show_text_max_blocks=0`` and
``show_text_max_chars=0`` short-circuit inside the resolver, and a turn
with no blocks serializes WITHOUT the key (A9 — every block-free frame
keeps its exact pre-phase bytes). The per-turn log line records
``verbatim_blocks=N`` after ``scaffold_stripped=N`` — the blocks
ACTUALLY SENT (after dedupe and the ``show_text_max_blocks`` cut), 0 on
every turn without them including deflected ones (the PLAN §9 additive
revision).
"""
from __future__ import annotations
@@ -235,7 +259,7 @@ import json
import logging
import time
from collections.abc import AsyncIterator, Sequence
from dataclasses import dataclass, field
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any
@@ -283,6 +307,9 @@ from app.rag.summarizer import (
IMAGE_MIMES,
image_data_url,
) # phase 123: the phase-122 mime map + the shared data-URL helper
from app.rag.verbatim import (
resolve_answer_blocks, # phase 182: fences + shown blocks -> the done frame
)
from app.schemas import (
ChatDoneEvent,
ChatErrorEvent,
@@ -292,6 +319,7 @@ from app.schemas import (
ChatToolEvent,
ChatToolResultEvent,
SourceRef,
VerbatimBlockRef,
)
logger = logging.getLogger("app.chat")
@@ -527,6 +555,12 @@ def plan_turn(
# carries the show_image clause exactly when the
# BOR_ANSWER_IMAGES toggle is on (off = byte-identical).
show_image=settings.answer_images,
# Phase 182 (task 04, LOCKED A1/A8): the same rule for the
# show_text clause — it rides exactly when the BOR_ANSWER_TEXT
# master switch is on (the default), and an off build is
# byte-identical to pre-phase (no clause, and agent_tools
# offers four tools, not five).
show_text=settings.answer_text,
),
suggested,
related_docs,
@@ -933,6 +967,16 @@ async def chat(
thinking_chars = 0
content_chars = 0 # phase 71: the turn's visible (clean) content
scaffold_stripped = 0 # phase 71: sum across the turn's requests
# Phase 182 (task 05, D3): the turn's clean VISIBLE answer
# text — the SAME pieces counted into ``content_chars``
# (post-scaffold-filter, never ``thinking``, so a locator
# can only ever come from the answer), kept so the ``done``
# builder can resolve the `` ```bor `` fences the answer
# wrote against the indexed rows. Bounded by the model's own
# output-token budget (``max_output_tokens``) — the answer
# already lives in memory for the duration of the turn, so
# this adds no new cap.
visible_parts: list[str] = []
# Phase 160 (task 02): the transport half of the cap path — a
# request made WITHOUT tools (the deflected turn, A8, and its
# recovery) never produces a ``tool`` frame: a structured
@@ -1082,6 +1126,9 @@ async def chat(
yield sse_event(ChatThinkingEvent(text=piece.text).model_dump())
else:
content_chars += len(piece.text)
# Phase 182 (task 05): the same clean piece kept
# as TEXT for the done-time locator pass.
visible_parts.append(piece.text)
yield sse_event({"type": "delta", "text": piece.text})
if dropped_calls:
logger.debug(
@@ -1269,12 +1316,46 @@ async def chat(
except Exception: # noqa: BLE001 — the answer already went out
logger.exception("chat: failed to write query_log question=%r", request.message)
# Phase 182 (task 05, D2/D9): resolve the turn's verbatim
# display blocks — the ``show_text`` tool's blocks
# (``holder.shown_texts``, read AFTER the stream exactly
# like ``shown_images``) fused with the blocks the answer's
# own `` ```bor `` locator fences name, over the CLEAN
# visible text accumulated above (never ``thinking``).
# A DEFLECTED turn resolves NOTHING (D9 — a fence
# hallucinated into a disclaimer must not leak KB text
# through the turn that just said it had nothing), which is
# also why this is the only branch with a resolved holder.
# A DB failure here must not break a stream whose answer
# already went out (the query_log precedent above): the
# turn degrades to NO blocks, the frame simply omits the
# key, and the client renders each fence in its honest
# unresolved state — never the model's own bytes.
blocks: list[VerbatimBlockRef] = []
if not plan.deflected:
try:
with SessionLocal() as verbatim_db:
resolved = resolve_answer_blocks(
verbatim_db,
"".join(visible_parts),
holder.shown_texts,
settings,
)
blocks = [VerbatimBlockRef(**asdict(b)) for b in resolved]
except Exception: # noqa: BLE001 — the answer went out
logger.exception(
"chat: verbatim block resolution failed question=%r",
request.message,
)
blocks = []
logger.info(
"question=%r embed_ms=%d top_score=%.3f fts_hits=%d summary_hits=%d "
"suggested=%d tuning=%d "
"kb_chars=%d history_msgs=%d threshold=%.2f deflected=%s sources=%r "
"thinking=%s thinking_chars=%d tool_calls=%d rounds=%d "
"cap_hit=%d total_ms=%d retries=%d scaffold_stripped=%d",
"cap_hit=%d total_ms=%d retries=%d scaffold_stripped=%d "
"verbatim_blocks=%d",
request.message,
embed_ms,
plan.top_score,
@@ -1299,6 +1380,12 @@ async def chat(
total_ms,
retries_used,
scaffold_stripped,
# phase 182 (task 05): the blocks ACTUALLY SENT (after
# the canonical-locator dedupe and the
# ``show_text_max_blocks`` cut) — 0 on every turn
# without them, deflected ones included (PLAN §9
# additive revision).
len(blocks),
)
settled = True # terminal: the done frame settles the turn
# Phase 112 (A8 revised, TODO L2): a deflected turn cites
@@ -1360,6 +1447,14 @@ async def chat(
# (``False`` → ABSENT on serialization) is
# correct by construction.
budget_exhausted=holder.cap_reached,
# Phase 182 (task 05): the answer's verbatim display
# blocks, resolved above in DISPLAY order (answer
# order first, then the tool blocks the answer never
# located). ``blocks or None`` — an empty list is
# ``None`` here and the serializer drops the key
# either way, so a block-free frame keeps its exact
# pre-phase bytes (LOCKED A9).
answer_text=blocks or None,
).model_dump()
)
finally:
+57
View File
@@ -294,6 +294,41 @@ class Settings(BaseSettings):
#: 2026-09-10, ``TODO.md`` L5): A7's never-truncated contract is for
#: the retrieval ``<documents>`` path, which stays whole.
read_max_chars: int = 128_000
#: Feature toggle for the verbatim text-display capability (phase 182,
#: ``BOR_ANSWER_TEXT``; ``0``/``false`` = off). The single knob that
#: turns the whole feature on or off. Gates the ANSWER side ONLY: while
#: on, the grounded agent is offered the ``show_text`` display tool,
#: the HIGH prompt carries its teaching clause, a `` ```bor `` locator
#: fence in the answer is resolved server-side, and the resolved blocks
#: ride the done frame's ``answer_text`` key (and the saved record's).
#: While off, behavior is byte-identical everywhere — no ``show_text``
#: tool offered, no prompt clause, no locator resolution, no
#: ``answer_text`` key in any frame or record. THE DEFAULT IS **TRUE**
#: (owner decision 2026-10-14, LOCKED A8): unlike phase 124's
#: ``answer_images`` (default off, a cosmetic extra), an answer whose
#: code block silently differs from the file is a correctness bug, so
#: the fix ships ON and this switch is an escape hatch, not an opt-in.
answer_text: bool = True
#: Cap in characters on ONE displayed verbatim slice (phase 182,
#: ``BOR_SHOW_TEXT_MAX_CHARS``): the slice the server resolves for a
#: ``show_text`` call or a `` ```bor `` locator fence is cut at the cap
#: AT A LINE BOUNDARY, the shared ``[…truncated…]`` marker
#: (``app/rag/retriever.py``) is appended and the caption's ``end_line``
#: shrinks to the last fully-shown line — the caption never lies (D4).
#: Worst-case ``done``-frame growth is
#: ``show_text_max_chars * show_text_max_blocks`` = 60 KB against the
#: 256 KB response budget (phase 167's byte line makes it visible). A
#: char cap like ``read_max_chars`` (no tokenizer in the repo); ``0``
#: is inert (nothing displays), a negative value is a typo and fails
#: startup (validator).
show_text_max_chars: int = 12_000
#: Cap on the verbatim blocks displayed per answer (phase 182,
#: ``BOR_SHOW_TEXT_MAX_BLOCKS``): the ``show_text`` calls and
#: `` ```bor `` locator fences of one turn, in ANSWER order, past this
#: count are not displayed (D8). ``0`` is inert (no block displays —
#: the feature renders as plain prose), a negative value is a typo and
#: fails startup (the ``agent_max_rounds`` validator pattern).
show_text_max_blocks: int = 5
# --- Chat concurrency (SEC-14-04, task 03) ---
#: Maximum concurrent chat turns allowed (SEC-14-04).
@@ -705,6 +740,28 @@ class Settings(BaseSettings):
raise ValueError("read_max_chars must be >= 0 (chars)")
return v
@field_validator("show_text_max_chars")
@classmethod
def _show_text_max_chars_non_negative(cls, v: int) -> int:
"""A negative per-block cap is a typo — it would slice from the END
of the content (negative indexing) instead of failing. Fail loud at
startup, naming the field (the ``read_max_chars`` pattern, phase
182). ``0`` is legal and inert (no block displays)."""
if v < 0:
raise ValueError("show_text_max_chars must be >= 0 (chars)")
return v
@field_validator("show_text_max_blocks")
@classmethod
def _show_text_max_blocks_non_negative(cls, v: int) -> int:
"""A negative per-answer block cap is a typo — it would count down
from the END of the block list instead of failing. Fail loud at
startup, naming the field (the ``agent_max_rounds`` pattern, phase
182). ``0`` is legal and inert (no block displays)."""
if v < 0:
raise ValueError("show_text_max_blocks must be >= 0 (blocks)")
return v
@field_validator("pdf_pictorial_min_chars")
@classmethod
def _pdf_pictorial_min_chars_non_negative(cls, v: int) -> int:
+7
View File
@@ -267,6 +267,13 @@ CATALOG: dict[str, SettingMeta] = {
"read_max_chars": SettingMeta(
"read_max_chars", "Agent tools", "Read max chars"
),
"answer_text": SettingMeta("answer_text", "Agent tools", "Answer text"),
"show_text_max_chars": SettingMeta(
"show_text_max_chars", "Agent tools", "Show text max chars"
),
"show_text_max_blocks": SettingMeta(
"show_text_max_blocks", "Agent tools", "Show text max blocks"
),
# --- Import & sources ---
"import_extensions": SettingMeta(
"import_extensions", "Import & sources", "Import extensions (CSV)"
+292 -13
View File
@@ -211,6 +211,29 @@ task 04):
— its ``"No matches for …"`` line is a (counted) result. A rejected
call still consumes a *round* in the loop, so a pathological stream
that keeps emitting rejected calls is bounded by the cap (point 4).
Display tools (phase 124 ``show_image``, phase 182 ``show_text`` —
both composed by :func:`agent_tools`, each behind its own flag): they
display content FOR THE USER. ``show_image`` never adds context; the
5th tool ``show_text`` (phase 182, task 03) returns the resolved
slice back to the model ONLY as the truth of what was displayed
(LOCKED A3 — with a do-NOT-retype instruction: the model describes
the block, the user copies the server's bytes), and its block rides
``holder.shown_texts``, never ``read_docs``. The LOCKED core contract
A1–A3/A6: **the model names a location, the server shows the bytes**
— the handler resolves the slice through
:func:`app.rag.verbatim.block_for` (same ``(source, path)`` identity,
same 1-based ``splitlines()`` indexing as ``read``/``grep``, same
char-budget-cut-at-a-line-boundary), the DISPLAYED EXTENT is the
server's (an overshooting ``limit`` clamps into the result's ``lines:``
header, an ``offset`` past the end is the :data:`OFFSET_PAST_END`
teaching naming the true line count, the char cap shrinks the extent
— the same strings ``read`` uses, one copy each), images belong to
``show_image`` (a show of an ``is_image`` row is the
:data:`SHOW_TEXT_NOT_A_TEXT_DOC` refusal, the exact mirror of
:data:`NOT_AN_IMAGE_DOC` — never a silent conversion), and the inline
``` ```bor ``` locator-fence form of the same contract (phase 182 D2)
is resolved at the API layer (task 05), NOT here.
4. Every call the model emits is appended back to the message history as
the assistant tool-call message + the tool result (refusals included),
consumes one round, and the model is called again. Every tool result
@@ -317,6 +340,7 @@ from app.rag.llm import (
from app.rag.retriever import TRUNCATION_MARKER, _fold
from app.rag.scaffolding import ScaffoldingFilter
from app.rag.source_removal import resolve_source_name
from app.rag.verbatim import VerbatimBlock, block_for
logger = logging.getLogger("app.agent")
@@ -535,20 +559,104 @@ SHOW_IMAGE_TOOL: dict[str, Any] = {
}
def agent_tools(settings: Settings) -> list[dict[str, Any]]:
"""The grounded turn's tool offer list (phase 124, task 02).
#: The 5th agent tool (phase 182, task 03; LOCKED A1–A3/A6) — offered
#: ONLY when ``settings.answer_text`` (``BOR_ANSWER_TEXT``, default ON):
#: :func:`agent_tools` composes the offer list (the three base tools stay
#: :data:`AGENT_TOOLS`, byte-identical). The description IS the behavior
#: contract (the phase-124 discipline): the model NAMES a location, the
#: SERVER shows the bytes verbatim — the user copies what is displayed,
#: so the model must NOT retype it. Same OpenAI function shape as the
#: other tools: one required string ``path`` (the combined
#: ``source/path`` identity — the bare-path-will-not-resolve teaching
#: mirrored from ``read``) plus the OPTIONAL ``offset``/``limit`` line
#: range (the ``read`` wording reused, so every line-number surface in
#: the app teaches one grammar).
SHOW_TEXT_TOOL: dict[str, Any] = {
"type": "function",
"function": {
"name": "show_text",
"description": (
"Display part of a document to the user EXACTLY as stored "
"— the user sees the server's bytes and can copy them. "
"Call it when the user wants to see, copy or run a file (a "
"quadlet, a config, a script): they copy from what is "
"displayed, so do NOT retype the content yourself after "
"calling it — describe it instead. `path` is the combined "
"`source/path` string exactly as shown in `ls`. `offset` "
"(1-based first line) and `limit` (line count) are "
"optional — omit them to display the whole document; use "
"them when `grep`/`read` told you which lines matter. The "
"displayed line numbers come from the server, not from "
"your arguments: an overshooting `limit` is clamped and an "
"`offset` past the end is refused with the true line "
"count. Only text documents — for an image use "
"`show_image`. Call one tool at a time — wait for this "
"result before your next call."
),
"parameters": {
"type": "object",
"properties": {
"path": {
"type": "string",
"description": (
"The document to display, as the combined "
"`source/path` string exactly as shown in the "
"`ls` output (e.g. "
"'homelab/active/container_caddy/caddy.md'). "
"A bare document path (without the source "
"name) will not resolve. Only text documents "
"— for an image use `show_image`."
),
},
"offset": {
"type": "integer",
"description": (
"The first line to display, 1-based. Omit it "
"to display from the beginning of the "
"document. Use it to display a specific part "
"of a large document — grep reports the line "
"numbers of what you found."
),
},
"limit": {
"type": "integer",
"description": (
"The maximum number of lines to display, "
"starting at offset. Omit it to display as "
"many lines as fit under the display cap. Use "
"it to show the part the user asked about — "
"you are showing a part, not the document."
),
},
},
"required": ["path"],
},
},
}
The three base tools (:data:`AGENT_TOOLS`, byte-identical) plus
def agent_tools(settings: Settings) -> list[dict[str, Any]]:
"""The grounded turn's tool offer list (phase 124, task 02; the 5th
tool, phase 182, task 03).
The three base tools (:data:`AGENT_TOOLS`, byte-identical) plus the
TWO display tools, each gated by its own flag:
:data:`SHOW_IMAGE_TOOL` when ``settings.answer_images``
(``BOR_ANSWER_IMAGES``) is true — otherwise exactly the three base
tools (the pre-phase-124 offer, byte-identical: no 4th tool in the
wire ``tools`` array). A FRESH list on every call — callers never
mutate the result or the constants (``run_agent`` passes it straight
to the LLM request).
(``BOR_ANSWER_IMAGES``) and :data:`SHOW_TEXT_TOOL` when
``settings.answer_text`` (``BOR_ANSWER_TEXT``) — the offer order is
``ls``, ``read``, ``grep``, ``show_image``, ``show_text``. With both
flags off: exactly the three base tools (the pre-phase-124 offer,
byte-identical: no display tool in the wire ``tools`` array); with
only ``answer_text`` off it is the pre-phase-182 offer
byte-identical. A FRESH list on every call — callers never mutate
the result or the constants (``run_agent`` passes it straight to the
LLM request).
"""
tools: list[dict[str, Any]] = list(AGENT_TOOLS)
if settings.answer_images:
tools.append(SHOW_IMAGE_TOOL)
if settings.answer_text:
tools.append(SHOW_TEXT_TOOL)
return tools
#: Tool refusal texts (phase 37): rejected calls count in nothing
@@ -587,6 +695,56 @@ ALREADY_SHOWING = (
"Answer from what you have."
)
#: The ``show_text`` refusals (phase 182, task 03): the same contract as
#: every other refusal — counts in nothing, consumes a round, changes no
#: context. ``MISSING_SHOW_TEXT_ARGS`` mirrors ``MISSING_SHOW_ARGS``.
#: ``SHOW_TEXT_NOT_A_TEXT_DOC``: one ``{arg}`` field, the argument echoed
#: as passed — the row is a standalone IMAGE; image display belongs to
#: ``show_image`` (LOCKED A6 — the exact mirror of
#: :data:`NOT_AN_IMAGE_DOC`, never a silent conversion). The range
#: problems REUSE :data:`BAD_RANGE_ARGS` and :data:`OFFSET_PAST_END`
#: verbatim — the same teaching, the same echoed values, one copy of
#: each string in the codebase.
MISSING_SHOW_TEXT_ARGS = "show_text requires a string argument 'path'."
SHOW_TEXT_NOT_A_TEXT_DOC = (
"'{arg}' is a standalone image document — use show_image to display "
"images."
)
#: One ``{max}`` field (``settings.show_text_max_blocks``): the
#: per-answer block budget, enforced SERVER-side BEFORE the resolve work
#: (the model is taught the budget in the description; the handler is
#: the actual gate). ``0`` therefore disables the tool's effect while
#: keeping it offered (the house kill-switch pattern).
SHOW_TEXT_BLOCK_CAP = (
"{max} text blocks are already displayed this answer — describe them "
"instead of showing more."
)
#: One ``{locator}`` field — the canonical ``source/path[#first-last]``
#: the model just asked for: the dedupe mirror of
#: :data:`ALREADY_SHOWING`, but per BLOCK under the block budget (the
#: budget allows SEVERAL different blocks; the same block twice teaches).
SHOW_TEXT_ALREADY_SHOWN = (
"{locator} is already on screen this answer — say so instead of "
"showing it again."
)
#: The ``show_text`` truncation notice (phase 182, task 03) — appended as
#: the result's LAST line when the per-block char budget
#: (``settings.show_text_max_chars``) cut the displayed slice (the block
#: text already ends with the shared :data:`TRUNCATION_MARKER` on its own
#: line, and the ``lines:`` extent already shrank to the last line
#: actually displayed). In the :data:`READ_TRUNCATION_NOTICE` spirit it
#: names the truth the model must not paper over — the rest is NOT
#: displayed — and the way the user gets it (open the document; NO
#: ``offset``/``limit`` re-show loop is implied, only the displayed
#: slice is what the user copies). Fields: ``{source}``/``{path}``, the
#: RESOLVED identity, plain (no ``!r``).
SHOWN_TEXT_TRUNCATION_NOTICE = (
"TRUNCATED — the rest of {source}/{path} is NOT displayed. The user "
"can open the document to read it in full; describe the displayed "
"lines instead of showing it again."
)
#: The read-truncation notice (phase 95, task 01; REWRITTEN phase 131,
#: task 01) — appended, after the shared :data:`TRUNCATION_MARKER`, to
#: a ``read`` result whose document is LONGER than
@@ -1642,6 +1800,17 @@ class AgentHolder:
nothing). Read by ``app.api.chat`` after the stream for the done
frame's ``answer_image`` field (task 04) — the ``read_docs``
precedent (an API-layer read-after-stream signal).
``shown_texts`` (phase 182, task 03): the verbatim blocks
``show_text`` displayed, in CALL order, capped by
``settings.show_text_max_blocks`` (a call over the budget is refused
with :data:`SHOW_TEXT_BLOCK_CAP` and a duplicate of a shown locator
with :data:`SHOW_TEXT_ALREADY_SHOWN` — both append nothing). Each
entry is the RESOLVED
:class:`app.rag.verbatim.VerbatimBlock` — the server's bytes plus
the server's extent (the canonical ``locator`` is the dedupe key and
the client's matching key, D7). Read by ``app.api.chat`` after the
stream for the done frame's ``answer_text`` array (task 05) — the
``shown_images``/``read_docs`` read-after-stream precedent.
``read_coverage`` (phase 141, task 01): one ``(source, path,
first_line, last_line)`` record per covered span — INCLUSIVE line
numbers over the document's ``content.splitlines()``, in read
@@ -1676,6 +1845,7 @@ class AgentHolder:
read_truncations: list[tuple[str, int, int]] = field(default_factory=list)
shown_images: list[Document] = field(default_factory=list)
read_coverage: list[tuple[str, str, int, int]] = field(default_factory=list)
shown_texts: list[VerbatimBlock] = field(default_factory=list)
def _execute_tool(
@@ -2065,6 +2235,113 @@ def _execute_tool(
f"{IMAGE_DOC_MARKER}\n"
f"{doc.content}"
)
if call.name == "show_text":
# Phase 182 (task 03; LOCKED A1–A3/A6): the 5th agent tool — the
# model NAMES a location, the SERVER shows the bytes verbatim.
# Offered only when ``settings.answer_text`` (``agent_tools``);
# the handler enforces the contract server-side. Every refusal
# below appends nothing, increments nothing and consumes a
# round, like every other refusal.
raw_path = call.arguments.get("path")
arg = raw_path.strip() if isinstance(raw_path, str) else ""
if not arg:
return MISSING_SHOW_TEXT_ARGS
if not settings.answer_text:
# Defensive — the tool is NOT OFFERED while the flag is off,
# so from the model's side this is a tool that does not
# exist: the same refusal an un-offered name gets (the
# tools-off posture — "Unknown tool."), counts in nothing.
return UNKNOWN_TOOL
doc, _source, _path = _resolve_path(db, arg)
if doc is None:
# The same phase-72 "did you mean …?" teaching as the read
# branch (a refusal — not counted, no context, nothing
# displayed).
return _no_document_refusal(db, arg)
if doc.is_image:
# LOCKED A6: an image row is refused, NEVER silently
# converted — image display is ``show_image``'s job (the
# mirror of the show_image branch's NOT_AN_IMAGE_DOC).
return SHOW_TEXT_NOT_A_TEXT_DOC.format(arg=arg)
# The OPTIONAL line range — the SAME validator, the SAME echoed
# BAD_RANGE_ARGS teaching as ``read`` (one copy of each string);
# ``offset`` without ``limit`` displays to the end, ``limit``
# without ``offset`` counts from line 1 — ``slice_lines`` applies
# both one-sided semantics with ``read``'s exact indexing.
ok, range_error, offset, limit = _read_range_args(
call.arguments.get("offset"), call.arguments.get("limit")
)
if not ok:
return range_error
# The per-answer block budget — checked BEFORE the resolve work
# (and, with a 0 char budget, before ``cap_slice`` could raise):
# either zeroed kill switch refuses like a spent budget, so
# ``0`` disables the tool's EFFECT while it stays offered (D8).
block_budget = (
settings.show_text_max_blocks
if settings.show_text_max_chars > 0
else 0
)
if len(holder.shown_texts) >= block_budget:
return SHOW_TEXT_BLOCK_CAP.format(max=block_budget)
block = block_for(
db,
arg,
offset=offset,
limit=limit,
cap=settings.show_text_max_chars,
doc=doc,
)
if block is None:
# An empty document or an ``offset`` past the last line —
# the SAME OFFSET_PAST_END teaching as ``read``, the true
# line count named (the same ``len(content.splitlines())``
# expression ``read`` uses). Counts in nothing; a corrected
# retry still proceeds.
return OFFSET_PAST_END.format(
source=doc.source,
path=doc.path,
total_lines=len(doc.content.splitlines()),
offset=offset if offset is not None else 1,
)
if any(shown.locator == block.locator for shown in holder.shown_texts):
# The canonical-locator dedupe (D7): the identical block is
# already on screen — say so instead of painting it twice.
return SHOW_TEXT_ALREADY_SHOWN.format(locator=block.locator)
# Success: the block rides the holder for the done frame
# (task 05) and the call counts (executed-only, the house rule).
holder.shown_texts.append(block)
holder.tool_calls += 1
# The result contract (D5 — the ``read``/``show_image`` header
# contract the E2E mock's ``_READ_RESULT_PREFIX`` parses): a NEW
# first line (absent from ``read`` — the model must know the
# show registered AND must not retype), then the exact document
# headers ``read`` established (date line, phase-145 origin
# lines), then the SERVER's resolved extent — clamped ``limit``
# and cap-shrunk end included, the caption never lies — and the
# verbatim slice itself (A21: LLM calls are free, and returning
# the text means the model's commentary can never disagree with
# the panel).
origin = origin_for_source(db, doc.source)
result = (
"Text shown to the user — these lines are displayed "
"verbatim; describe them, do NOT retype them.\n"
f"Document {doc.source}/{doc.path}:\n"
f"date: {doc.created_at:%Y-%m-%d}\n"
f"{_origin_lines(origin)}"
f"lines: {block.start_line}-{block.end_line} of "
f"{block.total_lines}\n"
f"{block.text}"
)
if block.truncated:
# The char cap cut the slice: the block already ends with
# the shared marker on its own line — append the pinned
# truth (the rest is NOT displayed; the user opens the
# document; no re-show loop implied).
result += "\n" + SHOWN_TEXT_TRUNCATION_NOTICE.format(
source=doc.source, path=doc.path
)
return result
if call.name == "grep":
raw_pattern = call.arguments.get("pattern")
pattern = raw_pattern.strip() if isinstance(raw_pattern, str) else ""
@@ -2229,11 +2506,13 @@ async def run_agent(
# Phase 45: no per-tool budgets — the tools stay offered for the
# whole turn, bounded by the round cap. ``0`` is the no-tools kill
# switch: exactly one request with ``tools=None`` (the pre-phase-37
# path). Phase 124 (task 02): the offer list is :func:`agent_tools`
# — the three base tools (byte-identical) plus ``show_image`` only
# when ``settings.answer_images`` (``BOR_ANSWER_IMAGES``) is on;
# off (the default) the offer is element-for-element the
# pre-phase-124 list.
# path). Phase 124 (task 02) / 182 (task 03): the offer list is
# :func:`agent_tools` — the three base tools (byte-identical) plus
# ``show_image`` only when ``settings.answer_images``
# (``BOR_ANSWER_IMAGES``) and ``show_text`` only when
# ``settings.answer_text`` (``BOR_ANSWER_TEXT``, default ON); both
# off: the offer is element-for-element the pre-phase-124 list,
# ``answer_text`` alone off: the pre-phase-182 list.
max_rounds = settings.agent_max_rounds
tools: list[dict[str, Any]] | None = (
agent_tools(settings) if max_rounds > 0 else None
+105 -7
View File
@@ -66,6 +66,20 @@ backstop). The ``DEFLECT_MODE`` marker and everything else in the
prompt stay put — the E2E mock LLM keys on the marker's *presence*,
not the wording, so that contract is unchanged.
Display-tool clauses (phase 124 ``show_image``; phase 182 ``show_text``):
the **HIGH** prompt's ``<tools>`` section can carry two extra clauses
before its closing tag, each appended only when its own setting is on —
:data:`SHOW_IMAGE_CLAUSE` when ``BOR_ANSWER_IMAGES`` (default OFF) and
:data:`SHOW_TEXT_CLAUSE` when ``BOR_ANSWER_TEXT`` (default ON). The text
clause teaches the two things the phase-182 incident needed said: the
user copies from what is DISPLAYED (so never retype a file), and the
inline `` ```bor <source/path> [range] ``` `` locator fence whose EMPTY
body the app replaces with the server-resolved bytes. Both clauses
compose through one mechanism (section minus its final tag + the present
clauses + the tag), so a build with neither clause is byte-identical to
pre-phase and a ``show_image``-only build reproduces the phase-124 bytes
exactly; the LOW prompt carries neither.
Disclosed-answer contract (phase 112, task 03; owner decision
2026-09-14, roadmap confirmation — option (iii) of the three the
2026-09-15 interactive deflection test raised): that test found the
@@ -388,6 +402,61 @@ SHOW_IMAGE_CLAUSE: str = (
"in text either way."
)
#: The ``show_text`` contract clause (phase 182, task 04, LOCKED A1–A3/A6) —
#: appended INSIDE the ``<tools>`` section of the HIGH prompt, after the
#: existing three-tool copy (and after :data:`SHOW_IMAGE_CLAUSE` when that
#: rides too), ONLY when :func:`build_high_prompt` is called with
#: ``show_text=True`` (the ``BOR_ANSWER_TEXT`` master switch, via
#: ``app.api.chat``'s grounded plan — the only call site that passes it).
#: It exists because the incident this phase fixes is a PROMPT gap, not a
#: tool gap: nothing ever told the model that the user copies from what is
#: DISPLAYED, so it retyped files out of context and every retype differed
#: from the stored bytes by a character. What each sentence pins, in order:
#: (1) the display contract — the tool shows the document EXACTLY as stored
#: and the copy source is the display, not the prose; (2) when to call it —
#: the user wants to SEE, COPY or RUN a file (the owner's quadlet / config /
#: script shapes) rather than read content out of the answer; (3) the do-
#: NOT-retype rule with its reason (anything the model types may differ by
#: one character and the user cannot tell — the phase's reason for
#: existing); (4) the placement syntax — the inline `` ```bor <source/path>
#: [range] ``` `` locator fence with an EMPTY body (D1/D7: a fence is a
#: resolution request, the app supplies the text, and D3 keeps the model's
#: body out of the DOM); (5) the honesty-about-numbers rule — line numbers
#: come from ``grep``/``read``, and when unsure the range is OMITTED
#: because the displayed extent is the server's (A2); (6) the failure rule
#: — refuse out loud, never reconstruct from memory (the persona's radical
#: honesty, D3's posture on the model side); (7) the image pointer
#: (LOCKED A6: images belong to ``show_image``, never silently converted).
#: ``TOOLS_SECTION`` itself stays BYTE-IDENTICAL (the sha pins in
#: :mod:`tests.unit.test_prompt_lock` hold); a ``show_text=False`` build
#: (the default — every pre-phase call site) is byte-identical to the
#: pre-task output, and a ``show_image=True, show_text=False`` build
#: reproduces the phase-124 bytes EXACTLY. The ``<tools>`` marker the E2E
#: mock keys on is untouched either way — both clauses ride on its
#: presence, not its wording. The LOW/deflection prompt never carries the
#: clause: a deflected turn has no tools, and phase 182's D9 forbids
#: resolving a locator on a turn that just disclaimed the knowledge base.
SHOW_TEXT_CLAUSE: str = (
"`show_text` displays part of a document to the user EXACTLY as stored "
"— the user copies from what is displayed, not from your prose. Call "
"it whenever the user wants to SEE, COPY or RUN a file — a quadlet, a "
"config, a script, a unit file — instead of writing the content into "
"your answer yourself. After calling it, do NOT repeat the displayed "
"lines in your answer — describe them, explain them, point at them. "
"Anything you type yourself may differ from the file by a character, "
"and the user cannot tell. To put a displayed block exactly where you "
"want it in your answer, write a locator line on its own: three "
"backticks followed by `bor`, a space, the combined `source/path`, and "
"optionally the line range (` 12-40`). Leave its body EMPTY — the app "
"replaces it with the exact stored text. Never type the file's content "
"inside a `bor` block. Never guess line numbers — use the numbers "
"`grep` and `read` reported. When you are not sure which lines, OMIT "
"the range and display the whole document: the numbers you display are "
"the server's, never yours. If `show_text` refuses, say what you could "
"not display — do not reconstruct it from memory. For an image use "
"`show_image` instead."
)
def history_to_messages(
history: Sequence[HistoryTurn],
@@ -762,6 +831,7 @@ def build_high_prompt(
kb_overview: str | None = None,
folder_lines: Sequence[str] = (),
show_image: bool = False,
show_text: bool = False,
) -> str:
"""Grounded turn: locked persona (+ steering, + KB overview) + the
top-ranked documents' SUMMARY blocks + the ``<tools>`` instructions
@@ -771,7 +841,8 @@ def build_high_prompt(
seeds SUMMARIES, never full texts — full text enters the context
only through the capped ``read`` tool; phase 119, LOCKED A4: the
suggested-folder context lines, below; phase 124, task 03, LOCKED
A5: the optional :data:`SHOW_IMAGE_CLAUSE`, below).
A5: the optional :data:`SHOW_IMAGE_CLAUSE`, below; phase 182, task 04,
LOCKED A1–A3: the optional :data:`SHOW_TEXT_CLAUSE`, below).
Section order: ``<relevance>`` → ``<knowledge_base>`` → ``<tuning>``
→ ``<documents>`` → ``<tools>``; empty steering/overview omit their
@@ -796,6 +867,22 @@ def build_high_prompt(
never carries the clause — a deflected turn has no tools and never
shows an image.
*show_text* (phase 182, task 04, LOCKED A1–A3): when true (the caller
passes the ``BOR_ANSWER_TEXT`` master switch — the same single grounded
call site), the :data:`SHOW_TEXT_CLAUSE` rides INSIDE the same
``<tools>`` section, AFTER :data:`SHOW_IMAGE_CLAUSE` when that is
present too and always before the closing tag. The two flags compose
independently through one mechanism (the built section is
``TOOLS_SECTION`` minus its final ``</tools>`` + the present clauses,
blank-line separated, + the closing tag): ``show_image`` only
reproduces the phase-124 bytes exactly, ``show_text`` only is the
three-tool copy plus the text clause, both is the image clause then
the text clause, neither is byte-identical to pre-phase. The default
``False`` keeps every pre-existing call byte-identical. The deflection
prompt NEVER carries it — a deflected turn has no tools, and phase
182's D9 forbids resolving a ``bor`` locator on a turn whose answer
just disclaimed the knowledge base.
When at least one block is present, the section leads with the
start-here :data:`SUGGEST_INTRO` line (before the first block — the
phase-15 ``_STEERING_INTRO`` / phase-31 ``_KB_INTRO`` precedent): the
@@ -861,13 +948,24 @@ def build_high_prompt(
for part in (build_kb_section(kb_overview or ""), build_steering_section(notes or [])):
if part:
prompt += "\n" + part
# Phase 124 (task 03, LOCKED A5) / phase 182 (task 04, LOCKED A1): the
# display-tool clauses ride inside <tools> — TOOLS_SECTION's text minus
# its final </tools>, the clauses offered for this build (each on its
# own line, blank-line separated, after the existing three-tool copy),
# then the closing tag. TOOLS_SECTION itself is unedited, the
# no-clause path keeps the exact pre-task concatenation, and a
# show_image-only build reproduces the phase-124 bytes exactly.
clauses: list[str] = []
if show_image:
# Phase 124 (task 03, LOCKED A5): the clause rides inside
# <tools> — TOOLS_SECTION's text minus its final </tools>, the
# clause (on its own line, after the existing three-tool copy),
# then the closing tag. TOOLS_SECTION itself is unedited and the
# show_image=False path keeps the exact pre-task concatenation.
tools_section = TOOLS_SECTION[: -len("</tools>")] + SHOW_IMAGE_CLAUSE + "\n</tools>"
clauses.append(SHOW_IMAGE_CLAUSE)
if show_text:
clauses.append(SHOW_TEXT_CLAUSE)
if clauses:
tools_section = (
TOOLS_SECTION[: -len("</tools>")]
+ "\n\n".join(clauses)
+ "\n</tools>"
)
else:
tools_section = TOOLS_SECTION
return prompt + "\n<documents>\n" + body + "\n</documents>\n" + tools_section
+520
View File
@@ -0,0 +1,520 @@
"""Phase 182 (task 02): the verbatim locator grammar, the slicer, the resolver.
The contract in one line: **the model names a location, this module supplies
the bytes** (PLAN §6's tool-surface extension, phase 182 D1–D7). An answer
never carries a model-retyped copy of a source file; it carries a ``bor``
locator fence (````` ```bor homelab/runbooks/nginx.md#12-40 `````), and the
bytes shown to the human are read out of the indexed ``documents.content``
row by :func:`block_for` / :func:`resolve_answer_blocks`. Because the
displayed text is never generated, it cannot be paraphrased, "corrected",
or hallucinated — the phase's whole point — and the same resolver serves
both entry points (the ``show_text`` tool's own block, task 03, and the
``done``-frame pass over the finished answer, task 05).
The module is deliberately split in two halves:
* the **pure** half — :func:`canonical_locator`, :func:`fences_in`,
:func:`slice_lines`, :func:`cap_slice` — no DB, no settings, no I/O;
* the **resolver** half — :func:`document_by_identity`, :func:`block_for`,
:func:`resolve_answer_blocks` — one bare ``SELECT`` per locator and the
kill-switch gate.
**Import rule (one-way, checkable with
``grep -n "import" app/rag/verbatim.py``):** this module must NEVER import
the agent module (:mod:`app.rag.agent`) — ``agent.py`` already depends on
the pure half of this module for the ``show_text`` tool branch, so a
reverse dependency cycles. Allowed here: ``sqlalchemy``,
:class:`app.models.Document`,
:data:`app.rag.retriever.TRUNCATION_MARKER`, :class:`app.config.Settings`,
stdlib.
Nothing calls this module yet (tasks 03 and 05 wire it in), so landing it
changes no behavior anywhere.
"""
from __future__ import annotations
import re
from collections.abc import Sequence
from dataclasses import dataclass
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.config import Settings
from app.models import Document
from app.rag.retriever import TRUNCATION_MARKER
__all__ = [
"LOCATOR_LANG",
"VerbatimBlock",
"SliceResult",
"block_for",
"cap_slice",
"canonical_locator",
"document_by_identity",
"fences_in",
"resolve_answer_blocks",
"slice_lines",
]
#: The fence info-string token that marks a locator fence (D2). The
#: markdown renderer (task 06) and the copy-path builder (task 07) key on
#: this SAME constant — one grammar, one spelling, no drifting copy.
LOCATOR_LANG = "bor"
#: The range separators a locator may use between the identity and the
#: line range: a space, a colon, a bare ``#``, or ``#L`` (D7 — the grammar
#: is TOLERANT on input because the string is model-written).
_SEP = r"[ \t:#]+"
#: A line range: ``12-40``, ``12–40`` (en dash), ``L12-L40``, or a bare
#: ``12`` (one line). An optional ``L`` prefixes either bound; whitespace
#: around the dash is tolerated. Anchored at the end of the string by the
#: caller, so a range-shaped tail followed by more text is junk, not a
#: range.
_RANGE = r"(?P<range>L?[0-9]+(?:[ \t]*[-\u2013][ \t]*L?[0-9]+)?)"
#: The trailing range of a locator string (see :data:`_SEP` /
#: :data:`_RANGE`). Case-insensitive only for the ``L`` prefix.
_TRAILING_RANGE_RE = re.compile(f"{_SEP}{_RANGE}$", re.IGNORECASE)
#: The opening line of a ``bor`` locator fence: the language token,
#: whitespace, then a NON-EMPTY remainder (the locator). A bare
#: `` ```bor `` carries no locator, so it never matches and stays an
#: ordinary code block.
_FENCE_OPEN_RE = re.compile(
rf"^```{re.escape(LOCATOR_LANG)}[ \t]+(?P<locator>.*[^ \t])[ \t]*$"
)
#: The two characters a bound may use as its line-number prefix.
_LINE_PREFIX = "Ll"
@dataclass(frozen=True)
class VerbatimBlock:
"""One displayed verbatim block — the bytes plus the truth about them.
``start_line`` / ``end_line`` are **1-based and INCLUSIVE** and describe
EXACTLY the lines :attr:`text` contains *after* any character cap — a
capped block's ``end_line`` is the last line actually shown, never the
line the locator asked for (the caption must not claim lines nobody
sees). ``total_lines`` is the whole document's line count, so the
caption can say ``lines 12-40 of 512``. :attr:`text` never carries a
trailing newline (lines are joined with ``"\\n"``), and
:attr:`truncated` says whether :data:`~app.rag.retriever.TRUNCATION_MARKER`
was appended on its own last line.
:attr:`locator` is the CANONICAL form (D7) built by the factory
(:func:`block_for`), never passed in by a caller — it is the dedupe key
of :func:`resolve_answer_blocks` and the matching key the copy-path
builder uses (task 07).
"""
source: str
path: str
title: str
start_line: int
end_line: int
total_lines: int
text: str
truncated: bool = False
locator: str = ""
@dataclass(frozen=True)
class SliceResult:
"""The outcome of one line-range slice (see :func:`slice_lines`).
``first_line`` / ``last_line`` are 1-based and inclusive; ``past_end``
is the caller's refusal signal and always travels with the true
``total_lines`` (the teaching refusal names the real line count).
"""
text: str
first_line: int
last_line: int
total_lines: int
past_end: bool
def canonical_locator(raw: str) -> str | None:
"""Normalize one model-written locator string, or refuse it (D7).
Accepted forms — ``source/path`` (the ``read``/``grep`` identity
contract, split at the FIRST ``'/'``) followed by an optional separator
and line range:
========================== ==========================
input canonical output
========================== ==========================
``homelab/a.md`` ``homelab/a.md``
``homelab/a.md 12-40`` ``homelab/a.md#12-40``
``homelab/a.md:12-40`` ``homelab/a.md#12-40``
``homelab/a.md#12-40`` ``homelab/a.md#12-40``
``homelab/a.md#L12-L40`` ``homelab/a.md#12-40``
``homelab/a.md 12\u201340`` ``homelab/a.md#12-40``
``homelab/a.md 12`` ``homelab/a.md#12-12``
========================== ==========================
The canonical form is always ``source/path`` + ``#start-end`` with
plain digits — never the en dash, never the ``L``, so the fence, the
tool block and the UI share ONE string (D7).
Returns ``None`` (NOT a locator) for: empty/whitespace input, a value
with no ``'/'`` in the path part, a non-numeric or non-positive bound,
``start > end``, trailing junk after the range, or a leading ``bor``
token the caller left in. No DB access, no case folding (paths are
case-sensitive row values). A locator carrying a range also refuses an
identity containing whitespace: the fence's info string is ONE
whitespace-free identity plus the range, so a path that contains a
space is expressed as a whole-document locator (no range) or not at
all — the documented limit of the tolerant grammar.
"""
text = raw.strip()
if not text:
return None
# A caller that pasted the fence's language token in has already lost
# the identity — refuse rather than guess which token was the path.
if text.split(" ", 1)[0] == LOCATOR_LANG:
return None
match = _TRAILING_RANGE_RE.search(text)
if match is None:
identity, range_part = text, None
else:
# ``match.start()`` is where the separator begins — the identity
# is everything before it (a multi-character separator, e.g. two
# spaces, is consumed whole).
identity, range_part = text[: match.start()], match.group("range")
# The identity is one whitespace-free ``source/path`` token, and it
# never ends in a dangling separator (``a.md#`` with no range is a
# typo, not a whole document).
if not identity or any(c.isspace() for c in identity):
return None
if identity.endswith(("#", ":", "-")):
return None
if "/" not in identity:
return None
if range_part is None:
return identity
bounds = [b.strip().lstrip(_LINE_PREFIX) for b in re.split(r"[-\u2013]", range_part)]
# Defensive: :data:`_RANGE` already guarantees digit bounds, so this
# only fires if the grammar above is ever loosened — it keeps
# ``canonical_locator`` total (never an ``int()`` ValueError to a
# caller that hands it model-written text).
if any(not b.isdigit() for b in bounds):
return None
start = int(bounds[0])
end = int(bounds[1]) if len(bounds) == 2 else start
if start < 1 or end < 1 or start > end:
return None
return f"{identity}#{start}-{end}"
def fences_in(answer: str) -> list[str]:
"""Every ``bor`` locator fence's RAW locator string, in answer order.
Rules (the same shape ``markdown.js`` applies, task 06):
* the opening line is a fence whose info string is exactly
`` ```bor `` + whitespace + a NON-empty remainder — a bare
`` ```bor `` with no path is an ordinary code block, not a locator,
and is not swallowed;
* the fence closes at the next line that is nothing but backticks;
* an **unclosed** fence contributes nothing (defensive: a half-streamed
answer must not resolve a locator the model never finished) and ends
the scan, since its body runs to the end of the answer;
* the body is read and DISCARDED — this module never looks at what the
model typed inside the fence (the bytes come from the row, D1);
* nested `` ``` `` inside a body is NOT supported: the fence ends at
the first closer (documented limitation, same rule as the renderer).
Duplicates collapse keeping the FIRST occurrence.
"""
lines = answer.splitlines()
locators: list[str] = []
index = 0
while index < len(lines):
match = _FENCE_OPEN_RE.match(lines[index].strip())
if match is None:
index += 1
continue
closer = _find_fence_close(lines, index + 1)
if closer is None:
# Unclosed: the body runs to the end of the answer, so there
# is nothing further to scan and this fence yields nothing.
break
locator = match.group("locator").strip()
if locator and locator not in locators:
locators.append(locator)
index = closer + 1
return locators
def _find_fence_close(lines: Sequence[str], start: int) -> int | None:
"""Index of the fence's closing line at or after ``start``, ``None`` if unclosed."""
for offset in range(start, len(lines)):
stripped = lines[offset].strip()
if stripped.startswith("```") and not stripped.strip("`"):
return offset
return None
def slice_lines(
content: str, offset: int | None = None, limit: int | None = None
) -> SliceResult:
"""Cut ``content`` to a 1-based line range with ``read``'s exact semantics.
Same indexing (``content.splitlines()``) and the same range arithmetic
as the ``read`` tool (``app/rag/agent.py``) and ``grep``, so a line
number the model got from ``grep`` names the same line here — the one
shared coordinate system the whole phase depends on:
* ``offset`` omitted → 1; ``limit`` omitted → read to the end;
* ``end = min(offset + limit - 1, total_lines)`` — an overshooting
``limit`` CLAMPS (``last_line`` is the document's last line);
* ``offset`` past the end, an empty document, or a ``limit`` below 1 →
``past_end=True`` with empty text and the TRUE ``total_lines`` (the
caller builds the teaching refusal).
Never raises for a range problem: the refusal is the caller's, the
total always travels with the result.
"""
lines = content.splitlines()
total_lines = len(lines)
first = 1 if offset is None else offset
end = total_lines if limit is None else min(first + limit - 1, total_lines)
if total_lines == 0 or first > total_lines or end < first:
return SliceResult(
text="",
first_line=first,
last_line=0,
total_lines=total_lines,
past_end=True,
)
return SliceResult(
text="\n".join(lines[first - 1 : end]),
first_line=first,
last_line=end,
total_lines=total_lines,
past_end=False,
)
def cap_slice(text: str, first_line: int, cap: int) -> tuple[str, int, bool]:
"""Apply the character budget to one slice AT A LINE boundary.
Returns ``(text_to_show, last_line_shown, truncated)``. The whole
point of cutting at a line boundary is that the caption's line range
stays true: ``last_line_shown`` is the last line the reader actually
sees, so a capped block never claims lines it did not show.
* ``cap <= 0`` raises :class:`ValueError` — a mis-wired caller must be
loud. The ``show_text_max_chars == 0`` kill switch is checked by
:func:`resolve_answer_blocks`, never by reaching here;
* ``len(text) <= cap`` → the text untouched, its own last line,
``truncated=False``;
* otherwise cut to the last WHOLE line that fits and append the shared
:data:`~app.rag.retriever.TRUNCATION_MARKER` on its own line — the
marker is not content and is not counted in ``last_line_shown`` (a
line counts as fitting only when its terminator is inside the
budget, or sits exactly at the budget's edge);
* a first line LONGER than ``cap`` cuts that one line mid-way (the only
sub-line cut, documented) and still ends with the marker.
"""
if cap <= 0:
raise ValueError(f"cap_slice needs a positive cap, got {cap!r}")
if len(text) <= cap:
if not text:
return text, first_line, False
# ``split("\\n")``, NOT ``splitlines()``: a slice that ends on a
# BLANK line legitimately ends with ``"\\n"`` (line 5 of
# ``"a\nb\n\n"`` is empty and present), and ``splitlines()`` drops
# that final empty element — which would report an extent one
# line short of the text actually shown (a caption that lies, and
# a canonical locator that no longer matches the same range
# resolved by another entry point).
return text, first_line + len(text.split("\n")) - 1, False
cut = text[:cap]
if text[cap] == "\n":
# The budget lands exactly on a line end — every line of the cut
# is whole, nothing to trim.
return (
f"{cut}\n{TRUNCATION_MARKER}",
first_line + len(cut.splitlines()) - 1,
True,
)
boundary = cut.rfind("\n")
if boundary <= 0:
# No whole line fits — the single documented mid-line cut.
return f"{cut}\n{TRUNCATION_MARKER}", first_line, True
whole = cut[:boundary]
return (
f"{whole}\n{TRUNCATION_MARKER}",
first_line + len(whole.splitlines()) - 1,
True,
)
def document_by_identity(db: Session, combined: str) -> Document | None:
"""The indexed row at the combined ``source/path`` identity, or ``None``.
Split at the FIRST ``'/'`` — source names are directory basenames and
can never contain one, so the part before is the source and everything
after is the path (the same identity contract as
``app.rag.agent._resolve_path``).
This is that contract **reimplemented as a bare lookup**, deliberately
NOT a call into ``agent.py``: the tool branch already has
``_resolve_path`` (with its did-you-mean suggestions), the fence
resolver only needs yes/no, and importing the agent from here would
cycle (``agent.py`` imports this module). A traversal string
(``../../etc/passwd``) is simply a row that does not exist — there is
no file access here to guard, only a ``WHERE`` clause.
"""
if "/" not in combined:
return None
source, _, path = combined.partition("/")
return db.scalar(
select(Document).where(Document.source == source, Document.path == path)
)
def block_for(
db: Session,
combined: str,
*,
offset: int | None = None,
limit: int | None = None,
cap: int,
doc: Document | None = None,
) -> VerbatimBlock | None:
"""Build the one block a locator names, or ``None`` (nothing displays).
Resolves ``combined`` (or takes the caller's already-resolved ``doc``)
and refuses ``None`` for: no such document (including every
traversal-shaped identity), an ``is_image`` document (image display
belongs to ``show_image`` — LOCKED A6), and an empty or past-the-end
slice. Otherwise ``slice_lines`` → ``cap_slice`` → the block, with the
canonical :attr:`~VerbatimBlock.locator` built here (``source/path``,
plus ``#first-last`` when the request named a range **or** the cap cut
the slice — the form task 07 matches on).
"""
if doc is None:
doc = document_by_identity(db, combined)
if doc is None or doc.is_image:
return None
sliced = slice_lines(doc.content, offset, limit)
if sliced.past_end or not sliced.text:
return None
text, last_line, truncated = cap_slice(sliced.text, sliced.first_line, cap)
locator = f"{doc.source}/{doc.path}"
if offset is not None or limit is not None or truncated:
locator = f"{locator}#{sliced.first_line}-{last_line}"
return VerbatimBlock(
source=doc.source,
path=doc.path,
title=doc.title,
start_line=sliced.first_line,
end_line=last_line,
total_lines=sliced.total_lines,
text=text,
truncated=truncated,
locator=locator,
)
def resolve_answer_blocks(
db: Session,
answer: str,
shown: Sequence[VerbatimBlock],
settings: Settings,
) -> list[VerbatimBlock]:
"""The blocks an answer displays, in DISPLAY ORDER (the done-time pass).
Fuses the two ways a block enters an answer — the `` ```bor `` locator
fences the model wrote (resolved here, from the row) and the blocks the
``show_text`` tool already showed during the turn (``shown``) — into
the ONE list task 05 puts on the ``done`` frame and persists with the
record.
**Kill switches first** (checked HERE so no caller has to remember
them): ``[]`` when ``settings.answer_text`` is off, or
``settings.show_text_max_blocks == 0``, or
``settings.show_text_max_chars == 0``.
**Ordering rule** (task 07's DOM order and the E2E assertions depend on
it): answer order first — one block per fence, in the order the fences
appear — then the ``shown`` blocks the answer never located, in their
original call order. Dedupe is by canonical locator, and the **fence
wins** over a same-locator entry from ``shown`` because the fence
controls placement. A fence that does not resolve (unknown path, image
document, past-the-end range, malformed locator) is simply absent: the
caller sees "not among the blocks", which is what task 07 renders as
the unresolved state — the answer's own fence text is left alone, so
nothing is silently deleted. The list is finally truncated to
``settings.show_text_max_blocks`` — and the fence scan STOPS once that
many blocks are in, because the same truncation makes every later
block unreachable (the bound keeps a fence-stuffed answer to at most
``show_text_max_blocks`` successful lookups instead of one ``SELECT``
per fence on the request's event loop).
"""
if not settings.answer_text:
return []
if settings.show_text_max_blocks == 0 or settings.show_text_max_chars == 0:
return []
cap = settings.show_text_max_chars
blocks: list[VerbatimBlock] = []
#: The locator strings already CLAIMED (canonical written forms) — a
#: repeated fence never resolves twice, and an unresolvable fence
#: still holds its slot against a same-locator ``shown`` block.
claimed: set[str] = set()
#: The locators of the blocks actually appended — a resolved block
#: can carry a NARROWER locator than the locator its fence wrote (the
#: char cap cut the slice), and the same range shown by the tool
#: carries exactly that narrower key, so both keys dedupe.
emitted: set[str] = set()
for raw in fences_in(answer):
canonical = canonical_locator(raw)
if canonical is None or canonical in claimed:
continue
# Reserve the key BEFORE resolving: an unresolvable fence must not
# let a same-locator ``shown`` block jump into the fence's slot.
claimed.add(canonical)
block = _block_for_canonical(db, canonical, cap)
if block is None or block.locator in emitted:
continue
claimed.add(block.locator)
emitted.add(block.locator)
blocks.append(block)
if len(blocks) >= settings.show_text_max_blocks:
# The display budget is already spent — the truncation below
# makes every later block unreachable, so stop LOOKING too:
# a fence-stuffed answer costs at most
# ``show_text_max_blocks`` successful lookups instead of one
# ``SELECT`` per fence on the request's event loop.
break
for block in shown:
if block.locator in emitted or block.locator in claimed:
continue
emitted.add(block.locator)
blocks.append(block)
return blocks[: settings.show_text_max_blocks]
def _block_for_canonical(db: Session, canonical: str, cap: int) -> VerbatimBlock | None:
"""Resolve one ALREADY-CANONICAL locator (the output of :func:`canonical_locator`)."""
identity, _, range_part = canonical.partition("#")
if not range_part:
return block_for(db, identity, cap=cap)
start_str, _, end_str = range_part.partition("-")
start = int(start_str)
end = int(end_str)
return block_for(db, identity, offset=start, limit=end - start + 1, cap=cap)
+124 -7
View File
@@ -182,6 +182,47 @@ class SourceRef(BaseModel):
return data
class VerbatimBlockRef(BaseModel):
"""One SERVER-resolved verbatim block of a chat answer (phase 182).
The item of :attr:`ChatDoneEvent.answer_text` and of the persisted
:class:`ChatMessage.answer_text` — the display block the
``show_text`` tool or a `` ```bor `` locator fence asked for, with
the exact bytes read out of ``documents.content`` by
:func:`app.rag.verbatim.block_for`.
**``text`` is SERVER-supplied bytes, never model output** (LOCKED
A4/A7): the model only ever names the location, so what the user
reads and copies cannot differ from the indexed file by a character
— the phase's whole reason for existing. The field carries the same
600 000 cap as the phase-83/164 answer text (deliberately generous
above the 12 000 default of ``BOR_SHOW_TEXT_MAX_CHARS``, so an
operator raising that budget never hits the wire limit instead).
``start_line`` / ``end_line`` are **1-based and INCLUSIVE** and
describe exactly the lines ``text`` contains AFTER any character cap
— they are the SERVER's resolved extent, never the model's claim
(LOCKED A2), which is what the panel's caption prints. The caps on
``source`` / ``path`` / ``title`` are :class:`SourceRef`'s (the same
``documents`` column lengths, phase 83 SEC-05); ``locator`` is the
canonical matching key (phase 182 D7 — ``source/path`` or
``source/path#12-40``) the client matches its rendered placeholder
against.
"""
source: str = Field(max_length=120)
path: str = Field(max_length=1000)
title: str = Field(max_length=500)
start_line: int = Field(ge=0)
end_line: int = Field(ge=0)
total_lines: int = Field(ge=0)
#: The verbatim bytes themselves (see the class docstring) — the
#: phase-83 answer-text cap, generous above the per-block budget.
text: str = Field(max_length=600_000)
truncated: bool = False
locator: str = Field(max_length=2100)
class ChatThinkingEvent(BaseModel):
"""SSE thinking event: one chunk of the model's reasoning (phase 17).
@@ -342,6 +383,23 @@ class ChatDoneEvent(BaseModel):
byte-identical to pre-phase (the byte-identical-off contract,
PLAN §13; the :attr:`answer_image` / :class:`SourceRef.image_url`
omission precedent).
Phase 182 (task 05, LOCKED A1/A4/A9): ``answer_text`` — the answer's
VERBATIM display blocks, the text sibling of :attr:`answer_image`
with two deliberate differences. (1) It is a LIST: one answer may
display several blocks (the ``show_text`` calls and/or the
`` ```bor `` locator fences the answer itself wrote — D1), in
DISPLAY order (answer order first, then the tool blocks the answer
never located — :func:`app.rag.verbatim.resolve_answer_blocks`).
(2) Each item carries the TEXT itself rather than a route: the bytes
ride inline (A4 — no new bytes route, no new anonymous surface, so a
shared page renders from its own record exactly like the phase-123
chat image), and every byte of it comes from ``documents.content``,
never from the model (A7). ``None`` / ``[]`` — every turn without
blocks, every deflected turn (D9 resolves nothing on a disclaimer),
every ``answer_text=False`` turn, and the whole pre-phase — is
DROPPED on serialization: ABSENT, never ``null``, never ``[]``, so
those frames stay byte-identical to pre-phase (the same contract).
"""
type: str = "done"
@@ -357,25 +415,36 @@ class ChatDoneEvent(BaseModel):
#: ONLY on the cap-reached turn; ``False`` (every other turn)
#: serializes WITHOUT the key — ABSENT, never ``false``.
budget_exhausted: bool = False
#: Phase 182 (task 05) — see the class docstring. ``None``/``[]``
#: (every turn without verbatim blocks) serializes WITHOUT the key —
#: ABSENT, never ``null``, never ``[]``.
answer_text: list[VerbatimBlockRef] | None = None
@model_serializer(mode="wrap")
def _serialize(self, handler: SerializerFunctionWrapHandler) -> Any:
"""The phase-124 answer-image omission rule: ``answer_image``
``None`` (every deflected / image-less / pre-phase turn)
serializes WITHOUT the key — ABSENT, never ``null`` — so those
done frames stay byte-identical to pre-phase (the
"""The additive-key omission rules: ``answer_image`` ``None``
(every deflected / image-less / pre-phase turn) serializes
WITHOUT the key — ABSENT, never ``null`` — so those done frames
stay byte-identical to pre-phase (the
:class:`SourceRef.image_url` / phase-123 ``image`` omission
precedents). A shown ref keeps all its fields (its own
serializer drops a ``None`` ``image_url`` — a shown doc is an
image doc, so in practice it always carries it). Phase 160
(task 03): ``budget_exhausted`` ``False`` (every non-cap turn)
serializes WITHOUT the key too — ABSENT, never ``false`` — the
same byte-identical-off contract."""
same byte-identical-off contract. Phase 182 (task 05):
``answer_text`` ``None`` OR EMPTY (a turn whose answer resolved
no block, a deflected turn, the whole ``answer_text=False``
configuration, and the pre-phase) serializes WITHOUT the key —
ABSENT, never ``null`` and never ``[]``, so every block-free
frame keeps its exact pre-phase bytes."""
data = handler(self)
if self.answer_image is None:
data.pop("answer_image", None)
if not self.budget_exhausted:
data.pop("budget_exhausted", None)
if not self.answer_text:
data.pop("answer_text", None)
return data
@@ -1478,6 +1547,28 @@ class ChatMessage(BaseModel):
# exactly like ``stopped`` — the phase-50 byte-identical
# round-trip contract covers the new key automatically.
budget_exhausted: bool | None = None
# Phase 182 (task 05, LOCKED A4/A9): the answer's VERBATIM display
# blocks — the persisted twin of the done frame's ``answer_text``
# (the ``show_text`` tool's blocks and/or the ones a `` ```bor ``
# locator fence resolved), on the BRAIN record only. Like
# ``answer_image`` the field EXISTS so the ``extra="forbid"``
# boundary does NOT 422 the done-time auto-save carrying it (the
# phase-113 lesson, applied up front), and the restore path plus the
# anonymous shared page re-render the panels from it — the bytes are
# part of the chat's own content, so they ride the same
# credential-is-the-row trust model and need NO new route (A4).
# No migration (``saved_chats.messages`` is raw JSONB, the phase-50
# / 160 precedent). ``None``/``[]`` serializes WITHOUT the key (the
# :meth:`_drop_image_when_absent` serializer below), so a turn with
# no blocks — and every pre-phase record — round-trips
# byte-identically (A9). The per-block ``text`` cap is
# :class:`VerbatimBlockRef`'s own; the list cap matches the
# ``tools``/``segments`` cap (50), far past the
# ``show_text_max_blocks`` default of 5 — and the block bytes count
# toward :data:`SAVED_CHAT_TOTAL_CHARS_BUDGET`, so the display blocks
# never become a second, unbounded channel on the anonymous save
# surface (the phase-83 SEC-05 invariant).
answer_text: list[VerbatimBlockRef] | None = Field(default=None, max_length=50)
@model_serializer(mode="wrap")
def _drop_image_when_absent(self, handler: SerializerFunctionWrapHandler) -> Any:
@@ -1492,12 +1583,21 @@ class ChatMessage(BaseModel):
record WITH one keeps it — the user bubble, the refreshed page,
and the shared chat all render the image from the served route
(the image is part of the chat's content, so it rides the same
public/credential-is-the-id trust model)."""
public/credential-is-the-id trust model).
Phase 182 (task 05) extends the same rule to ``answer_text``:
``None`` OR an empty list (every record without verbatim blocks,
and every pre-phase-182 record) serializes WITHOUT the key —
ABSENT, never ``null``, never ``[]`` (A9), so those records
round-trip byte-identically while a record WITH blocks keeps the
server's bytes for the restore path and the shared page."""
data = handler(self)
if self.image is None:
data.pop("image", None)
if self.answer_image is None:
data.pop("answer_image", None)
if not self.answer_text:
data.pop("answer_text", None)
# Phase 158 (task 01, D4): the timeline is ABSENT (never
# null) when there is no run to render — the pre-phase /
# no-thinking-no-tools shape stays byte-identical (the
@@ -1529,11 +1629,28 @@ def _require_within_total_budget(messages: list[ChatMessage]) -> list[ChatMessag
(phase 164): ``len(text) + len(thinking or "")`` over every message
must stay within :data:`SAVED_CHAT_TOTAL_CHARS_BUDGET`.
Phase 182 (task 05) counts the VERBATIM BLOCKS into the same sum: a
block's ``text`` carries the phase-83 answer-text cap (600 000) so an
operator raising ``BOR_SHOW_TEXT_MAX_CHARS`` never hits the wire limit,
and the anonymous write surface (``POST/PUT /api/chats`` is public —
phase 55 A1) must not gain a second, UNBOUNDED byte channel next to
the bounded one (the SEC-05 invariant: the total is what keeps the
audited worst case at the audited order). A legitimate turn carries at
most ``BOR_SHOW_TEXT_MAX_BLOCKS`` × ``BOR_SHOW_TEXT_MAX_CHARS`` bytes
(5 × 12 000 = 60 KB by default), orders of magnitude below the budget,
so nothing the server produces is affected — the bound only rejects a
client stuffing blocks.
Raises :class:`ValueError` naming the budget, which Pydantic turns
into the ordinary 422 validation response (``loc`` = the model
itself, ``msg`` carrying the budget wording) — the same boundary-
only style as every other phase-83/164 cap."""
total = sum(len(m.text) + len(m.thinking or "") for m in messages)
total = sum(
len(m.text)
+ len(m.thinking or "")
+ sum(len(b.text) for b in m.answer_text or [])
for m in messages
)
if total > SAVED_CHAT_TOTAL_CHARS_BUDGET:
raise ValueError(
f"saved chat exceeds the {SAVED_CHAT_TOTAL_CHARS_BUDGET:,}-character budget"