phase: 182_verbatim_show_text
**Phase 182 — final verification pass (all 8 tasks already complete; this was verification + docs + one defect fix).**
- Verified: `app/rag/verbatim.py` (locator grammar, `splitlines()` slice, line-boundary cap, `resolve_answer_blocks`), `SHOW_TEXT_TOOL` + `shown_texts`, `SHOW_TEXT_CLAUSE`, `done.answer_text` (9 fields, key dropped when empty) + `ChatMessageRecord.answer_text` (≤50, 600 000-char cap counted in the 8 M budget), `verbatim.js` + 4 `{verbatim:true}` renderers, `Containerfile` minify step, `.env.example` 3 vars.
- `uv run python -c "…answer_text, show_text_max_chars, show_text_max_blocks"` → `True 12000 5`; each negative refuses startup naming its field; `0` accepted as inert ✅
- `bash .agents/validate.sh` → **validation OK**: 4320 unit+integration tests green, `TOTAL 6849 17 → 99%` (gate app/ >90% OK), `ruff check .` "All checks passed!", `pyright` 0 errors (re-run after my edits, 08:28).
- E2E, each in isolation `--no-cov`: `test_verbatim_show_text.py` 13 passed (panel bytes == `documents.content`, server caption, clipboard == server string, `/document.html?source=…&path=…`, honest unresolved state, reload+share, `show_image` regression, flag-off app) ✅; green: `test_answer_images` `test_harness_aligned_tools` `test_agent_document_tools` `test_markdown_tables` `test_markdown_inline_links_images` `test_svg_document_preview` `test_smoke` `test_document_viewer` `test_markdown_blockquotes` `test_chat_persistence` `test_thinking_timeline` `test_chat_rag` `test_settings_mobile_value_width`.
- **Defect found & fixed (pre-existing, not phase 182):** `tests/e2e/test_admin_settings_panel.py::test_admin_sees_groups_and_source_badges` failed — its whole-page `assert "reese" not in page.content()` tripped on the shell's onboarding chips, which follow the newest saved chats in the one shared E2E DB (a leftover `reeselink.json` question from a tool suite). Scoped the strict sweep to `.settings-shell` and kept a whole-page `reese:reese` credential needle → 12 passed with the leftovers still present.
- Docs: README fixed 2 inconsistencies — Quick Start omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the app fail-loud refuses boot) and the broken anchor `#chat--` → `#chat-`; full re-read of README (1816 lines) + AGENTS confirmed the phase-182 feature/settings/log/UI-contract text matches the code, and every `.env.example` var + every `Settings` field is documented.
- Notable: kept `resolve_answer_blocks` synchronous on the SSE terminal path (bounded indexed SELECTs, matching the existing `query_log` precedent and the phase's documented fence-lookup bound); no live-infra changes made.
- Next pending phase: none — `todo/` holds only 182 (its 8 task files are in `complete/`; the harness commits + moves `00_phase.md`).
This commit is contained in:
@@ -244,16 +244,16 @@ def _refuse_embedded_credentials(key: str, value: str) -> None:
|
||||
def get_catalogue(
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
) -> AdminSettingsOut:
|
||||
"""The whole catalogue (78 items, ``GROUPS`` order — the stable
|
||||
order the E2E pins against).
|
||||
"""The whole catalogue (one item per ``Settings`` field, ``GROUPS``
|
||||
order — the stable order the E2E pins against).
|
||||
|
||||
Reads the table once, resolves the effective ``Settings`` through
|
||||
``get_settings()`` (the overlay — DB-over-env, task 02), and
|
||||
emits one item per field. GETs log nothing here: the phase-167
|
||||
``ResponseBudgetMiddleware`` already writes the
|
||||
``api method=… bytes=…`` line, and the whole catalogue must stay
|
||||
under ``BOR_LOG_LARGE_RESPONSE_BYTES``' INFO bar (78 items ≈ 20 KB
|
||||
— verified in task 06 with the audit script).
|
||||
under ``BOR_LOG_LARGE_RESPONSE_BYTES``' INFO bar (the whole catalogue
|
||||
is ≈ 20 KB — verified in task 06 with the audit script).
|
||||
"""
|
||||
rows = _rows(db)
|
||||
effective = get_settings()
|
||||
@@ -371,7 +371,7 @@ def reset_value(
|
||||
@router.delete("", response_model=AdminSettingsCleared)
|
||||
def reset_all(db: Session = Depends(get_db)) -> AdminSettingsCleared: # noqa: B008
|
||||
"""The panel's "Clear all overrides": delete every row (the env
|
||||
values win again for all 78 settings), invalidate (task 03) and
|
||||
values win again for every setting), invalidate (task 03) and
|
||||
report how many rows went."""
|
||||
result = db.execute(delete(AppSetting).returning(AppSetting.key))
|
||||
db.commit()
|
||||
|
||||
+97
-2
@@ -227,6 +227,30 @@ pre-phase. A question image is turn-local: it is NEVER indexed as a
|
||||
document, and prior turns' images are never replayed into the model's
|
||||
history (``history_to_messages`` is unchanged — the text of a prior
|
||||
turn that had an image stands alone, LOCKED A7).
|
||||
|
||||
Verbatim display (phase 182, tasks 03/05; LOCKED A1/A4/A7/A9): a turn
|
||||
that showed file contents through the ``show_text`` tool (task 03, the
|
||||
bytes the SERVER resolved) — or whose ANSWER carries a `` ```bor ``
|
||||
locator fence (task 02's resolver, placement sugar that resolves
|
||||
standalone) — gains one additive ``done`` key, ``answer_text``: the
|
||||
list of :class:`~app.schemas.VerbatimBlockRef` blocks in DISPLAY order,
|
||||
each carrying the exact ``documents.content`` slice plus the SERVER's
|
||||
resolved 1-based, inclusive line extent and its canonical ``locator``.
|
||||
The model never supplies those bytes: the endpoint accumulates the
|
||||
turn's CLEAN VISIBLE answer text (the same post-scaffold-filter pieces
|
||||
that feed ``content_chars`` — never ``thinking``, so a locator can only
|
||||
ever come from the answer) and resolves it through
|
||||
:func:`app.rag.verbatim.resolve_answer_blocks` when the ``done`` frame
|
||||
is built. A deflected turn resolves NOTHING (D9 — a fence hallucinated
|
||||
into a disclaimer must not leak KB text through the turn that just said
|
||||
it had nothing); ``answer_text=False``, ``show_text_max_blocks=0`` and
|
||||
``show_text_max_chars=0`` short-circuit inside the resolver, and a turn
|
||||
with no blocks serializes WITHOUT the key (A9 — every block-free frame
|
||||
keeps its exact pre-phase bytes). The per-turn log line records
|
||||
``verbatim_blocks=N`` after ``scaffold_stripped=N`` — the blocks
|
||||
ACTUALLY SENT (after dedupe and the ``show_text_max_blocks`` cut), 0 on
|
||||
every turn without them including deflected ones (the PLAN §9 additive
|
||||
revision).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -235,7 +259,7 @@ import json
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import AsyncIterator, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
@@ -283,6 +307,9 @@ from app.rag.summarizer import (
|
||||
IMAGE_MIMES,
|
||||
image_data_url,
|
||||
) # phase 123: the phase-122 mime map + the shared data-URL helper
|
||||
from app.rag.verbatim import (
|
||||
resolve_answer_blocks, # phase 182: fences + shown blocks -> the done frame
|
||||
)
|
||||
from app.schemas import (
|
||||
ChatDoneEvent,
|
||||
ChatErrorEvent,
|
||||
@@ -292,6 +319,7 @@ from app.schemas import (
|
||||
ChatToolEvent,
|
||||
ChatToolResultEvent,
|
||||
SourceRef,
|
||||
VerbatimBlockRef,
|
||||
)
|
||||
|
||||
logger = logging.getLogger("app.chat")
|
||||
@@ -527,6 +555,12 @@ def plan_turn(
|
||||
# carries the show_image clause exactly when the
|
||||
# BOR_ANSWER_IMAGES toggle is on (off = byte-identical).
|
||||
show_image=settings.answer_images,
|
||||
# Phase 182 (task 04, LOCKED A1/A8): the same rule for the
|
||||
# show_text clause — it rides exactly when the BOR_ANSWER_TEXT
|
||||
# master switch is on (the default), and an off build is
|
||||
# byte-identical to pre-phase (no clause, and agent_tools
|
||||
# offers four tools, not five).
|
||||
show_text=settings.answer_text,
|
||||
),
|
||||
suggested,
|
||||
related_docs,
|
||||
@@ -933,6 +967,16 @@ async def chat(
|
||||
thinking_chars = 0
|
||||
content_chars = 0 # phase 71: the turn's visible (clean) content
|
||||
scaffold_stripped = 0 # phase 71: sum across the turn's requests
|
||||
# Phase 182 (task 05, D3): the turn's clean VISIBLE answer
|
||||
# text — the SAME pieces counted into ``content_chars``
|
||||
# (post-scaffold-filter, never ``thinking``, so a locator
|
||||
# can only ever come from the answer), kept so the ``done``
|
||||
# builder can resolve the `` ```bor `` fences the answer
|
||||
# wrote against the indexed rows. Bounded by the model's own
|
||||
# output-token budget (``max_output_tokens``) — the answer
|
||||
# already lives in memory for the duration of the turn, so
|
||||
# this adds no new cap.
|
||||
visible_parts: list[str] = []
|
||||
# Phase 160 (task 02): the transport half of the cap path — a
|
||||
# request made WITHOUT tools (the deflected turn, A8, and its
|
||||
# recovery) never produces a ``tool`` frame: a structured
|
||||
@@ -1082,6 +1126,9 @@ async def chat(
|
||||
yield sse_event(ChatThinkingEvent(text=piece.text).model_dump())
|
||||
else:
|
||||
content_chars += len(piece.text)
|
||||
# Phase 182 (task 05): the same clean piece kept
|
||||
# as TEXT for the done-time locator pass.
|
||||
visible_parts.append(piece.text)
|
||||
yield sse_event({"type": "delta", "text": piece.text})
|
||||
if dropped_calls:
|
||||
logger.debug(
|
||||
@@ -1269,12 +1316,46 @@ async def chat(
|
||||
except Exception: # noqa: BLE001 — the answer already went out
|
||||
logger.exception("chat: failed to write query_log question=%r", request.message)
|
||||
|
||||
# Phase 182 (task 05, D2/D9): resolve the turn's verbatim
|
||||
# display blocks — the ``show_text`` tool's blocks
|
||||
# (``holder.shown_texts``, read AFTER the stream exactly
|
||||
# like ``shown_images``) fused with the blocks the answer's
|
||||
# own `` ```bor `` locator fences name, over the CLEAN
|
||||
# visible text accumulated above (never ``thinking``).
|
||||
# A DEFLECTED turn resolves NOTHING (D9 — a fence
|
||||
# hallucinated into a disclaimer must not leak KB text
|
||||
# through the turn that just said it had nothing), which is
|
||||
# also why this is the only branch with a resolved holder.
|
||||
# A DB failure here must not break a stream whose answer
|
||||
# already went out (the query_log precedent above): the
|
||||
# turn degrades to NO blocks, the frame simply omits the
|
||||
# key, and the client renders each fence in its honest
|
||||
# unresolved state — never the model's own bytes.
|
||||
blocks: list[VerbatimBlockRef] = []
|
||||
if not plan.deflected:
|
||||
try:
|
||||
with SessionLocal() as verbatim_db:
|
||||
resolved = resolve_answer_blocks(
|
||||
verbatim_db,
|
||||
"".join(visible_parts),
|
||||
holder.shown_texts,
|
||||
settings,
|
||||
)
|
||||
blocks = [VerbatimBlockRef(**asdict(b)) for b in resolved]
|
||||
except Exception: # noqa: BLE001 — the answer went out
|
||||
logger.exception(
|
||||
"chat: verbatim block resolution failed question=%r",
|
||||
request.message,
|
||||
)
|
||||
blocks = []
|
||||
|
||||
logger.info(
|
||||
"question=%r embed_ms=%d top_score=%.3f fts_hits=%d summary_hits=%d "
|
||||
"suggested=%d tuning=%d "
|
||||
"kb_chars=%d history_msgs=%d threshold=%.2f deflected=%s sources=%r "
|
||||
"thinking=%s thinking_chars=%d tool_calls=%d rounds=%d "
|
||||
"cap_hit=%d total_ms=%d retries=%d scaffold_stripped=%d",
|
||||
"cap_hit=%d total_ms=%d retries=%d scaffold_stripped=%d "
|
||||
"verbatim_blocks=%d",
|
||||
request.message,
|
||||
embed_ms,
|
||||
plan.top_score,
|
||||
@@ -1299,6 +1380,12 @@ async def chat(
|
||||
total_ms,
|
||||
retries_used,
|
||||
scaffold_stripped,
|
||||
# phase 182 (task 05): the blocks ACTUALLY SENT (after
|
||||
# the canonical-locator dedupe and the
|
||||
# ``show_text_max_blocks`` cut) — 0 on every turn
|
||||
# without them, deflected ones included (PLAN §9
|
||||
# additive revision).
|
||||
len(blocks),
|
||||
)
|
||||
settled = True # terminal: the done frame settles the turn
|
||||
# Phase 112 (A8 revised, TODO L2): a deflected turn cites
|
||||
@@ -1360,6 +1447,14 @@ async def chat(
|
||||
# (``False`` → ABSENT on serialization) is
|
||||
# correct by construction.
|
||||
budget_exhausted=holder.cap_reached,
|
||||
# Phase 182 (task 05): the answer's verbatim display
|
||||
# blocks, resolved above in DISPLAY order (answer
|
||||
# order first, then the tool blocks the answer never
|
||||
# located). ``blocks or None`` — an empty list is
|
||||
# ``None`` here and the serializer drops the key
|
||||
# either way, so a block-free frame keeps its exact
|
||||
# pre-phase bytes (LOCKED A9).
|
||||
answer_text=blocks or None,
|
||||
).model_dump()
|
||||
)
|
||||
finally:
|
||||
|
||||
@@ -294,6 +294,41 @@ class Settings(BaseSettings):
|
||||
#: 2026-09-10, ``TODO.md`` L5): A7's never-truncated contract is for
|
||||
#: the retrieval ``<documents>`` path, which stays whole.
|
||||
read_max_chars: int = 128_000
|
||||
#: Feature toggle for the verbatim text-display capability (phase 182,
|
||||
#: ``BOR_ANSWER_TEXT``; ``0``/``false`` = off). The single knob that
|
||||
#: turns the whole feature on or off. Gates the ANSWER side ONLY: while
|
||||
#: on, the grounded agent is offered the ``show_text`` display tool,
|
||||
#: the HIGH prompt carries its teaching clause, a `` ```bor `` locator
|
||||
#: fence in the answer is resolved server-side, and the resolved blocks
|
||||
#: ride the done frame's ``answer_text`` key (and the saved record's).
|
||||
#: While off, behavior is byte-identical everywhere — no ``show_text``
|
||||
#: tool offered, no prompt clause, no locator resolution, no
|
||||
#: ``answer_text`` key in any frame or record. THE DEFAULT IS **TRUE**
|
||||
#: (owner decision 2026-10-14, LOCKED A8): unlike phase 124's
|
||||
#: ``answer_images`` (default off, a cosmetic extra), an answer whose
|
||||
#: code block silently differs from the file is a correctness bug, so
|
||||
#: the fix ships ON and this switch is an escape hatch, not an opt-in.
|
||||
answer_text: bool = True
|
||||
#: Cap in characters on ONE displayed verbatim slice (phase 182,
|
||||
#: ``BOR_SHOW_TEXT_MAX_CHARS``): the slice the server resolves for a
|
||||
#: ``show_text`` call or a `` ```bor `` locator fence is cut at the cap
|
||||
#: AT A LINE BOUNDARY, the shared ``[…truncated…]`` marker
|
||||
#: (``app/rag/retriever.py``) is appended and the caption's ``end_line``
|
||||
#: shrinks to the last fully-shown line — the caption never lies (D4).
|
||||
#: Worst-case ``done``-frame growth is
|
||||
#: ``show_text_max_chars * show_text_max_blocks`` = 60 KB against the
|
||||
#: 256 KB response budget (phase 167's byte line makes it visible). A
|
||||
#: char cap like ``read_max_chars`` (no tokenizer in the repo); ``0``
|
||||
#: is inert (nothing displays), a negative value is a typo and fails
|
||||
#: startup (validator).
|
||||
show_text_max_chars: int = 12_000
|
||||
#: Cap on the verbatim blocks displayed per answer (phase 182,
|
||||
#: ``BOR_SHOW_TEXT_MAX_BLOCKS``): the ``show_text`` calls and
|
||||
#: `` ```bor `` locator fences of one turn, in ANSWER order, past this
|
||||
#: count are not displayed (D8). ``0`` is inert (no block displays —
|
||||
#: the feature renders as plain prose), a negative value is a typo and
|
||||
#: fails startup (the ``agent_max_rounds`` validator pattern).
|
||||
show_text_max_blocks: int = 5
|
||||
|
||||
# --- Chat concurrency (SEC-14-04, task 03) ---
|
||||
#: Maximum concurrent chat turns allowed (SEC-14-04).
|
||||
@@ -705,6 +740,28 @@ class Settings(BaseSettings):
|
||||
raise ValueError("read_max_chars must be >= 0 (chars)")
|
||||
return v
|
||||
|
||||
@field_validator("show_text_max_chars")
|
||||
@classmethod
|
||||
def _show_text_max_chars_non_negative(cls, v: int) -> int:
|
||||
"""A negative per-block cap is a typo — it would slice from the END
|
||||
of the content (negative indexing) instead of failing. Fail loud at
|
||||
startup, naming the field (the ``read_max_chars`` pattern, phase
|
||||
182). ``0`` is legal and inert (no block displays)."""
|
||||
if v < 0:
|
||||
raise ValueError("show_text_max_chars must be >= 0 (chars)")
|
||||
return v
|
||||
|
||||
@field_validator("show_text_max_blocks")
|
||||
@classmethod
|
||||
def _show_text_max_blocks_non_negative(cls, v: int) -> int:
|
||||
"""A negative per-answer block cap is a typo — it would count down
|
||||
from the END of the block list instead of failing. Fail loud at
|
||||
startup, naming the field (the ``agent_max_rounds`` pattern, phase
|
||||
182). ``0`` is legal and inert (no block displays)."""
|
||||
if v < 0:
|
||||
raise ValueError("show_text_max_blocks must be >= 0 (blocks)")
|
||||
return v
|
||||
|
||||
@field_validator("pdf_pictorial_min_chars")
|
||||
@classmethod
|
||||
def _pdf_pictorial_min_chars_non_negative(cls, v: int) -> int:
|
||||
|
||||
@@ -267,6 +267,13 @@ CATALOG: dict[str, SettingMeta] = {
|
||||
"read_max_chars": SettingMeta(
|
||||
"read_max_chars", "Agent tools", "Read max chars"
|
||||
),
|
||||
"answer_text": SettingMeta("answer_text", "Agent tools", "Answer text"),
|
||||
"show_text_max_chars": SettingMeta(
|
||||
"show_text_max_chars", "Agent tools", "Show text max chars"
|
||||
),
|
||||
"show_text_max_blocks": SettingMeta(
|
||||
"show_text_max_blocks", "Agent tools", "Show text max blocks"
|
||||
),
|
||||
# --- Import & sources ---
|
||||
"import_extensions": SettingMeta(
|
||||
"import_extensions", "Import & sources", "Import extensions (CSV)"
|
||||
|
||||
+292
-13
@@ -211,6 +211,29 @@ task 04):
|
||||
— its ``"No matches for …"`` line is a (counted) result. A rejected
|
||||
call still consumes a *round* in the loop, so a pathological stream
|
||||
that keeps emitting rejected calls is bounded by the cap (point 4).
|
||||
|
||||
Display tools (phase 124 ``show_image``, phase 182 ``show_text`` —
|
||||
both composed by :func:`agent_tools`, each behind its own flag): they
|
||||
display content FOR THE USER. ``show_image`` never adds context; the
|
||||
5th tool ``show_text`` (phase 182, task 03) returns the resolved
|
||||
slice back to the model ONLY as the truth of what was displayed
|
||||
(LOCKED A3 — with a do-NOT-retype instruction: the model describes
|
||||
the block, the user copies the server's bytes), and its block rides
|
||||
``holder.shown_texts``, never ``read_docs``. The LOCKED core contract
|
||||
A1–A3/A6: **the model names a location, the server shows the bytes**
|
||||
— the handler resolves the slice through
|
||||
:func:`app.rag.verbatim.block_for` (same ``(source, path)`` identity,
|
||||
same 1-based ``splitlines()`` indexing as ``read``/``grep``, same
|
||||
char-budget-cut-at-a-line-boundary), the DISPLAYED EXTENT is the
|
||||
server's (an overshooting ``limit`` clamps into the result's ``lines:``
|
||||
header, an ``offset`` past the end is the :data:`OFFSET_PAST_END`
|
||||
teaching naming the true line count, the char cap shrinks the extent
|
||||
— the same strings ``read`` uses, one copy each), images belong to
|
||||
``show_image`` (a show of an ``is_image`` row is the
|
||||
:data:`SHOW_TEXT_NOT_A_TEXT_DOC` refusal, the exact mirror of
|
||||
:data:`NOT_AN_IMAGE_DOC` — never a silent conversion), and the inline
|
||||
``` ```bor ``` locator-fence form of the same contract (phase 182 D2)
|
||||
is resolved at the API layer (task 05), NOT here.
|
||||
4. Every call the model emits is appended back to the message history as
|
||||
the assistant tool-call message + the tool result (refusals included),
|
||||
consumes one round, and the model is called again. Every tool result
|
||||
@@ -317,6 +340,7 @@ from app.rag.llm import (
|
||||
from app.rag.retriever import TRUNCATION_MARKER, _fold
|
||||
from app.rag.scaffolding import ScaffoldingFilter
|
||||
from app.rag.source_removal import resolve_source_name
|
||||
from app.rag.verbatim import VerbatimBlock, block_for
|
||||
|
||||
logger = logging.getLogger("app.agent")
|
||||
|
||||
@@ -535,20 +559,104 @@ SHOW_IMAGE_TOOL: dict[str, Any] = {
|
||||
}
|
||||
|
||||
|
||||
def agent_tools(settings: Settings) -> list[dict[str, Any]]:
|
||||
"""The grounded turn's tool offer list (phase 124, task 02).
|
||||
#: The 5th agent tool (phase 182, task 03; LOCKED A1–A3/A6) — offered
|
||||
#: ONLY when ``settings.answer_text`` (``BOR_ANSWER_TEXT``, default ON):
|
||||
#: :func:`agent_tools` composes the offer list (the three base tools stay
|
||||
#: :data:`AGENT_TOOLS`, byte-identical). The description IS the behavior
|
||||
#: contract (the phase-124 discipline): the model NAMES a location, the
|
||||
#: SERVER shows the bytes verbatim — the user copies what is displayed,
|
||||
#: so the model must NOT retype it. Same OpenAI function shape as the
|
||||
#: other tools: one required string ``path`` (the combined
|
||||
#: ``source/path`` identity — the bare-path-will-not-resolve teaching
|
||||
#: mirrored from ``read``) plus the OPTIONAL ``offset``/``limit`` line
|
||||
#: range (the ``read`` wording reused, so every line-number surface in
|
||||
#: the app teaches one grammar).
|
||||
SHOW_TEXT_TOOL: dict[str, Any] = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "show_text",
|
||||
"description": (
|
||||
"Display part of a document to the user EXACTLY as stored "
|
||||
"— the user sees the server's bytes and can copy them. "
|
||||
"Call it when the user wants to see, copy or run a file (a "
|
||||
"quadlet, a config, a script): they copy from what is "
|
||||
"displayed, so do NOT retype the content yourself after "
|
||||
"calling it — describe it instead. `path` is the combined "
|
||||
"`source/path` string exactly as shown in `ls`. `offset` "
|
||||
"(1-based first line) and `limit` (line count) are "
|
||||
"optional — omit them to display the whole document; use "
|
||||
"them when `grep`/`read` told you which lines matter. The "
|
||||
"displayed line numbers come from the server, not from "
|
||||
"your arguments: an overshooting `limit` is clamped and an "
|
||||
"`offset` past the end is refused with the true line "
|
||||
"count. Only text documents — for an image use "
|
||||
"`show_image`. Call one tool at a time — wait for this "
|
||||
"result before your next call."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"path": {
|
||||
"type": "string",
|
||||
"description": (
|
||||
"The document to display, as the combined "
|
||||
"`source/path` string exactly as shown in the "
|
||||
"`ls` output (e.g. "
|
||||
"'homelab/active/container_caddy/caddy.md'). "
|
||||
"A bare document path (without the source "
|
||||
"name) will not resolve. Only text documents "
|
||||
"— for an image use `show_image`."
|
||||
),
|
||||
},
|
||||
"offset": {
|
||||
"type": "integer",
|
||||
"description": (
|
||||
"The first line to display, 1-based. Omit it "
|
||||
"to display from the beginning of the "
|
||||
"document. Use it to display a specific part "
|
||||
"of a large document — grep reports the line "
|
||||
"numbers of what you found."
|
||||
),
|
||||
},
|
||||
"limit": {
|
||||
"type": "integer",
|
||||
"description": (
|
||||
"The maximum number of lines to display, "
|
||||
"starting at offset. Omit it to display as "
|
||||
"many lines as fit under the display cap. Use "
|
||||
"it to show the part the user asked about — "
|
||||
"you are showing a part, not the document."
|
||||
),
|
||||
},
|
||||
},
|
||||
"required": ["path"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
The three base tools (:data:`AGENT_TOOLS`, byte-identical) plus
|
||||
|
||||
def agent_tools(settings: Settings) -> list[dict[str, Any]]:
|
||||
"""The grounded turn's tool offer list (phase 124, task 02; the 5th
|
||||
tool, phase 182, task 03).
|
||||
|
||||
The three base tools (:data:`AGENT_TOOLS`, byte-identical) plus the
|
||||
TWO display tools, each gated by its own flag:
|
||||
:data:`SHOW_IMAGE_TOOL` when ``settings.answer_images``
|
||||
(``BOR_ANSWER_IMAGES``) is true — otherwise exactly the three base
|
||||
tools (the pre-phase-124 offer, byte-identical: no 4th tool in the
|
||||
wire ``tools`` array). A FRESH list on every call — callers never
|
||||
mutate the result or the constants (``run_agent`` passes it straight
|
||||
to the LLM request).
|
||||
(``BOR_ANSWER_IMAGES``) and :data:`SHOW_TEXT_TOOL` when
|
||||
``settings.answer_text`` (``BOR_ANSWER_TEXT``) — the offer order is
|
||||
``ls``, ``read``, ``grep``, ``show_image``, ``show_text``. With both
|
||||
flags off: exactly the three base tools (the pre-phase-124 offer,
|
||||
byte-identical: no display tool in the wire ``tools`` array); with
|
||||
only ``answer_text`` off it is the pre-phase-182 offer
|
||||
byte-identical. A FRESH list on every call — callers never mutate
|
||||
the result or the constants (``run_agent`` passes it straight to the
|
||||
LLM request).
|
||||
"""
|
||||
tools: list[dict[str, Any]] = list(AGENT_TOOLS)
|
||||
if settings.answer_images:
|
||||
tools.append(SHOW_IMAGE_TOOL)
|
||||
if settings.answer_text:
|
||||
tools.append(SHOW_TEXT_TOOL)
|
||||
return tools
|
||||
|
||||
#: Tool refusal texts (phase 37): rejected calls count in nothing
|
||||
@@ -587,6 +695,56 @@ ALREADY_SHOWING = (
|
||||
"Answer from what you have."
|
||||
)
|
||||
|
||||
#: The ``show_text`` refusals (phase 182, task 03): the same contract as
|
||||
#: every other refusal — counts in nothing, consumes a round, changes no
|
||||
#: context. ``MISSING_SHOW_TEXT_ARGS`` mirrors ``MISSING_SHOW_ARGS``.
|
||||
#: ``SHOW_TEXT_NOT_A_TEXT_DOC``: one ``{arg}`` field, the argument echoed
|
||||
#: as passed — the row is a standalone IMAGE; image display belongs to
|
||||
#: ``show_image`` (LOCKED A6 — the exact mirror of
|
||||
#: :data:`NOT_AN_IMAGE_DOC`, never a silent conversion). The range
|
||||
#: problems REUSE :data:`BAD_RANGE_ARGS` and :data:`OFFSET_PAST_END`
|
||||
#: verbatim — the same teaching, the same echoed values, one copy of
|
||||
#: each string in the codebase.
|
||||
MISSING_SHOW_TEXT_ARGS = "show_text requires a string argument 'path'."
|
||||
SHOW_TEXT_NOT_A_TEXT_DOC = (
|
||||
"'{arg}' is a standalone image document — use show_image to display "
|
||||
"images."
|
||||
)
|
||||
#: One ``{max}`` field (``settings.show_text_max_blocks``): the
|
||||
#: per-answer block budget, enforced SERVER-side BEFORE the resolve work
|
||||
#: (the model is taught the budget in the description; the handler is
|
||||
#: the actual gate). ``0`` therefore disables the tool's effect while
|
||||
#: keeping it offered (the house kill-switch pattern).
|
||||
SHOW_TEXT_BLOCK_CAP = (
|
||||
"{max} text blocks are already displayed this answer — describe them "
|
||||
"instead of showing more."
|
||||
)
|
||||
#: One ``{locator}`` field — the canonical ``source/path[#first-last]``
|
||||
#: the model just asked for: the dedupe mirror of
|
||||
#: :data:`ALREADY_SHOWING`, but per BLOCK under the block budget (the
|
||||
#: budget allows SEVERAL different blocks; the same block twice teaches).
|
||||
SHOW_TEXT_ALREADY_SHOWN = (
|
||||
"{locator} is already on screen this answer — say so instead of "
|
||||
"showing it again."
|
||||
)
|
||||
|
||||
#: The ``show_text`` truncation notice (phase 182, task 03) — appended as
|
||||
#: the result's LAST line when the per-block char budget
|
||||
#: (``settings.show_text_max_chars``) cut the displayed slice (the block
|
||||
#: text already ends with the shared :data:`TRUNCATION_MARKER` on its own
|
||||
#: line, and the ``lines:`` extent already shrank to the last line
|
||||
#: actually displayed). In the :data:`READ_TRUNCATION_NOTICE` spirit it
|
||||
#: names the truth the model must not paper over — the rest is NOT
|
||||
#: displayed — and the way the user gets it (open the document; NO
|
||||
#: ``offset``/``limit`` re-show loop is implied, only the displayed
|
||||
#: slice is what the user copies). Fields: ``{source}``/``{path}``, the
|
||||
#: RESOLVED identity, plain (no ``!r``).
|
||||
SHOWN_TEXT_TRUNCATION_NOTICE = (
|
||||
"TRUNCATED — the rest of {source}/{path} is NOT displayed. The user "
|
||||
"can open the document to read it in full; describe the displayed "
|
||||
"lines instead of showing it again."
|
||||
)
|
||||
|
||||
#: The read-truncation notice (phase 95, task 01; REWRITTEN phase 131,
|
||||
#: task 01) — appended, after the shared :data:`TRUNCATION_MARKER`, to
|
||||
#: a ``read`` result whose document is LONGER than
|
||||
@@ -1642,6 +1800,17 @@ class AgentHolder:
|
||||
nothing). Read by ``app.api.chat`` after the stream for the done
|
||||
frame's ``answer_image`` field (task 04) — the ``read_docs``
|
||||
precedent (an API-layer read-after-stream signal).
|
||||
``shown_texts`` (phase 182, task 03): the verbatim blocks
|
||||
``show_text`` displayed, in CALL order, capped by
|
||||
``settings.show_text_max_blocks`` (a call over the budget is refused
|
||||
with :data:`SHOW_TEXT_BLOCK_CAP` and a duplicate of a shown locator
|
||||
with :data:`SHOW_TEXT_ALREADY_SHOWN` — both append nothing). Each
|
||||
entry is the RESOLVED
|
||||
:class:`app.rag.verbatim.VerbatimBlock` — the server's bytes plus
|
||||
the server's extent (the canonical ``locator`` is the dedupe key and
|
||||
the client's matching key, D7). Read by ``app.api.chat`` after the
|
||||
stream for the done frame's ``answer_text`` array (task 05) — the
|
||||
``shown_images``/``read_docs`` read-after-stream precedent.
|
||||
``read_coverage`` (phase 141, task 01): one ``(source, path,
|
||||
first_line, last_line)`` record per covered span — INCLUSIVE line
|
||||
numbers over the document's ``content.splitlines()``, in read
|
||||
@@ -1676,6 +1845,7 @@ class AgentHolder:
|
||||
read_truncations: list[tuple[str, int, int]] = field(default_factory=list)
|
||||
shown_images: list[Document] = field(default_factory=list)
|
||||
read_coverage: list[tuple[str, str, int, int]] = field(default_factory=list)
|
||||
shown_texts: list[VerbatimBlock] = field(default_factory=list)
|
||||
|
||||
|
||||
def _execute_tool(
|
||||
@@ -2065,6 +2235,113 @@ def _execute_tool(
|
||||
f"{IMAGE_DOC_MARKER}\n"
|
||||
f"{doc.content}"
|
||||
)
|
||||
if call.name == "show_text":
|
||||
# Phase 182 (task 03; LOCKED A1–A3/A6): the 5th agent tool — the
|
||||
# model NAMES a location, the SERVER shows the bytes verbatim.
|
||||
# Offered only when ``settings.answer_text`` (``agent_tools``);
|
||||
# the handler enforces the contract server-side. Every refusal
|
||||
# below appends nothing, increments nothing and consumes a
|
||||
# round, like every other refusal.
|
||||
raw_path = call.arguments.get("path")
|
||||
arg = raw_path.strip() if isinstance(raw_path, str) else ""
|
||||
if not arg:
|
||||
return MISSING_SHOW_TEXT_ARGS
|
||||
if not settings.answer_text:
|
||||
# Defensive — the tool is NOT OFFERED while the flag is off,
|
||||
# so from the model's side this is a tool that does not
|
||||
# exist: the same refusal an un-offered name gets (the
|
||||
# tools-off posture — "Unknown tool."), counts in nothing.
|
||||
return UNKNOWN_TOOL
|
||||
doc, _source, _path = _resolve_path(db, arg)
|
||||
if doc is None:
|
||||
# The same phase-72 "did you mean …?" teaching as the read
|
||||
# branch (a refusal — not counted, no context, nothing
|
||||
# displayed).
|
||||
return _no_document_refusal(db, arg)
|
||||
if doc.is_image:
|
||||
# LOCKED A6: an image row is refused, NEVER silently
|
||||
# converted — image display is ``show_image``'s job (the
|
||||
# mirror of the show_image branch's NOT_AN_IMAGE_DOC).
|
||||
return SHOW_TEXT_NOT_A_TEXT_DOC.format(arg=arg)
|
||||
# The OPTIONAL line range — the SAME validator, the SAME echoed
|
||||
# BAD_RANGE_ARGS teaching as ``read`` (one copy of each string);
|
||||
# ``offset`` without ``limit`` displays to the end, ``limit``
|
||||
# without ``offset`` counts from line 1 — ``slice_lines`` applies
|
||||
# both one-sided semantics with ``read``'s exact indexing.
|
||||
ok, range_error, offset, limit = _read_range_args(
|
||||
call.arguments.get("offset"), call.arguments.get("limit")
|
||||
)
|
||||
if not ok:
|
||||
return range_error
|
||||
# The per-answer block budget — checked BEFORE the resolve work
|
||||
# (and, with a 0 char budget, before ``cap_slice`` could raise):
|
||||
# either zeroed kill switch refuses like a spent budget, so
|
||||
# ``0`` disables the tool's EFFECT while it stays offered (D8).
|
||||
block_budget = (
|
||||
settings.show_text_max_blocks
|
||||
if settings.show_text_max_chars > 0
|
||||
else 0
|
||||
)
|
||||
if len(holder.shown_texts) >= block_budget:
|
||||
return SHOW_TEXT_BLOCK_CAP.format(max=block_budget)
|
||||
block = block_for(
|
||||
db,
|
||||
arg,
|
||||
offset=offset,
|
||||
limit=limit,
|
||||
cap=settings.show_text_max_chars,
|
||||
doc=doc,
|
||||
)
|
||||
if block is None:
|
||||
# An empty document or an ``offset`` past the last line —
|
||||
# the SAME OFFSET_PAST_END teaching as ``read``, the true
|
||||
# line count named (the same ``len(content.splitlines())``
|
||||
# expression ``read`` uses). Counts in nothing; a corrected
|
||||
# retry still proceeds.
|
||||
return OFFSET_PAST_END.format(
|
||||
source=doc.source,
|
||||
path=doc.path,
|
||||
total_lines=len(doc.content.splitlines()),
|
||||
offset=offset if offset is not None else 1,
|
||||
)
|
||||
if any(shown.locator == block.locator for shown in holder.shown_texts):
|
||||
# The canonical-locator dedupe (D7): the identical block is
|
||||
# already on screen — say so instead of painting it twice.
|
||||
return SHOW_TEXT_ALREADY_SHOWN.format(locator=block.locator)
|
||||
# Success: the block rides the holder for the done frame
|
||||
# (task 05) and the call counts (executed-only, the house rule).
|
||||
holder.shown_texts.append(block)
|
||||
holder.tool_calls += 1
|
||||
# The result contract (D5 — the ``read``/``show_image`` header
|
||||
# contract the E2E mock's ``_READ_RESULT_PREFIX`` parses): a NEW
|
||||
# first line (absent from ``read`` — the model must know the
|
||||
# show registered AND must not retype), then the exact document
|
||||
# headers ``read`` established (date line, phase-145 origin
|
||||
# lines), then the SERVER's resolved extent — clamped ``limit``
|
||||
# and cap-shrunk end included, the caption never lies — and the
|
||||
# verbatim slice itself (A21: LLM calls are free, and returning
|
||||
# the text means the model's commentary can never disagree with
|
||||
# the panel).
|
||||
origin = origin_for_source(db, doc.source)
|
||||
result = (
|
||||
"Text shown to the user — these lines are displayed "
|
||||
"verbatim; describe them, do NOT retype them.\n"
|
||||
f"Document {doc.source}/{doc.path}:\n"
|
||||
f"date: {doc.created_at:%Y-%m-%d}\n"
|
||||
f"{_origin_lines(origin)}"
|
||||
f"lines: {block.start_line}-{block.end_line} of "
|
||||
f"{block.total_lines}\n"
|
||||
f"{block.text}"
|
||||
)
|
||||
if block.truncated:
|
||||
# The char cap cut the slice: the block already ends with
|
||||
# the shared marker on its own line — append the pinned
|
||||
# truth (the rest is NOT displayed; the user opens the
|
||||
# document; no re-show loop implied).
|
||||
result += "\n" + SHOWN_TEXT_TRUNCATION_NOTICE.format(
|
||||
source=doc.source, path=doc.path
|
||||
)
|
||||
return result
|
||||
if call.name == "grep":
|
||||
raw_pattern = call.arguments.get("pattern")
|
||||
pattern = raw_pattern.strip() if isinstance(raw_pattern, str) else ""
|
||||
@@ -2229,11 +2506,13 @@ async def run_agent(
|
||||
# Phase 45: no per-tool budgets — the tools stay offered for the
|
||||
# whole turn, bounded by the round cap. ``0`` is the no-tools kill
|
||||
# switch: exactly one request with ``tools=None`` (the pre-phase-37
|
||||
# path). Phase 124 (task 02): the offer list is :func:`agent_tools`
|
||||
# — the three base tools (byte-identical) plus ``show_image`` only
|
||||
# when ``settings.answer_images`` (``BOR_ANSWER_IMAGES``) is on;
|
||||
# off (the default) the offer is element-for-element the
|
||||
# pre-phase-124 list.
|
||||
# path). Phase 124 (task 02) / 182 (task 03): the offer list is
|
||||
# :func:`agent_tools` — the three base tools (byte-identical) plus
|
||||
# ``show_image`` only when ``settings.answer_images``
|
||||
# (``BOR_ANSWER_IMAGES``) and ``show_text`` only when
|
||||
# ``settings.answer_text`` (``BOR_ANSWER_TEXT``, default ON); both
|
||||
# off: the offer is element-for-element the pre-phase-124 list,
|
||||
# ``answer_text`` alone off: the pre-phase-182 list.
|
||||
max_rounds = settings.agent_max_rounds
|
||||
tools: list[dict[str, Any]] | None = (
|
||||
agent_tools(settings) if max_rounds > 0 else None
|
||||
|
||||
+105
-7
@@ -66,6 +66,20 @@ backstop). The ``DEFLECT_MODE`` marker and everything else in the
|
||||
prompt stay put — the E2E mock LLM keys on the marker's *presence*,
|
||||
not the wording, so that contract is unchanged.
|
||||
|
||||
Display-tool clauses (phase 124 ``show_image``; phase 182 ``show_text``):
|
||||
the **HIGH** prompt's ``<tools>`` section can carry two extra clauses
|
||||
before its closing tag, each appended only when its own setting is on —
|
||||
:data:`SHOW_IMAGE_CLAUSE` when ``BOR_ANSWER_IMAGES`` (default OFF) and
|
||||
:data:`SHOW_TEXT_CLAUSE` when ``BOR_ANSWER_TEXT`` (default ON). The text
|
||||
clause teaches the two things the phase-182 incident needed said: the
|
||||
user copies from what is DISPLAYED (so never retype a file), and the
|
||||
inline `` ```bor <source/path> [range] ``` `` locator fence whose EMPTY
|
||||
body the app replaces with the server-resolved bytes. Both clauses
|
||||
compose through one mechanism (section minus its final tag + the present
|
||||
clauses + the tag), so a build with neither clause is byte-identical to
|
||||
pre-phase and a ``show_image``-only build reproduces the phase-124 bytes
|
||||
exactly; the LOW prompt carries neither.
|
||||
|
||||
Disclosed-answer contract (phase 112, task 03; owner decision
|
||||
2026-09-14, roadmap confirmation — option (iii) of the three the
|
||||
2026-09-15 interactive deflection test raised): that test found the
|
||||
@@ -388,6 +402,61 @@ SHOW_IMAGE_CLAUSE: str = (
|
||||
"in text either way."
|
||||
)
|
||||
|
||||
#: The ``show_text`` contract clause (phase 182, task 04, LOCKED A1–A3/A6) —
|
||||
#: appended INSIDE the ``<tools>`` section of the HIGH prompt, after the
|
||||
#: existing three-tool copy (and after :data:`SHOW_IMAGE_CLAUSE` when that
|
||||
#: rides too), ONLY when :func:`build_high_prompt` is called with
|
||||
#: ``show_text=True`` (the ``BOR_ANSWER_TEXT`` master switch, via
|
||||
#: ``app.api.chat``'s grounded plan — the only call site that passes it).
|
||||
#: It exists because the incident this phase fixes is a PROMPT gap, not a
|
||||
#: tool gap: nothing ever told the model that the user copies from what is
|
||||
#: DISPLAYED, so it retyped files out of context and every retype differed
|
||||
#: from the stored bytes by a character. What each sentence pins, in order:
|
||||
#: (1) the display contract — the tool shows the document EXACTLY as stored
|
||||
#: and the copy source is the display, not the prose; (2) when to call it —
|
||||
#: the user wants to SEE, COPY or RUN a file (the owner's quadlet / config /
|
||||
#: script shapes) rather than read content out of the answer; (3) the do-
|
||||
#: NOT-retype rule with its reason (anything the model types may differ by
|
||||
#: one character and the user cannot tell — the phase's reason for
|
||||
#: existing); (4) the placement syntax — the inline `` ```bor <source/path>
|
||||
#: [range] ``` `` locator fence with an EMPTY body (D1/D7: a fence is a
|
||||
#: resolution request, the app supplies the text, and D3 keeps the model's
|
||||
#: body out of the DOM); (5) the honesty-about-numbers rule — line numbers
|
||||
#: come from ``grep``/``read``, and when unsure the range is OMITTED
|
||||
#: because the displayed extent is the server's (A2); (6) the failure rule
|
||||
#: — refuse out loud, never reconstruct from memory (the persona's radical
|
||||
#: honesty, D3's posture on the model side); (7) the image pointer
|
||||
#: (LOCKED A6: images belong to ``show_image``, never silently converted).
|
||||
#: ``TOOLS_SECTION`` itself stays BYTE-IDENTICAL (the sha pins in
|
||||
#: :mod:`tests.unit.test_prompt_lock` hold); a ``show_text=False`` build
|
||||
#: (the default — every pre-phase call site) is byte-identical to the
|
||||
#: pre-task output, and a ``show_image=True, show_text=False`` build
|
||||
#: reproduces the phase-124 bytes EXACTLY. The ``<tools>`` marker the E2E
|
||||
#: mock keys on is untouched either way — both clauses ride on its
|
||||
#: presence, not its wording. The LOW/deflection prompt never carries the
|
||||
#: clause: a deflected turn has no tools, and phase 182's D9 forbids
|
||||
#: resolving a locator on a turn that just disclaimed the knowledge base.
|
||||
SHOW_TEXT_CLAUSE: str = (
|
||||
"`show_text` displays part of a document to the user EXACTLY as stored "
|
||||
"— the user copies from what is displayed, not from your prose. Call "
|
||||
"it whenever the user wants to SEE, COPY or RUN a file — a quadlet, a "
|
||||
"config, a script, a unit file — instead of writing the content into "
|
||||
"your answer yourself. After calling it, do NOT repeat the displayed "
|
||||
"lines in your answer — describe them, explain them, point at them. "
|
||||
"Anything you type yourself may differ from the file by a character, "
|
||||
"and the user cannot tell. To put a displayed block exactly where you "
|
||||
"want it in your answer, write a locator line on its own: three "
|
||||
"backticks followed by `bor`, a space, the combined `source/path`, and "
|
||||
"optionally the line range (` 12-40`). Leave its body EMPTY — the app "
|
||||
"replaces it with the exact stored text. Never type the file's content "
|
||||
"inside a `bor` block. Never guess line numbers — use the numbers "
|
||||
"`grep` and `read` reported. When you are not sure which lines, OMIT "
|
||||
"the range and display the whole document: the numbers you display are "
|
||||
"the server's, never yours. If `show_text` refuses, say what you could "
|
||||
"not display — do not reconstruct it from memory. For an image use "
|
||||
"`show_image` instead."
|
||||
)
|
||||
|
||||
|
||||
def history_to_messages(
|
||||
history: Sequence[HistoryTurn],
|
||||
@@ -762,6 +831,7 @@ def build_high_prompt(
|
||||
kb_overview: str | None = None,
|
||||
folder_lines: Sequence[str] = (),
|
||||
show_image: bool = False,
|
||||
show_text: bool = False,
|
||||
) -> str:
|
||||
"""Grounded turn: locked persona (+ steering, + KB overview) + the
|
||||
top-ranked documents' SUMMARY blocks + the ``<tools>`` instructions
|
||||
@@ -771,7 +841,8 @@ def build_high_prompt(
|
||||
seeds SUMMARIES, never full texts — full text enters the context
|
||||
only through the capped ``read`` tool; phase 119, LOCKED A4: the
|
||||
suggested-folder context lines, below; phase 124, task 03, LOCKED
|
||||
A5: the optional :data:`SHOW_IMAGE_CLAUSE`, below).
|
||||
A5: the optional :data:`SHOW_IMAGE_CLAUSE`, below; phase 182, task 04,
|
||||
LOCKED A1–A3: the optional :data:`SHOW_TEXT_CLAUSE`, below).
|
||||
|
||||
Section order: ``<relevance>`` → ``<knowledge_base>`` → ``<tuning>``
|
||||
→ ``<documents>`` → ``<tools>``; empty steering/overview omit their
|
||||
@@ -796,6 +867,22 @@ def build_high_prompt(
|
||||
never carries the clause — a deflected turn has no tools and never
|
||||
shows an image.
|
||||
|
||||
*show_text* (phase 182, task 04, LOCKED A1–A3): when true (the caller
|
||||
passes the ``BOR_ANSWER_TEXT`` master switch — the same single grounded
|
||||
call site), the :data:`SHOW_TEXT_CLAUSE` rides INSIDE the same
|
||||
``<tools>`` section, AFTER :data:`SHOW_IMAGE_CLAUSE` when that is
|
||||
present too and always before the closing tag. The two flags compose
|
||||
independently through one mechanism (the built section is
|
||||
``TOOLS_SECTION`` minus its final ``</tools>`` + the present clauses,
|
||||
blank-line separated, + the closing tag): ``show_image`` only
|
||||
reproduces the phase-124 bytes exactly, ``show_text`` only is the
|
||||
three-tool copy plus the text clause, both is the image clause then
|
||||
the text clause, neither is byte-identical to pre-phase. The default
|
||||
``False`` keeps every pre-existing call byte-identical. The deflection
|
||||
prompt NEVER carries it — a deflected turn has no tools, and phase
|
||||
182's D9 forbids resolving a ``bor`` locator on a turn whose answer
|
||||
just disclaimed the knowledge base.
|
||||
|
||||
When at least one block is present, the section leads with the
|
||||
start-here :data:`SUGGEST_INTRO` line (before the first block — the
|
||||
phase-15 ``_STEERING_INTRO`` / phase-31 ``_KB_INTRO`` precedent): the
|
||||
@@ -861,13 +948,24 @@ def build_high_prompt(
|
||||
for part in (build_kb_section(kb_overview or ""), build_steering_section(notes or [])):
|
||||
if part:
|
||||
prompt += "\n" + part
|
||||
# Phase 124 (task 03, LOCKED A5) / phase 182 (task 04, LOCKED A1): the
|
||||
# display-tool clauses ride inside <tools> — TOOLS_SECTION's text minus
|
||||
# its final </tools>, the clauses offered for this build (each on its
|
||||
# own line, blank-line separated, after the existing three-tool copy),
|
||||
# then the closing tag. TOOLS_SECTION itself is unedited, the
|
||||
# no-clause path keeps the exact pre-task concatenation, and a
|
||||
# show_image-only build reproduces the phase-124 bytes exactly.
|
||||
clauses: list[str] = []
|
||||
if show_image:
|
||||
# Phase 124 (task 03, LOCKED A5): the clause rides inside
|
||||
# <tools> — TOOLS_SECTION's text minus its final </tools>, the
|
||||
# clause (on its own line, after the existing three-tool copy),
|
||||
# then the closing tag. TOOLS_SECTION itself is unedited and the
|
||||
# show_image=False path keeps the exact pre-task concatenation.
|
||||
tools_section = TOOLS_SECTION[: -len("</tools>")] + SHOW_IMAGE_CLAUSE + "\n</tools>"
|
||||
clauses.append(SHOW_IMAGE_CLAUSE)
|
||||
if show_text:
|
||||
clauses.append(SHOW_TEXT_CLAUSE)
|
||||
if clauses:
|
||||
tools_section = (
|
||||
TOOLS_SECTION[: -len("</tools>")]
|
||||
+ "\n\n".join(clauses)
|
||||
+ "\n</tools>"
|
||||
)
|
||||
else:
|
||||
tools_section = TOOLS_SECTION
|
||||
return prompt + "\n<documents>\n" + body + "\n</documents>\n" + tools_section
|
||||
|
||||
@@ -0,0 +1,520 @@
|
||||
"""Phase 182 (task 02): the verbatim locator grammar, the slicer, the resolver.
|
||||
|
||||
The contract in one line: **the model names a location, this module supplies
|
||||
the bytes** (PLAN §6's tool-surface extension, phase 182 D1–D7). An answer
|
||||
never carries a model-retyped copy of a source file; it carries a ``bor``
|
||||
locator fence (````` ```bor homelab/runbooks/nginx.md#12-40 `````), and the
|
||||
bytes shown to the human are read out of the indexed ``documents.content``
|
||||
row by :func:`block_for` / :func:`resolve_answer_blocks`. Because the
|
||||
displayed text is never generated, it cannot be paraphrased, "corrected",
|
||||
or hallucinated — the phase's whole point — and the same resolver serves
|
||||
both entry points (the ``show_text`` tool's own block, task 03, and the
|
||||
``done``-frame pass over the finished answer, task 05).
|
||||
|
||||
The module is deliberately split in two halves:
|
||||
|
||||
* the **pure** half — :func:`canonical_locator`, :func:`fences_in`,
|
||||
:func:`slice_lines`, :func:`cap_slice` — no DB, no settings, no I/O;
|
||||
* the **resolver** half — :func:`document_by_identity`, :func:`block_for`,
|
||||
:func:`resolve_answer_blocks` — one bare ``SELECT`` per locator and the
|
||||
kill-switch gate.
|
||||
|
||||
**Import rule (one-way, checkable with
|
||||
``grep -n "import" app/rag/verbatim.py``):** this module must NEVER import
|
||||
the agent module (:mod:`app.rag.agent`) — ``agent.py`` already depends on
|
||||
the pure half of this module for the ``show_text`` tool branch, so a
|
||||
reverse dependency cycles. Allowed here: ``sqlalchemy``,
|
||||
:class:`app.models.Document`,
|
||||
:data:`app.rag.retriever.TRUNCATION_MARKER`, :class:`app.config.Settings`,
|
||||
stdlib.
|
||||
|
||||
Nothing calls this module yet (tasks 03 and 05 wire it in), so landing it
|
||||
changes no behavior anywhere.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import Document
|
||||
from app.rag.retriever import TRUNCATION_MARKER
|
||||
|
||||
__all__ = [
|
||||
"LOCATOR_LANG",
|
||||
"VerbatimBlock",
|
||||
"SliceResult",
|
||||
"block_for",
|
||||
"cap_slice",
|
||||
"canonical_locator",
|
||||
"document_by_identity",
|
||||
"fences_in",
|
||||
"resolve_answer_blocks",
|
||||
"slice_lines",
|
||||
]
|
||||
|
||||
#: The fence info-string token that marks a locator fence (D2). The
|
||||
#: markdown renderer (task 06) and the copy-path builder (task 07) key on
|
||||
#: this SAME constant — one grammar, one spelling, no drifting copy.
|
||||
LOCATOR_LANG = "bor"
|
||||
|
||||
#: The range separators a locator may use between the identity and the
|
||||
#: line range: a space, a colon, a bare ``#``, or ``#L`` (D7 — the grammar
|
||||
#: is TOLERANT on input because the string is model-written).
|
||||
_SEP = r"[ \t:#]+"
|
||||
|
||||
#: A line range: ``12-40``, ``12–40`` (en dash), ``L12-L40``, or a bare
|
||||
#: ``12`` (one line). An optional ``L`` prefixes either bound; whitespace
|
||||
#: around the dash is tolerated. Anchored at the end of the string by the
|
||||
#: caller, so a range-shaped tail followed by more text is junk, not a
|
||||
#: range.
|
||||
_RANGE = r"(?P<range>L?[0-9]+(?:[ \t]*[-\u2013][ \t]*L?[0-9]+)?)"
|
||||
|
||||
#: The trailing range of a locator string (see :data:`_SEP` /
|
||||
#: :data:`_RANGE`). Case-insensitive only for the ``L`` prefix.
|
||||
_TRAILING_RANGE_RE = re.compile(f"{_SEP}{_RANGE}$", re.IGNORECASE)
|
||||
|
||||
#: The opening line of a ``bor`` locator fence: the language token,
|
||||
#: whitespace, then a NON-EMPTY remainder (the locator). A bare
|
||||
#: `` ```bor `` carries no locator, so it never matches and stays an
|
||||
#: ordinary code block.
|
||||
_FENCE_OPEN_RE = re.compile(
|
||||
rf"^```{re.escape(LOCATOR_LANG)}[ \t]+(?P<locator>.*[^ \t])[ \t]*$"
|
||||
)
|
||||
|
||||
#: The two characters a bound may use as its line-number prefix.
|
||||
_LINE_PREFIX = "Ll"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class VerbatimBlock:
|
||||
"""One displayed verbatim block — the bytes plus the truth about them.
|
||||
|
||||
``start_line`` / ``end_line`` are **1-based and INCLUSIVE** and describe
|
||||
EXACTLY the lines :attr:`text` contains *after* any character cap — a
|
||||
capped block's ``end_line`` is the last line actually shown, never the
|
||||
line the locator asked for (the caption must not claim lines nobody
|
||||
sees). ``total_lines`` is the whole document's line count, so the
|
||||
caption can say ``lines 12-40 of 512``. :attr:`text` never carries a
|
||||
trailing newline (lines are joined with ``"\\n"``), and
|
||||
:attr:`truncated` says whether :data:`~app.rag.retriever.TRUNCATION_MARKER`
|
||||
was appended on its own last line.
|
||||
|
||||
:attr:`locator` is the CANONICAL form (D7) built by the factory
|
||||
(:func:`block_for`), never passed in by a caller — it is the dedupe key
|
||||
of :func:`resolve_answer_blocks` and the matching key the copy-path
|
||||
builder uses (task 07).
|
||||
"""
|
||||
|
||||
source: str
|
||||
path: str
|
||||
title: str
|
||||
start_line: int
|
||||
end_line: int
|
||||
total_lines: int
|
||||
text: str
|
||||
truncated: bool = False
|
||||
locator: str = ""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SliceResult:
|
||||
"""The outcome of one line-range slice (see :func:`slice_lines`).
|
||||
|
||||
``first_line`` / ``last_line`` are 1-based and inclusive; ``past_end``
|
||||
is the caller's refusal signal and always travels with the true
|
||||
``total_lines`` (the teaching refusal names the real line count).
|
||||
"""
|
||||
|
||||
text: str
|
||||
first_line: int
|
||||
last_line: int
|
||||
total_lines: int
|
||||
past_end: bool
|
||||
|
||||
|
||||
def canonical_locator(raw: str) -> str | None:
|
||||
"""Normalize one model-written locator string, or refuse it (D7).
|
||||
|
||||
Accepted forms — ``source/path`` (the ``read``/``grep`` identity
|
||||
contract, split at the FIRST ``'/'``) followed by an optional separator
|
||||
and line range:
|
||||
|
||||
========================== ==========================
|
||||
input canonical output
|
||||
========================== ==========================
|
||||
``homelab/a.md`` ``homelab/a.md``
|
||||
``homelab/a.md 12-40`` ``homelab/a.md#12-40``
|
||||
``homelab/a.md:12-40`` ``homelab/a.md#12-40``
|
||||
``homelab/a.md#12-40`` ``homelab/a.md#12-40``
|
||||
``homelab/a.md#L12-L40`` ``homelab/a.md#12-40``
|
||||
``homelab/a.md 12\u201340`` ``homelab/a.md#12-40``
|
||||
``homelab/a.md 12`` ``homelab/a.md#12-12``
|
||||
========================== ==========================
|
||||
|
||||
The canonical form is always ``source/path`` + ``#start-end`` with
|
||||
plain digits — never the en dash, never the ``L``, so the fence, the
|
||||
tool block and the UI share ONE string (D7).
|
||||
|
||||
Returns ``None`` (NOT a locator) for: empty/whitespace input, a value
|
||||
with no ``'/'`` in the path part, a non-numeric or non-positive bound,
|
||||
``start > end``, trailing junk after the range, or a leading ``bor``
|
||||
token the caller left in. No DB access, no case folding (paths are
|
||||
case-sensitive row values). A locator carrying a range also refuses an
|
||||
identity containing whitespace: the fence's info string is ONE
|
||||
whitespace-free identity plus the range, so a path that contains a
|
||||
space is expressed as a whole-document locator (no range) or not at
|
||||
all — the documented limit of the tolerant grammar.
|
||||
"""
|
||||
text = raw.strip()
|
||||
if not text:
|
||||
return None
|
||||
# A caller that pasted the fence's language token in has already lost
|
||||
# the identity — refuse rather than guess which token was the path.
|
||||
if text.split(" ", 1)[0] == LOCATOR_LANG:
|
||||
return None
|
||||
|
||||
match = _TRAILING_RANGE_RE.search(text)
|
||||
if match is None:
|
||||
identity, range_part = text, None
|
||||
else:
|
||||
# ``match.start()`` is where the separator begins — the identity
|
||||
# is everything before it (a multi-character separator, e.g. two
|
||||
# spaces, is consumed whole).
|
||||
identity, range_part = text[: match.start()], match.group("range")
|
||||
|
||||
# The identity is one whitespace-free ``source/path`` token, and it
|
||||
# never ends in a dangling separator (``a.md#`` with no range is a
|
||||
# typo, not a whole document).
|
||||
if not identity or any(c.isspace() for c in identity):
|
||||
return None
|
||||
if identity.endswith(("#", ":", "-")):
|
||||
return None
|
||||
if "/" not in identity:
|
||||
return None
|
||||
if range_part is None:
|
||||
return identity
|
||||
|
||||
bounds = [b.strip().lstrip(_LINE_PREFIX) for b in re.split(r"[-\u2013]", range_part)]
|
||||
# Defensive: :data:`_RANGE` already guarantees digit bounds, so this
|
||||
# only fires if the grammar above is ever loosened — it keeps
|
||||
# ``canonical_locator`` total (never an ``int()`` ValueError to a
|
||||
# caller that hands it model-written text).
|
||||
if any(not b.isdigit() for b in bounds):
|
||||
return None
|
||||
start = int(bounds[0])
|
||||
end = int(bounds[1]) if len(bounds) == 2 else start
|
||||
if start < 1 or end < 1 or start > end:
|
||||
return None
|
||||
return f"{identity}#{start}-{end}"
|
||||
|
||||
|
||||
def fences_in(answer: str) -> list[str]:
|
||||
"""Every ``bor`` locator fence's RAW locator string, in answer order.
|
||||
|
||||
Rules (the same shape ``markdown.js`` applies, task 06):
|
||||
|
||||
* the opening line is a fence whose info string is exactly
|
||||
`` ```bor `` + whitespace + a NON-empty remainder — a bare
|
||||
`` ```bor `` with no path is an ordinary code block, not a locator,
|
||||
and is not swallowed;
|
||||
* the fence closes at the next line that is nothing but backticks;
|
||||
* an **unclosed** fence contributes nothing (defensive: a half-streamed
|
||||
answer must not resolve a locator the model never finished) and ends
|
||||
the scan, since its body runs to the end of the answer;
|
||||
* the body is read and DISCARDED — this module never looks at what the
|
||||
model typed inside the fence (the bytes come from the row, D1);
|
||||
* nested `` ``` `` inside a body is NOT supported: the fence ends at
|
||||
the first closer (documented limitation, same rule as the renderer).
|
||||
|
||||
Duplicates collapse keeping the FIRST occurrence.
|
||||
"""
|
||||
lines = answer.splitlines()
|
||||
locators: list[str] = []
|
||||
index = 0
|
||||
while index < len(lines):
|
||||
match = _FENCE_OPEN_RE.match(lines[index].strip())
|
||||
if match is None:
|
||||
index += 1
|
||||
continue
|
||||
closer = _find_fence_close(lines, index + 1)
|
||||
if closer is None:
|
||||
# Unclosed: the body runs to the end of the answer, so there
|
||||
# is nothing further to scan and this fence yields nothing.
|
||||
break
|
||||
locator = match.group("locator").strip()
|
||||
if locator and locator not in locators:
|
||||
locators.append(locator)
|
||||
index = closer + 1
|
||||
return locators
|
||||
|
||||
|
||||
def _find_fence_close(lines: Sequence[str], start: int) -> int | None:
|
||||
"""Index of the fence's closing line at or after ``start``, ``None`` if unclosed."""
|
||||
for offset in range(start, len(lines)):
|
||||
stripped = lines[offset].strip()
|
||||
if stripped.startswith("```") and not stripped.strip("`"):
|
||||
return offset
|
||||
return None
|
||||
|
||||
|
||||
def slice_lines(
|
||||
content: str, offset: int | None = None, limit: int | None = None
|
||||
) -> SliceResult:
|
||||
"""Cut ``content`` to a 1-based line range with ``read``'s exact semantics.
|
||||
|
||||
Same indexing (``content.splitlines()``) and the same range arithmetic
|
||||
as the ``read`` tool (``app/rag/agent.py``) and ``grep``, so a line
|
||||
number the model got from ``grep`` names the same line here — the one
|
||||
shared coordinate system the whole phase depends on:
|
||||
|
||||
* ``offset`` omitted → 1; ``limit`` omitted → read to the end;
|
||||
* ``end = min(offset + limit - 1, total_lines)`` — an overshooting
|
||||
``limit`` CLAMPS (``last_line`` is the document's last line);
|
||||
* ``offset`` past the end, an empty document, or a ``limit`` below 1 →
|
||||
``past_end=True`` with empty text and the TRUE ``total_lines`` (the
|
||||
caller builds the teaching refusal).
|
||||
|
||||
Never raises for a range problem: the refusal is the caller's, the
|
||||
total always travels with the result.
|
||||
"""
|
||||
lines = content.splitlines()
|
||||
total_lines = len(lines)
|
||||
first = 1 if offset is None else offset
|
||||
end = total_lines if limit is None else min(first + limit - 1, total_lines)
|
||||
if total_lines == 0 or first > total_lines or end < first:
|
||||
return SliceResult(
|
||||
text="",
|
||||
first_line=first,
|
||||
last_line=0,
|
||||
total_lines=total_lines,
|
||||
past_end=True,
|
||||
)
|
||||
return SliceResult(
|
||||
text="\n".join(lines[first - 1 : end]),
|
||||
first_line=first,
|
||||
last_line=end,
|
||||
total_lines=total_lines,
|
||||
past_end=False,
|
||||
)
|
||||
|
||||
|
||||
def cap_slice(text: str, first_line: int, cap: int) -> tuple[str, int, bool]:
|
||||
"""Apply the character budget to one slice AT A LINE boundary.
|
||||
|
||||
Returns ``(text_to_show, last_line_shown, truncated)``. The whole
|
||||
point of cutting at a line boundary is that the caption's line range
|
||||
stays true: ``last_line_shown`` is the last line the reader actually
|
||||
sees, so a capped block never claims lines it did not show.
|
||||
|
||||
* ``cap <= 0`` raises :class:`ValueError` — a mis-wired caller must be
|
||||
loud. The ``show_text_max_chars == 0`` kill switch is checked by
|
||||
:func:`resolve_answer_blocks`, never by reaching here;
|
||||
* ``len(text) <= cap`` → the text untouched, its own last line,
|
||||
``truncated=False``;
|
||||
* otherwise cut to the last WHOLE line that fits and append the shared
|
||||
:data:`~app.rag.retriever.TRUNCATION_MARKER` on its own line — the
|
||||
marker is not content and is not counted in ``last_line_shown`` (a
|
||||
line counts as fitting only when its terminator is inside the
|
||||
budget, or sits exactly at the budget's edge);
|
||||
* a first line LONGER than ``cap`` cuts that one line mid-way (the only
|
||||
sub-line cut, documented) and still ends with the marker.
|
||||
"""
|
||||
if cap <= 0:
|
||||
raise ValueError(f"cap_slice needs a positive cap, got {cap!r}")
|
||||
if len(text) <= cap:
|
||||
if not text:
|
||||
return text, first_line, False
|
||||
# ``split("\\n")``, NOT ``splitlines()``: a slice that ends on a
|
||||
# BLANK line legitimately ends with ``"\\n"`` (line 5 of
|
||||
# ``"a\nb\n\n"`` is empty and present), and ``splitlines()`` drops
|
||||
# that final empty element — which would report an extent one
|
||||
# line short of the text actually shown (a caption that lies, and
|
||||
# a canonical locator that no longer matches the same range
|
||||
# resolved by another entry point).
|
||||
return text, first_line + len(text.split("\n")) - 1, False
|
||||
cut = text[:cap]
|
||||
if text[cap] == "\n":
|
||||
# The budget lands exactly on a line end — every line of the cut
|
||||
# is whole, nothing to trim.
|
||||
return (
|
||||
f"{cut}\n{TRUNCATION_MARKER}",
|
||||
first_line + len(cut.splitlines()) - 1,
|
||||
True,
|
||||
)
|
||||
boundary = cut.rfind("\n")
|
||||
if boundary <= 0:
|
||||
# No whole line fits — the single documented mid-line cut.
|
||||
return f"{cut}\n{TRUNCATION_MARKER}", first_line, True
|
||||
whole = cut[:boundary]
|
||||
return (
|
||||
f"{whole}\n{TRUNCATION_MARKER}",
|
||||
first_line + len(whole.splitlines()) - 1,
|
||||
True,
|
||||
)
|
||||
|
||||
|
||||
def document_by_identity(db: Session, combined: str) -> Document | None:
|
||||
"""The indexed row at the combined ``source/path`` identity, or ``None``.
|
||||
|
||||
Split at the FIRST ``'/'`` — source names are directory basenames and
|
||||
can never contain one, so the part before is the source and everything
|
||||
after is the path (the same identity contract as
|
||||
``app.rag.agent._resolve_path``).
|
||||
|
||||
This is that contract **reimplemented as a bare lookup**, deliberately
|
||||
NOT a call into ``agent.py``: the tool branch already has
|
||||
``_resolve_path`` (with its did-you-mean suggestions), the fence
|
||||
resolver only needs yes/no, and importing the agent from here would
|
||||
cycle (``agent.py`` imports this module). A traversal string
|
||||
(``../../etc/passwd``) is simply a row that does not exist — there is
|
||||
no file access here to guard, only a ``WHERE`` clause.
|
||||
"""
|
||||
if "/" not in combined:
|
||||
return None
|
||||
source, _, path = combined.partition("/")
|
||||
return db.scalar(
|
||||
select(Document).where(Document.source == source, Document.path == path)
|
||||
)
|
||||
|
||||
|
||||
def block_for(
|
||||
db: Session,
|
||||
combined: str,
|
||||
*,
|
||||
offset: int | None = None,
|
||||
limit: int | None = None,
|
||||
cap: int,
|
||||
doc: Document | None = None,
|
||||
) -> VerbatimBlock | None:
|
||||
"""Build the one block a locator names, or ``None`` (nothing displays).
|
||||
|
||||
Resolves ``combined`` (or takes the caller's already-resolved ``doc``)
|
||||
and refuses ``None`` for: no such document (including every
|
||||
traversal-shaped identity), an ``is_image`` document (image display
|
||||
belongs to ``show_image`` — LOCKED A6), and an empty or past-the-end
|
||||
slice. Otherwise ``slice_lines`` → ``cap_slice`` → the block, with the
|
||||
canonical :attr:`~VerbatimBlock.locator` built here (``source/path``,
|
||||
plus ``#first-last`` when the request named a range **or** the cap cut
|
||||
the slice — the form task 07 matches on).
|
||||
"""
|
||||
if doc is None:
|
||||
doc = document_by_identity(db, combined)
|
||||
if doc is None or doc.is_image:
|
||||
return None
|
||||
sliced = slice_lines(doc.content, offset, limit)
|
||||
if sliced.past_end or not sliced.text:
|
||||
return None
|
||||
text, last_line, truncated = cap_slice(sliced.text, sliced.first_line, cap)
|
||||
locator = f"{doc.source}/{doc.path}"
|
||||
if offset is not None or limit is not None or truncated:
|
||||
locator = f"{locator}#{sliced.first_line}-{last_line}"
|
||||
return VerbatimBlock(
|
||||
source=doc.source,
|
||||
path=doc.path,
|
||||
title=doc.title,
|
||||
start_line=sliced.first_line,
|
||||
end_line=last_line,
|
||||
total_lines=sliced.total_lines,
|
||||
text=text,
|
||||
truncated=truncated,
|
||||
locator=locator,
|
||||
)
|
||||
|
||||
|
||||
def resolve_answer_blocks(
|
||||
db: Session,
|
||||
answer: str,
|
||||
shown: Sequence[VerbatimBlock],
|
||||
settings: Settings,
|
||||
) -> list[VerbatimBlock]:
|
||||
"""The blocks an answer displays, in DISPLAY ORDER (the done-time pass).
|
||||
|
||||
Fuses the two ways a block enters an answer — the `` ```bor `` locator
|
||||
fences the model wrote (resolved here, from the row) and the blocks the
|
||||
``show_text`` tool already showed during the turn (``shown``) — into
|
||||
the ONE list task 05 puts on the ``done`` frame and persists with the
|
||||
record.
|
||||
|
||||
**Kill switches first** (checked HERE so no caller has to remember
|
||||
them): ``[]`` when ``settings.answer_text`` is off, or
|
||||
``settings.show_text_max_blocks == 0``, or
|
||||
``settings.show_text_max_chars == 0``.
|
||||
|
||||
**Ordering rule** (task 07's DOM order and the E2E assertions depend on
|
||||
it): answer order first — one block per fence, in the order the fences
|
||||
appear — then the ``shown`` blocks the answer never located, in their
|
||||
original call order. Dedupe is by canonical locator, and the **fence
|
||||
wins** over a same-locator entry from ``shown`` because the fence
|
||||
controls placement. A fence that does not resolve (unknown path, image
|
||||
document, past-the-end range, malformed locator) is simply absent: the
|
||||
caller sees "not among the blocks", which is what task 07 renders as
|
||||
the unresolved state — the answer's own fence text is left alone, so
|
||||
nothing is silently deleted. The list is finally truncated to
|
||||
``settings.show_text_max_blocks`` — and the fence scan STOPS once that
|
||||
many blocks are in, because the same truncation makes every later
|
||||
block unreachable (the bound keeps a fence-stuffed answer to at most
|
||||
``show_text_max_blocks`` successful lookups instead of one ``SELECT``
|
||||
per fence on the request's event loop).
|
||||
"""
|
||||
if not settings.answer_text:
|
||||
return []
|
||||
if settings.show_text_max_blocks == 0 or settings.show_text_max_chars == 0:
|
||||
return []
|
||||
|
||||
cap = settings.show_text_max_chars
|
||||
blocks: list[VerbatimBlock] = []
|
||||
#: The locator strings already CLAIMED (canonical written forms) — a
|
||||
#: repeated fence never resolves twice, and an unresolvable fence
|
||||
#: still holds its slot against a same-locator ``shown`` block.
|
||||
claimed: set[str] = set()
|
||||
#: The locators of the blocks actually appended — a resolved block
|
||||
#: can carry a NARROWER locator than the locator its fence wrote (the
|
||||
#: char cap cut the slice), and the same range shown by the tool
|
||||
#: carries exactly that narrower key, so both keys dedupe.
|
||||
emitted: set[str] = set()
|
||||
|
||||
for raw in fences_in(answer):
|
||||
canonical = canonical_locator(raw)
|
||||
if canonical is None or canonical in claimed:
|
||||
continue
|
||||
# Reserve the key BEFORE resolving: an unresolvable fence must not
|
||||
# let a same-locator ``shown`` block jump into the fence's slot.
|
||||
claimed.add(canonical)
|
||||
block = _block_for_canonical(db, canonical, cap)
|
||||
if block is None or block.locator in emitted:
|
||||
continue
|
||||
claimed.add(block.locator)
|
||||
emitted.add(block.locator)
|
||||
blocks.append(block)
|
||||
if len(blocks) >= settings.show_text_max_blocks:
|
||||
# The display budget is already spent — the truncation below
|
||||
# makes every later block unreachable, so stop LOOKING too:
|
||||
# a fence-stuffed answer costs at most
|
||||
# ``show_text_max_blocks`` successful lookups instead of one
|
||||
# ``SELECT`` per fence on the request's event loop.
|
||||
break
|
||||
|
||||
for block in shown:
|
||||
if block.locator in emitted or block.locator in claimed:
|
||||
continue
|
||||
emitted.add(block.locator)
|
||||
blocks.append(block)
|
||||
|
||||
return blocks[: settings.show_text_max_blocks]
|
||||
|
||||
|
||||
def _block_for_canonical(db: Session, canonical: str, cap: int) -> VerbatimBlock | None:
|
||||
"""Resolve one ALREADY-CANONICAL locator (the output of :func:`canonical_locator`)."""
|
||||
identity, _, range_part = canonical.partition("#")
|
||||
if not range_part:
|
||||
return block_for(db, identity, cap=cap)
|
||||
start_str, _, end_str = range_part.partition("-")
|
||||
start = int(start_str)
|
||||
end = int(end_str)
|
||||
return block_for(db, identity, offset=start, limit=end - start + 1, cap=cap)
|
||||
+124
-7
@@ -182,6 +182,47 @@ class SourceRef(BaseModel):
|
||||
return data
|
||||
|
||||
|
||||
class VerbatimBlockRef(BaseModel):
|
||||
"""One SERVER-resolved verbatim block of a chat answer (phase 182).
|
||||
|
||||
The item of :attr:`ChatDoneEvent.answer_text` and of the persisted
|
||||
:class:`ChatMessage.answer_text` — the display block the
|
||||
``show_text`` tool or a `` ```bor `` locator fence asked for, with
|
||||
the exact bytes read out of ``documents.content`` by
|
||||
:func:`app.rag.verbatim.block_for`.
|
||||
|
||||
**``text`` is SERVER-supplied bytes, never model output** (LOCKED
|
||||
A4/A7): the model only ever names the location, so what the user
|
||||
reads and copies cannot differ from the indexed file by a character
|
||||
— the phase's whole reason for existing. The field carries the same
|
||||
600 000 cap as the phase-83/164 answer text (deliberately generous
|
||||
above the 12 000 default of ``BOR_SHOW_TEXT_MAX_CHARS``, so an
|
||||
operator raising that budget never hits the wire limit instead).
|
||||
|
||||
``start_line`` / ``end_line`` are **1-based and INCLUSIVE** and
|
||||
describe exactly the lines ``text`` contains AFTER any character cap
|
||||
— they are the SERVER's resolved extent, never the model's claim
|
||||
(LOCKED A2), which is what the panel's caption prints. The caps on
|
||||
``source`` / ``path`` / ``title`` are :class:`SourceRef`'s (the same
|
||||
``documents`` column lengths, phase 83 SEC-05); ``locator`` is the
|
||||
canonical matching key (phase 182 D7 — ``source/path`` or
|
||||
``source/path#12-40``) the client matches its rendered placeholder
|
||||
against.
|
||||
"""
|
||||
|
||||
source: str = Field(max_length=120)
|
||||
path: str = Field(max_length=1000)
|
||||
title: str = Field(max_length=500)
|
||||
start_line: int = Field(ge=0)
|
||||
end_line: int = Field(ge=0)
|
||||
total_lines: int = Field(ge=0)
|
||||
#: The verbatim bytes themselves (see the class docstring) — the
|
||||
#: phase-83 answer-text cap, generous above the per-block budget.
|
||||
text: str = Field(max_length=600_000)
|
||||
truncated: bool = False
|
||||
locator: str = Field(max_length=2100)
|
||||
|
||||
|
||||
class ChatThinkingEvent(BaseModel):
|
||||
"""SSE thinking event: one chunk of the model's reasoning (phase 17).
|
||||
|
||||
@@ -342,6 +383,23 @@ class ChatDoneEvent(BaseModel):
|
||||
byte-identical to pre-phase (the byte-identical-off contract,
|
||||
PLAN §13; the :attr:`answer_image` / :class:`SourceRef.image_url`
|
||||
omission precedent).
|
||||
|
||||
Phase 182 (task 05, LOCKED A1/A4/A9): ``answer_text`` — the answer's
|
||||
VERBATIM display blocks, the text sibling of :attr:`answer_image`
|
||||
with two deliberate differences. (1) It is a LIST: one answer may
|
||||
display several blocks (the ``show_text`` calls and/or the
|
||||
`` ```bor `` locator fences the answer itself wrote — D1), in
|
||||
DISPLAY order (answer order first, then the tool blocks the answer
|
||||
never located — :func:`app.rag.verbatim.resolve_answer_blocks`).
|
||||
(2) Each item carries the TEXT itself rather than a route: the bytes
|
||||
ride inline (A4 — no new bytes route, no new anonymous surface, so a
|
||||
shared page renders from its own record exactly like the phase-123
|
||||
chat image), and every byte of it comes from ``documents.content``,
|
||||
never from the model (A7). ``None`` / ``[]`` — every turn without
|
||||
blocks, every deflected turn (D9 resolves nothing on a disclaimer),
|
||||
every ``answer_text=False`` turn, and the whole pre-phase — is
|
||||
DROPPED on serialization: ABSENT, never ``null``, never ``[]``, so
|
||||
those frames stay byte-identical to pre-phase (the same contract).
|
||||
"""
|
||||
|
||||
type: str = "done"
|
||||
@@ -357,25 +415,36 @@ class ChatDoneEvent(BaseModel):
|
||||
#: ONLY on the cap-reached turn; ``False`` (every other turn)
|
||||
#: serializes WITHOUT the key — ABSENT, never ``false``.
|
||||
budget_exhausted: bool = False
|
||||
#: Phase 182 (task 05) — see the class docstring. ``None``/``[]``
|
||||
#: (every turn without verbatim blocks) serializes WITHOUT the key —
|
||||
#: ABSENT, never ``null``, never ``[]``.
|
||||
answer_text: list[VerbatimBlockRef] | None = None
|
||||
|
||||
@model_serializer(mode="wrap")
|
||||
def _serialize(self, handler: SerializerFunctionWrapHandler) -> Any:
|
||||
"""The phase-124 answer-image omission rule: ``answer_image``
|
||||
``None`` (every deflected / image-less / pre-phase turn)
|
||||
serializes WITHOUT the key — ABSENT, never ``null`` — so those
|
||||
done frames stay byte-identical to pre-phase (the
|
||||
"""The additive-key omission rules: ``answer_image`` ``None``
|
||||
(every deflected / image-less / pre-phase turn) serializes
|
||||
WITHOUT the key — ABSENT, never ``null`` — so those done frames
|
||||
stay byte-identical to pre-phase (the
|
||||
:class:`SourceRef.image_url` / phase-123 ``image`` omission
|
||||
precedents). A shown ref keeps all its fields (its own
|
||||
serializer drops a ``None`` ``image_url`` — a shown doc is an
|
||||
image doc, so in practice it always carries it). Phase 160
|
||||
(task 03): ``budget_exhausted`` ``False`` (every non-cap turn)
|
||||
serializes WITHOUT the key too — ABSENT, never ``false`` — the
|
||||
same byte-identical-off contract."""
|
||||
same byte-identical-off contract. Phase 182 (task 05):
|
||||
``answer_text`` ``None`` OR EMPTY (a turn whose answer resolved
|
||||
no block, a deflected turn, the whole ``answer_text=False``
|
||||
configuration, and the pre-phase) serializes WITHOUT the key —
|
||||
ABSENT, never ``null`` and never ``[]``, so every block-free
|
||||
frame keeps its exact pre-phase bytes."""
|
||||
data = handler(self)
|
||||
if self.answer_image is None:
|
||||
data.pop("answer_image", None)
|
||||
if not self.budget_exhausted:
|
||||
data.pop("budget_exhausted", None)
|
||||
if not self.answer_text:
|
||||
data.pop("answer_text", None)
|
||||
return data
|
||||
|
||||
|
||||
@@ -1478,6 +1547,28 @@ class ChatMessage(BaseModel):
|
||||
# exactly like ``stopped`` — the phase-50 byte-identical
|
||||
# round-trip contract covers the new key automatically.
|
||||
budget_exhausted: bool | None = None
|
||||
# Phase 182 (task 05, LOCKED A4/A9): the answer's VERBATIM display
|
||||
# blocks — the persisted twin of the done frame's ``answer_text``
|
||||
# (the ``show_text`` tool's blocks and/or the ones a `` ```bor ``
|
||||
# locator fence resolved), on the BRAIN record only. Like
|
||||
# ``answer_image`` the field EXISTS so the ``extra="forbid"``
|
||||
# boundary does NOT 422 the done-time auto-save carrying it (the
|
||||
# phase-113 lesson, applied up front), and the restore path plus the
|
||||
# anonymous shared page re-render the panels from it — the bytes are
|
||||
# part of the chat's own content, so they ride the same
|
||||
# credential-is-the-row trust model and need NO new route (A4).
|
||||
# No migration (``saved_chats.messages`` is raw JSONB, the phase-50
|
||||
# / 160 precedent). ``None``/``[]`` serializes WITHOUT the key (the
|
||||
# :meth:`_drop_image_when_absent` serializer below), so a turn with
|
||||
# no blocks — and every pre-phase record — round-trips
|
||||
# byte-identically (A9). The per-block ``text`` cap is
|
||||
# :class:`VerbatimBlockRef`'s own; the list cap matches the
|
||||
# ``tools``/``segments`` cap (50), far past the
|
||||
# ``show_text_max_blocks`` default of 5 — and the block bytes count
|
||||
# toward :data:`SAVED_CHAT_TOTAL_CHARS_BUDGET`, so the display blocks
|
||||
# never become a second, unbounded channel on the anonymous save
|
||||
# surface (the phase-83 SEC-05 invariant).
|
||||
answer_text: list[VerbatimBlockRef] | None = Field(default=None, max_length=50)
|
||||
|
||||
@model_serializer(mode="wrap")
|
||||
def _drop_image_when_absent(self, handler: SerializerFunctionWrapHandler) -> Any:
|
||||
@@ -1492,12 +1583,21 @@ class ChatMessage(BaseModel):
|
||||
record WITH one keeps it — the user bubble, the refreshed page,
|
||||
and the shared chat all render the image from the served route
|
||||
(the image is part of the chat's content, so it rides the same
|
||||
public/credential-is-the-id trust model)."""
|
||||
public/credential-is-the-id trust model).
|
||||
|
||||
Phase 182 (task 05) extends the same rule to ``answer_text``:
|
||||
``None`` OR an empty list (every record without verbatim blocks,
|
||||
and every pre-phase-182 record) serializes WITHOUT the key —
|
||||
ABSENT, never ``null``, never ``[]`` (A9), so those records
|
||||
round-trip byte-identically while a record WITH blocks keeps the
|
||||
server's bytes for the restore path and the shared page."""
|
||||
data = handler(self)
|
||||
if self.image is None:
|
||||
data.pop("image", None)
|
||||
if self.answer_image is None:
|
||||
data.pop("answer_image", None)
|
||||
if not self.answer_text:
|
||||
data.pop("answer_text", None)
|
||||
# Phase 158 (task 01, D4): the timeline is ABSENT (never
|
||||
# null) when there is no run to render — the pre-phase /
|
||||
# no-thinking-no-tools shape stays byte-identical (the
|
||||
@@ -1529,11 +1629,28 @@ def _require_within_total_budget(messages: list[ChatMessage]) -> list[ChatMessag
|
||||
(phase 164): ``len(text) + len(thinking or "")`` over every message
|
||||
must stay within :data:`SAVED_CHAT_TOTAL_CHARS_BUDGET`.
|
||||
|
||||
Phase 182 (task 05) counts the VERBATIM BLOCKS into the same sum: a
|
||||
block's ``text`` carries the phase-83 answer-text cap (600 000) so an
|
||||
operator raising ``BOR_SHOW_TEXT_MAX_CHARS`` never hits the wire limit,
|
||||
and the anonymous write surface (``POST/PUT /api/chats`` is public —
|
||||
phase 55 A1) must not gain a second, UNBOUNDED byte channel next to
|
||||
the bounded one (the SEC-05 invariant: the total is what keeps the
|
||||
audited worst case at the audited order). A legitimate turn carries at
|
||||
most ``BOR_SHOW_TEXT_MAX_BLOCKS`` × ``BOR_SHOW_TEXT_MAX_CHARS`` bytes
|
||||
(5 × 12 000 = 60 KB by default), orders of magnitude below the budget,
|
||||
so nothing the server produces is affected — the bound only rejects a
|
||||
client stuffing blocks.
|
||||
|
||||
Raises :class:`ValueError` naming the budget, which Pydantic turns
|
||||
into the ordinary 422 validation response (``loc`` = the model
|
||||
itself, ``msg`` carrying the budget wording) — the same boundary-
|
||||
only style as every other phase-83/164 cap."""
|
||||
total = sum(len(m.text) + len(m.thinking or "") for m in messages)
|
||||
total = sum(
|
||||
len(m.text)
|
||||
+ len(m.thinking or "")
|
||||
+ sum(len(b.text) for b in m.answer_text or [])
|
||||
for m in messages
|
||||
)
|
||||
if total > SAVED_CHAT_TOTAL_CHARS_BUDGET:
|
||||
raise ValueError(
|
||||
f"saved chat exceeds the {SAVED_CHAT_TOTAL_CHARS_BUDGET:,}-character budget"
|
||||
|
||||
Reference in New Issue
Block a user