Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 182_verbatim_show_text
Build and Push Containers / build-and-push-app (push) Successful in 1m46s
Details
Build and Push Containers / build-and-push-db (push) Successful in 12s
Details

Browse Source 
**Phase 182 — final verification pass (all 8 tasks already complete; this was verification + docs + one defect fix).**

- Verified: `app/rag/verbatim.py` (locator grammar, `splitlines()` slice, line-boundary cap, `resolve_answer_blocks`), `SHOW_TEXT_TOOL` + `shown_texts`, `SHOW_TEXT_CLAUSE`, `done.answer_text` (9 fields, key dropped when empty) + `ChatMessageRecord.answer_text` (≤50, 600 000-char cap counted in the 8 M budget), `verbatim.js` + 4 `{verbatim:true}` renderers, `Containerfile` minify step, `.env.example` 3 vars.
- `uv run python -c "…answer_text, show_text_max_chars, show_text_max_blocks"` → `True 12000 5`; each negative refuses startup naming its field; `0` accepted as inert ✅
- `bash .agents/validate.sh` → **validation OK**: 4320 unit+integration tests green, `TOTAL 6849 17 → 99%` (gate app/ >90% OK), `ruff check .` "All checks passed!", `pyright` 0 errors (re-run after my edits, 08:28).
- E2E, each in isolation `--no-cov`: `test_verbatim_show_text.py` 13 passed (panel bytes == `documents.content`, server caption, clipboard == server string, `/document.html?source=…&path=…`, honest unresolved state, reload+share, `show_image` regression, flag-off app) ✅; green: `test_answer_images` `test_harness_aligned_tools` `test_agent_document_tools` `test_markdown_tables` `test_markdown_inline_links_images` `test_svg_document_preview` `test_smoke` `test_document_viewer` `test_markdown_blockquotes` `test_chat_persistence` `test_thinking_timeline` `test_chat_rag` `test_settings_mobile_value_width`.
- **Defect found & fixed (pre-existing, not phase 182):** `tests/e2e/test_admin_settings_panel.py::test_admin_sees_groups_and_source_badges` failed — its whole-page `assert "reese" not in page.content()` tripped on the shell's onboarding chips, which follow the newest saved chats in the one shared E2E DB (a leftover `reeselink.json` question from a tool suite). Scoped the strict sweep to `.settings-shell` and kept a whole-page `reese:reese` credential needle → 12 passed with the leftovers still present.
- Docs: README fixed 2 inconsistencies — Quick Start omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the app fail-loud refuses boot) and the broken anchor `#chat--` → `#chat-`; full re-read of README (1816 lines) + AGENTS confirmed the phase-182 feature/settings/log/UI-contract text matches the code, and every `.env.example` var + every `Settings` field is documented.
- Notable: kept `resolve_answer_blocks` synchronous on the SSE terminal path (bounded indexed SELECTs, matching the existing `query_log` precedent and the phase's documented fence-lookup bound); no live-infra changes made.
- Next pending phase: none — `todo/` holds only 182 (its 8 task files are in `complete/`; the harness commits + moves `00_phase.md`).
This commit is contained in:
ducoterra
2026-10-15 08:33:12 -04:00
parent 814a400e6d
commit b34b8ca672
104 changed files with 10852 additions and 203 deletions
Download Patch File Download Diff File Expand all files Collapse all files
app/rag/agent.py
+292 -13
View File
@@ -211,6 +211,29 @@ task 04):
— its ``"No matches for …"`` line is a (counted) result. A rejected
call still consumes a *round* in the loop, so a pathological stream
that keeps emitting rejected calls is bounded by the cap (point 4).
Display tools (phase 124 ``show_image``, phase 182 ``show_text`` —
both composed by :func:`agent_tools`, each behind its own flag): they
display content FOR THE USER. ``show_image`` never adds context; the
5th tool ``show_text`` (phase 182, task 03) returns the resolved
slice back to the model ONLY as the truth of what was displayed
(LOCKED A3 — with a do-NOT-retype instruction: the model describes
the block, the user copies the server's bytes), and its block rides
``holder.shown_texts``, never ``read_docs``. The LOCKED core contract
A1–A3/A6: **the model names a location, the server shows the bytes**
— the handler resolves the slice through
:func:`app.rag.verbatim.block_for` (same ``(source, path)`` identity,
same 1-based ``splitlines()`` indexing as ``read``/``grep``, same
char-budget-cut-at-a-line-boundary), the DISPLAYED EXTENT is the
server's (an overshooting ``limit`` clamps into the result's ``lines:``
header, an ``offset`` past the end is the :data:`OFFSET_PAST_END`
teaching naming the true line count, the char cap shrinks the extent
— the same strings ``read`` uses, one copy each), images belong to
``show_image`` (a show of an ``is_image`` row is the
:data:`SHOW_TEXT_NOT_A_TEXT_DOC` refusal, the exact mirror of
:data:`NOT_AN_IMAGE_DOC` — never a silent conversion), and the inline
``` ```bor ``` locator-fence form of the same contract (phase 182 D2)
is resolved at the API layer (task 05), NOT here.
4. Every call the model emits is appended back to the message history as
the assistant tool-call message + the tool result (refusals included),
consumes one round, and the model is called again. Every tool result
@@ -317,6 +340,7 @@ from app.rag.llm import (
from app.rag.retriever import TRUNCATION_MARKER, _fold
from app.rag.scaffolding import ScaffoldingFilter
from app.rag.source_removal import resolve_source_name
from app.rag.verbatim import VerbatimBlock, block_for
logger = logging.getLogger("app.agent")
@@ -535,20 +559,104 @@ SHOW_IMAGE_TOOL: dict[str, Any] = {
}
def agent_tools(settings: Settings) -> list[dict[str, Any]]:
"""The grounded turn's tool offer list (phase 124, task 02).
#: The 5th agent tool (phase 182, task 03; LOCKED A1–A3/A6) — offered
#: ONLY when ``settings.answer_text`` (``BOR_ANSWER_TEXT``, default ON):
#: :func:`agent_tools` composes the offer list (the three base tools stay
#: :data:`AGENT_TOOLS`, byte-identical). The description IS the behavior
#: contract (the phase-124 discipline): the model NAMES a location, the
#: SERVER shows the bytes verbatim — the user copies what is displayed,
#: so the model must NOT retype it. Same OpenAI function shape as the
#: other tools: one required string ``path`` (the combined
#: ``source/path`` identity — the bare-path-will-not-resolve teaching
#: mirrored from ``read``) plus the OPTIONAL ``offset``/``limit`` line
#: range (the ``read`` wording reused, so every line-number surface in
#: the app teaches one grammar).
SHOW_TEXT_TOOL: dict[str, Any] = {
"type": "function",
"function": {
"name": "show_text",
"description": (
"Display part of a document to the user EXACTLY as stored "
"— the user sees the server's bytes and can copy them. "
"Call it when the user wants to see, copy or run a file (a "
"quadlet, a config, a script): they copy from what is "
"displayed, so do NOT retype the content yourself after "
"calling it — describe it instead. `path` is the combined "
"`source/path` string exactly as shown in `ls`. `offset` "
"(1-based first line) and `limit` (line count) are "
"optional — omit them to display the whole document; use "
"them when `grep`/`read` told you which lines matter. The "
"displayed line numbers come from the server, not from "
"your arguments: an overshooting `limit` is clamped and an "
"`offset` past the end is refused with the true line "
"count. Only text documents — for an image use "
"`show_image`. Call one tool at a time — wait for this "
"result before your next call."
),
"parameters": {
"type": "object",
"properties": {
"path": {
"type": "string",
"description": (
"The document to display, as the combined "
"`source/path` string exactly as shown in the "
"`ls` output (e.g. "
"'homelab/active/container_caddy/caddy.md'). "
"A bare document path (without the source "
"name) will not resolve. Only text documents "
"— for an image use `show_image`."
),
},
"offset": {
"type": "integer",
"description": (
"The first line to display, 1-based. Omit it "
"to display from the beginning of the "
"document. Use it to display a specific part "
"of a large document — grep reports the line "
"numbers of what you found."
),
},
"limit": {
"type": "integer",
"description": (
"The maximum number of lines to display, "
"starting at offset. Omit it to display as "
"many lines as fit under the display cap. Use "
"it to show the part the user asked about — "
"you are showing a part, not the document."
),
},
},
"required": ["path"],
},
},
}
The three base tools (:data:`AGENT_TOOLS`, byte-identical) plus
def agent_tools(settings: Settings) -> list[dict[str, Any]]:
"""The grounded turn's tool offer list (phase 124, task 02; the 5th
tool, phase 182, task 03).
The three base tools (:data:`AGENT_TOOLS`, byte-identical) plus the
TWO display tools, each gated by its own flag:
:data:`SHOW_IMAGE_TOOL` when ``settings.answer_images``
(``BOR_ANSWER_IMAGES``) is true — otherwise exactly the three base
tools (the pre-phase-124 offer, byte-identical: no 4th tool in the
wire ``tools`` array). A FRESH list on every call — callers never
mutate the result or the constants (``run_agent`` passes it straight
to the LLM request).
(``BOR_ANSWER_IMAGES``) and :data:`SHOW_TEXT_TOOL` when
``settings.answer_text`` (``BOR_ANSWER_TEXT``) — the offer order is
``ls``, ``read``, ``grep``, ``show_image``, ``show_text``. With both
flags off: exactly the three base tools (the pre-phase-124 offer,
byte-identical: no display tool in the wire ``tools`` array); with
only ``answer_text`` off it is the pre-phase-182 offer
byte-identical. A FRESH list on every call — callers never mutate
the result or the constants (``run_agent`` passes it straight to the
LLM request).
"""
tools: list[dict[str, Any]] = list(AGENT_TOOLS)
if settings.answer_images:
tools.append(SHOW_IMAGE_TOOL)
if settings.answer_text:
tools.append(SHOW_TEXT_TOOL)
return tools
#: Tool refusal texts (phase 37): rejected calls count in nothing
@@ -587,6 +695,56 @@ ALREADY_SHOWING = (
"Answer from what you have."
)
#: The ``show_text`` refusals (phase 182, task 03): the same contract as
#: every other refusal — counts in nothing, consumes a round, changes no
#: context. ``MISSING_SHOW_TEXT_ARGS`` mirrors ``MISSING_SHOW_ARGS``.
#: ``SHOW_TEXT_NOT_A_TEXT_DOC``: one ``{arg}`` field, the argument echoed
#: as passed — the row is a standalone IMAGE; image display belongs to
#: ``show_image`` (LOCKED A6 — the exact mirror of
#: :data:`NOT_AN_IMAGE_DOC`, never a silent conversion). The range
#: problems REUSE :data:`BAD_RANGE_ARGS` and :data:`OFFSET_PAST_END`
#: verbatim — the same teaching, the same echoed values, one copy of
#: each string in the codebase.
MISSING_SHOW_TEXT_ARGS = "show_text requires a string argument 'path'."
SHOW_TEXT_NOT_A_TEXT_DOC = (
"'{arg}' is a standalone image document — use show_image to display "
"images."
)
#: One ``{max}`` field (``settings.show_text_max_blocks``): the
#: per-answer block budget, enforced SERVER-side BEFORE the resolve work
#: (the model is taught the budget in the description; the handler is
#: the actual gate). ``0`` therefore disables the tool's effect while
#: keeping it offered (the house kill-switch pattern).
SHOW_TEXT_BLOCK_CAP = (
"{max} text blocks are already displayed this answer — describe them "
"instead of showing more."
)
#: One ``{locator}`` field — the canonical ``source/path[#first-last]``
#: the model just asked for: the dedupe mirror of
#: :data:`ALREADY_SHOWING`, but per BLOCK under the block budget (the
#: budget allows SEVERAL different blocks; the same block twice teaches).
SHOW_TEXT_ALREADY_SHOWN = (
"{locator} is already on screen this answer — say so instead of "
"showing it again."
)
#: The ``show_text`` truncation notice (phase 182, task 03) — appended as
#: the result's LAST line when the per-block char budget
#: (``settings.show_text_max_chars``) cut the displayed slice (the block
#: text already ends with the shared :data:`TRUNCATION_MARKER` on its own
#: line, and the ``lines:`` extent already shrank to the last line
#: actually displayed). In the :data:`READ_TRUNCATION_NOTICE` spirit it
#: names the truth the model must not paper over — the rest is NOT
#: displayed — and the way the user gets it (open the document; NO
#: ``offset``/``limit`` re-show loop is implied, only the displayed
#: slice is what the user copies). Fields: ``{source}``/``{path}``, the
#: RESOLVED identity, plain (no ``!r``).
SHOWN_TEXT_TRUNCATION_NOTICE = (
"TRUNCATED — the rest of {source}/{path} is NOT displayed. The user "
"can open the document to read it in full; describe the displayed "
"lines instead of showing it again."
)
#: The read-truncation notice (phase 95, task 01; REWRITTEN phase 131,
#: task 01) — appended, after the shared :data:`TRUNCATION_MARKER`, to
#: a ``read`` result whose document is LONGER than
@@ -1642,6 +1800,17 @@ class AgentHolder:
nothing). Read by ``app.api.chat`` after the stream for the done
frame's ``answer_image`` field (task 04) — the ``read_docs``
precedent (an API-layer read-after-stream signal).
``shown_texts`` (phase 182, task 03): the verbatim blocks
``show_text`` displayed, in CALL order, capped by
``settings.show_text_max_blocks`` (a call over the budget is refused
with :data:`SHOW_TEXT_BLOCK_CAP` and a duplicate of a shown locator
with :data:`SHOW_TEXT_ALREADY_SHOWN` — both append nothing). Each
entry is the RESOLVED
:class:`app.rag.verbatim.VerbatimBlock` — the server's bytes plus
the server's extent (the canonical ``locator`` is the dedupe key and
the client's matching key, D7). Read by ``app.api.chat`` after the
stream for the done frame's ``answer_text`` array (task 05) — the
``shown_images``/``read_docs`` read-after-stream precedent.
``read_coverage`` (phase 141, task 01): one ``(source, path,
first_line, last_line)`` record per covered span — INCLUSIVE line
numbers over the document's ``content.splitlines()``, in read
@@ -1676,6 +1845,7 @@ class AgentHolder:
read_truncations: list[tuple[str, int, int]] = field(default_factory=list)
shown_images: list[Document] = field(default_factory=list)
read_coverage: list[tuple[str, str, int, int]] = field(default_factory=list)
shown_texts: list[VerbatimBlock] = field(default_factory=list)
def _execute_tool(
@@ -2065,6 +2235,113 @@ def _execute_tool(
f"{IMAGE_DOC_MARKER}\n"
f"{doc.content}"
)
if call.name == "show_text":
# Phase 182 (task 03; LOCKED A1–A3/A6): the 5th agent tool — the
# model NAMES a location, the SERVER shows the bytes verbatim.
# Offered only when ``settings.answer_text`` (``agent_tools``);
# the handler enforces the contract server-side. Every refusal
# below appends nothing, increments nothing and consumes a
# round, like every other refusal.
raw_path = call.arguments.get("path")
arg = raw_path.strip() if isinstance(raw_path, str) else ""
if not arg:
return MISSING_SHOW_TEXT_ARGS
if not settings.answer_text:
# Defensive — the tool is NOT OFFERED while the flag is off,
# so from the model's side this is a tool that does not
# exist: the same refusal an un-offered name gets (the
# tools-off posture — "Unknown tool."), counts in nothing.
return UNKNOWN_TOOL
doc, _source, _path = _resolve_path(db, arg)
if doc is None:
# The same phase-72 "did you mean …?" teaching as the read
# branch (a refusal — not counted, no context, nothing
# displayed).
return _no_document_refusal(db, arg)
if doc.is_image:
# LOCKED A6: an image row is refused, NEVER silently
# converted — image display is ``show_image``'s job (the
# mirror of the show_image branch's NOT_AN_IMAGE_DOC).
return SHOW_TEXT_NOT_A_TEXT_DOC.format(arg=arg)
# The OPTIONAL line range — the SAME validator, the SAME echoed
# BAD_RANGE_ARGS teaching as ``read`` (one copy of each string);
# ``offset`` without ``limit`` displays to the end, ``limit``
# without ``offset`` counts from line 1 — ``slice_lines`` applies
# both one-sided semantics with ``read``'s exact indexing.
ok, range_error, offset, limit = _read_range_args(
call.arguments.get("offset"), call.arguments.get("limit")
)
if not ok:
return range_error
# The per-answer block budget — checked BEFORE the resolve work
# (and, with a 0 char budget, before ``cap_slice`` could raise):
# either zeroed kill switch refuses like a spent budget, so
# ``0`` disables the tool's EFFECT while it stays offered (D8).
block_budget = (
settings.show_text_max_blocks
if settings.show_text_max_chars > 0
else 0
)
if len(holder.shown_texts) >= block_budget:
return SHOW_TEXT_BLOCK_CAP.format(max=block_budget)
block = block_for(
db,
arg,
offset=offset,
limit=limit,
cap=settings.show_text_max_chars,
doc=doc,
)
if block is None:
# An empty document or an ``offset`` past the last line —
# the SAME OFFSET_PAST_END teaching as ``read``, the true
# line count named (the same ``len(content.splitlines())``
# expression ``read`` uses). Counts in nothing; a corrected
# retry still proceeds.
return OFFSET_PAST_END.format(
source=doc.source,
path=doc.path,
total_lines=len(doc.content.splitlines()),
offset=offset if offset is not None else 1,
)
if any(shown.locator == block.locator for shown in holder.shown_texts):
# The canonical-locator dedupe (D7): the identical block is
# already on screen — say so instead of painting it twice.
return SHOW_TEXT_ALREADY_SHOWN.format(locator=block.locator)
# Success: the block rides the holder for the done frame
# (task 05) and the call counts (executed-only, the house rule).
holder.shown_texts.append(block)
holder.tool_calls += 1
# The result contract (D5 — the ``read``/``show_image`` header
# contract the E2E mock's ``_READ_RESULT_PREFIX`` parses): a NEW
# first line (absent from ``read`` — the model must know the
# show registered AND must not retype), then the exact document
# headers ``read`` established (date line, phase-145 origin
# lines), then the SERVER's resolved extent — clamped ``limit``
# and cap-shrunk end included, the caption never lies — and the
# verbatim slice itself (A21: LLM calls are free, and returning
# the text means the model's commentary can never disagree with
# the panel).
origin = origin_for_source(db, doc.source)
result = (
"Text shown to the user — these lines are displayed "
"verbatim; describe them, do NOT retype them.\n"
f"Document {doc.source}/{doc.path}:\n"
f"date: {doc.created_at:%Y-%m-%d}\n"
f"{_origin_lines(origin)}"
f"lines: {block.start_line}-{block.end_line} of "
f"{block.total_lines}\n"
f"{block.text}"
)
if block.truncated:
# The char cap cut the slice: the block already ends with
# the shared marker on its own line — append the pinned
# truth (the rest is NOT displayed; the user opens the
# document; no re-show loop implied).
result += "\n" + SHOWN_TEXT_TRUNCATION_NOTICE.format(
source=doc.source, path=doc.path
)
return result
if call.name == "grep":
raw_pattern = call.arguments.get("pattern")
pattern = raw_pattern.strip() if isinstance(raw_pattern, str) else ""
@@ -2229,11 +2506,13 @@ async def run_agent(
# Phase 45: no per-tool budgets — the tools stay offered for the
# whole turn, bounded by the round cap. ``0`` is the no-tools kill
# switch: exactly one request with ``tools=None`` (the pre-phase-37
# path). Phase 124 (task 02): the offer list is :func:`agent_tools`
# — the three base tools (byte-identical) plus ``show_image`` only
# when ``settings.answer_images`` (``BOR_ANSWER_IMAGES``) is on;
# off (the default) the offer is element-for-element the
# pre-phase-124 list.
# path). Phase 124 (task 02) / 182 (task 03): the offer list is
# :func:`agent_tools` — the three base tools (byte-identical) plus
# ``show_image`` only when ``settings.answer_images``
# (``BOR_ANSWER_IMAGES``) and ``show_text`` only when
# ``settings.answer_text`` (``BOR_ANSWER_TEXT``, default ON); both
# off: the offer is element-for-element the pre-phase-124 list,
# ``answer_text`` alone off: the pre-phase-182 list.
max_rounds = settings.agent_max_rounds
tools: list[dict[str, Any]] | None = (
agent_tools(settings) if max_rounds > 0 else None
app/rag/prompts.py
+105 -7
View File
@@ -66,6 +66,20 @@ backstop). The ``DEFLECT_MODE`` marker and everything else in the
prompt stay put — the E2E mock LLM keys on the marker's *presence*,
not the wording, so that contract is unchanged.
Display-tool clauses (phase 124 ``show_image``; phase 182 ``show_text``):
the **HIGH** prompt's ``<tools>`` section can carry two extra clauses
before its closing tag, each appended only when its own setting is on —
:data:`SHOW_IMAGE_CLAUSE` when ``BOR_ANSWER_IMAGES`` (default OFF) and
:data:`SHOW_TEXT_CLAUSE` when ``BOR_ANSWER_TEXT`` (default ON). The text
clause teaches the two things the phase-182 incident needed said: the
user copies from what is DISPLAYED (so never retype a file), and the
inline `` ```bor <source/path> [range] ``` `` locator fence whose EMPTY
body the app replaces with the server-resolved bytes. Both clauses
compose through one mechanism (section minus its final tag + the present
clauses + the tag), so a build with neither clause is byte-identical to
pre-phase and a ``show_image``-only build reproduces the phase-124 bytes
exactly; the LOW prompt carries neither.
Disclosed-answer contract (phase 112, task 03; owner decision
2026-09-14, roadmap confirmation — option (iii) of the three the
2026-09-15 interactive deflection test raised): that test found the
@@ -388,6 +402,61 @@ SHOW_IMAGE_CLAUSE: str = (
"in text either way."
)
#: The ``show_text`` contract clause (phase 182, task 04, LOCKED A1–A3/A6) —
#: appended INSIDE the ``<tools>`` section of the HIGH prompt, after the
#: existing three-tool copy (and after :data:`SHOW_IMAGE_CLAUSE` when that
#: rides too), ONLY when :func:`build_high_prompt` is called with
#: ``show_text=True`` (the ``BOR_ANSWER_TEXT`` master switch, via
#: ``app.api.chat``'s grounded plan — the only call site that passes it).
#: It exists because the incident this phase fixes is a PROMPT gap, not a
#: tool gap: nothing ever told the model that the user copies from what is
#: DISPLAYED, so it retyped files out of context and every retype differed
#: from the stored bytes by a character. What each sentence pins, in order:
#: (1) the display contract — the tool shows the document EXACTLY as stored
#: and the copy source is the display, not the prose; (2) when to call it —
#: the user wants to SEE, COPY or RUN a file (the owner's quadlet / config /
#: script shapes) rather than read content out of the answer; (3) the do-
#: NOT-retype rule with its reason (anything the model types may differ by
#: one character and the user cannot tell — the phase's reason for
#: existing); (4) the placement syntax — the inline `` ```bor <source/path>
#: [range] ``` `` locator fence with an EMPTY body (D1/D7: a fence is a
#: resolution request, the app supplies the text, and D3 keeps the model's
#: body out of the DOM); (5) the honesty-about-numbers rule — line numbers
#: come from ``grep``/``read``, and when unsure the range is OMITTED
#: because the displayed extent is the server's (A2); (6) the failure rule
#: — refuse out loud, never reconstruct from memory (the persona's radical
#: honesty, D3's posture on the model side); (7) the image pointer
#: (LOCKED A6: images belong to ``show_image``, never silently converted).
#: ``TOOLS_SECTION`` itself stays BYTE-IDENTICAL (the sha pins in
#: :mod:`tests.unit.test_prompt_lock` hold); a ``show_text=False`` build
#: (the default — every pre-phase call site) is byte-identical to the
#: pre-task output, and a ``show_image=True, show_text=False`` build
#: reproduces the phase-124 bytes EXACTLY. The ``<tools>`` marker the E2E
#: mock keys on is untouched either way — both clauses ride on its
#: presence, not its wording. The LOW/deflection prompt never carries the
#: clause: a deflected turn has no tools, and phase 182's D9 forbids
#: resolving a locator on a turn that just disclaimed the knowledge base.
SHOW_TEXT_CLAUSE: str = (
"`show_text` displays part of a document to the user EXACTLY as stored "
"— the user copies from what is displayed, not from your prose. Call "
"it whenever the user wants to SEE, COPY or RUN a file — a quadlet, a "
"config, a script, a unit file — instead of writing the content into "
"your answer yourself. After calling it, do NOT repeat the displayed "
"lines in your answer — describe them, explain them, point at them. "
"Anything you type yourself may differ from the file by a character, "
"and the user cannot tell. To put a displayed block exactly where you "
"want it in your answer, write a locator line on its own: three "
"backticks followed by `bor`, a space, the combined `source/path`, and "
"optionally the line range (` 12-40`). Leave its body EMPTY — the app "
"replaces it with the exact stored text. Never type the file's content "
"inside a `bor` block. Never guess line numbers — use the numbers "
"`grep` and `read` reported. When you are not sure which lines, OMIT "
"the range and display the whole document: the numbers you display are "
"the server's, never yours. If `show_text` refuses, say what you could "
"not display — do not reconstruct it from memory. For an image use "
"`show_image` instead."
)
def history_to_messages(
history: Sequence[HistoryTurn],
@@ -762,6 +831,7 @@ def build_high_prompt(
kb_overview: str | None = None,
folder_lines: Sequence[str] = (),
show_image: bool = False,
show_text: bool = False,
) -> str:
"""Grounded turn: locked persona (+ steering, + KB overview) + the
top-ranked documents' SUMMARY blocks + the ``<tools>`` instructions
@@ -771,7 +841,8 @@ def build_high_prompt(
seeds SUMMARIES, never full texts — full text enters the context
only through the capped ``read`` tool; phase 119, LOCKED A4: the
suggested-folder context lines, below; phase 124, task 03, LOCKED
A5: the optional :data:`SHOW_IMAGE_CLAUSE`, below).
A5: the optional :data:`SHOW_IMAGE_CLAUSE`, below; phase 182, task 04,
LOCKED A1–A3: the optional :data:`SHOW_TEXT_CLAUSE`, below).
Section order: ``<relevance>`` → ``<knowledge_base>`` → ``<tuning>``
→ ``<documents>`` → ``<tools>``; empty steering/overview omit their
@@ -796,6 +867,22 @@ def build_high_prompt(
never carries the clause — a deflected turn has no tools and never
shows an image.
*show_text* (phase 182, task 04, LOCKED A1–A3): when true (the caller
passes the ``BOR_ANSWER_TEXT`` master switch — the same single grounded
call site), the :data:`SHOW_TEXT_CLAUSE` rides INSIDE the same
``<tools>`` section, AFTER :data:`SHOW_IMAGE_CLAUSE` when that is
present too and always before the closing tag. The two flags compose
independently through one mechanism (the built section is
``TOOLS_SECTION`` minus its final ``</tools>`` + the present clauses,
blank-line separated, + the closing tag): ``show_image`` only
reproduces the phase-124 bytes exactly, ``show_text`` only is the
three-tool copy plus the text clause, both is the image clause then
the text clause, neither is byte-identical to pre-phase. The default
``False`` keeps every pre-existing call byte-identical. The deflection
prompt NEVER carries it — a deflected turn has no tools, and phase
182's D9 forbids resolving a ``bor`` locator on a turn whose answer
just disclaimed the knowledge base.
When at least one block is present, the section leads with the
start-here :data:`SUGGEST_INTRO` line (before the first block — the
phase-15 ``_STEERING_INTRO`` / phase-31 ``_KB_INTRO`` precedent): the
@@ -861,13 +948,24 @@ def build_high_prompt(
for part in (build_kb_section(kb_overview or ""), build_steering_section(notes or [])):
if part:
prompt += "\n" + part
# Phase 124 (task 03, LOCKED A5) / phase 182 (task 04, LOCKED A1): the
# display-tool clauses ride inside <tools> — TOOLS_SECTION's text minus
# its final </tools>, the clauses offered for this build (each on its
# own line, blank-line separated, after the existing three-tool copy),
# then the closing tag. TOOLS_SECTION itself is unedited, the
# no-clause path keeps the exact pre-task concatenation, and a
# show_image-only build reproduces the phase-124 bytes exactly.
clauses: list[str] = []
if show_image:
# Phase 124 (task 03, LOCKED A5): the clause rides inside
# <tools> — TOOLS_SECTION's text minus its final </tools>, the
# clause (on its own line, after the existing three-tool copy),
# then the closing tag. TOOLS_SECTION itself is unedited and the
# show_image=False path keeps the exact pre-task concatenation.
tools_section = TOOLS_SECTION[: -len("</tools>")] + SHOW_IMAGE_CLAUSE + "\n</tools>"
clauses.append(SHOW_IMAGE_CLAUSE)
if show_text:
clauses.append(SHOW_TEXT_CLAUSE)
if clauses:
tools_section = (
TOOLS_SECTION[: -len("</tools>")]
+ "\n\n".join(clauses)
+ "\n</tools>"
)
else:
tools_section = TOOLS_SECTION
return prompt + "\n<documents>\n" + body + "\n</documents>\n" + tools_section
app/rag/verbatim.py
+520
View File
@@ -0,0 +1,520 @@
"""Phase 182 (task 02): the verbatim locator grammar, the slicer, the resolver.
The contract in one line: **the model names a location, this module supplies
the bytes** (PLAN §6's tool-surface extension, phase 182 D1–D7). An answer
never carries a model-retyped copy of a source file; it carries a ``bor``
locator fence (````` ```bor homelab/runbooks/nginx.md#12-40 `````), and the
bytes shown to the human are read out of the indexed ``documents.content``
row by :func:`block_for` / :func:`resolve_answer_blocks`. Because the
displayed text is never generated, it cannot be paraphrased, "corrected",
or hallucinated — the phase's whole point — and the same resolver serves
both entry points (the ``show_text`` tool's own block, task 03, and the
``done``-frame pass over the finished answer, task 05).
The module is deliberately split in two halves:
* the **pure** half — :func:`canonical_locator`, :func:`fences_in`,
:func:`slice_lines`, :func:`cap_slice` — no DB, no settings, no I/O;
* the **resolver** half — :func:`document_by_identity`, :func:`block_for`,
:func:`resolve_answer_blocks` — one bare ``SELECT`` per locator and the
kill-switch gate.
**Import rule (one-way, checkable with
``grep -n "import" app/rag/verbatim.py``):** this module must NEVER import
the agent module (:mod:`app.rag.agent`) — ``agent.py`` already depends on
the pure half of this module for the ``show_text`` tool branch, so a
reverse dependency cycles. Allowed here: ``sqlalchemy``,
:class:`app.models.Document`,
:data:`app.rag.retriever.TRUNCATION_MARKER`, :class:`app.config.Settings`,
stdlib.
Nothing calls this module yet (tasks 03 and 05 wire it in), so landing it
changes no behavior anywhere.
"""
from __future__ import annotations
import re
from collections.abc import Sequence
from dataclasses import dataclass
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.config import Settings
from app.models import Document
from app.rag.retriever import TRUNCATION_MARKER
__all__ = [
"LOCATOR_LANG",
"VerbatimBlock",
"SliceResult",
"block_for",
"cap_slice",
"canonical_locator",
"document_by_identity",
"fences_in",
"resolve_answer_blocks",
"slice_lines",
]
#: The fence info-string token that marks a locator fence (D2). The
#: markdown renderer (task 06) and the copy-path builder (task 07) key on
#: this SAME constant — one grammar, one spelling, no drifting copy.
LOCATOR_LANG = "bor"
#: The range separators a locator may use between the identity and the
#: line range: a space, a colon, a bare ``#``, or ``#L`` (D7 — the grammar
#: is TOLERANT on input because the string is model-written).
_SEP = r"[ \t:#]+"
#: A line range: ``12-40``, ``12–40`` (en dash), ``L12-L40``, or a bare
#: ``12`` (one line). An optional ``L`` prefixes either bound; whitespace
#: around the dash is tolerated. Anchored at the end of the string by the
#: caller, so a range-shaped tail followed by more text is junk, not a
#: range.
_RANGE = r"(?P<range>L?[0-9]+(?:[ \t]*[-\u2013][ \t]*L?[0-9]+)?)"
#: The trailing range of a locator string (see :data:`_SEP` /
#: :data:`_RANGE`). Case-insensitive only for the ``L`` prefix.
_TRAILING_RANGE_RE = re.compile(f"{_SEP}{_RANGE}$", re.IGNORECASE)
#: The opening line of a ``bor`` locator fence: the language token,
#: whitespace, then a NON-EMPTY remainder (the locator). A bare
#: `` ```bor `` carries no locator, so it never matches and stays an
#: ordinary code block.
_FENCE_OPEN_RE = re.compile(
rf"^```{re.escape(LOCATOR_LANG)}[ \t]+(?P<locator>.*[^ \t])[ \t]*$"
)
#: The two characters a bound may use as its line-number prefix.
_LINE_PREFIX = "Ll"
@dataclass(frozen=True)
class VerbatimBlock:
"""One displayed verbatim block — the bytes plus the truth about them.
``start_line`` / ``end_line`` are **1-based and INCLUSIVE** and describe
EXACTLY the lines :attr:`text` contains *after* any character cap — a
capped block's ``end_line`` is the last line actually shown, never the
line the locator asked for (the caption must not claim lines nobody
sees). ``total_lines`` is the whole document's line count, so the
caption can say ``lines 12-40 of 512``. :attr:`text` never carries a
trailing newline (lines are joined with ``"\\n"``), and
:attr:`truncated` says whether :data:`~app.rag.retriever.TRUNCATION_MARKER`
was appended on its own last line.
:attr:`locator` is the CANONICAL form (D7) built by the factory
(:func:`block_for`), never passed in by a caller — it is the dedupe key
of :func:`resolve_answer_blocks` and the matching key the copy-path
builder uses (task 07).
"""
source: str
path: str
title: str
start_line: int
end_line: int
total_lines: int
text: str
truncated: bool = False
locator: str = ""
@dataclass(frozen=True)
class SliceResult:
"""The outcome of one line-range slice (see :func:`slice_lines`).
``first_line`` / ``last_line`` are 1-based and inclusive; ``past_end``
is the caller's refusal signal and always travels with the true
``total_lines`` (the teaching refusal names the real line count).
"""
text: str
first_line: int
last_line: int
total_lines: int
past_end: bool
def canonical_locator(raw: str) -> str | None:
"""Normalize one model-written locator string, or refuse it (D7).
Accepted forms — ``source/path`` (the ``read``/``grep`` identity
contract, split at the FIRST ``'/'``) followed by an optional separator
and line range:
========================== ==========================
input canonical output
========================== ==========================
``homelab/a.md`` ``homelab/a.md``
``homelab/a.md 12-40`` ``homelab/a.md#12-40``
``homelab/a.md:12-40`` ``homelab/a.md#12-40``
``homelab/a.md#12-40`` ``homelab/a.md#12-40``
``homelab/a.md#L12-L40`` ``homelab/a.md#12-40``
``homelab/a.md 12\u201340`` ``homelab/a.md#12-40``
``homelab/a.md 12`` ``homelab/a.md#12-12``
========================== ==========================
The canonical form is always ``source/path`` + ``#start-end`` with
plain digits — never the en dash, never the ``L``, so the fence, the
tool block and the UI share ONE string (D7).
Returns ``None`` (NOT a locator) for: empty/whitespace input, a value
with no ``'/'`` in the path part, a non-numeric or non-positive bound,
``start > end``, trailing junk after the range, or a leading ``bor``
token the caller left in. No DB access, no case folding (paths are
case-sensitive row values). A locator carrying a range also refuses an
identity containing whitespace: the fence's info string is ONE
whitespace-free identity plus the range, so a path that contains a
space is expressed as a whole-document locator (no range) or not at
all — the documented limit of the tolerant grammar.
"""
text = raw.strip()
if not text:
return None
# A caller that pasted the fence's language token in has already lost
# the identity — refuse rather than guess which token was the path.
if text.split(" ", 1)[0] == LOCATOR_LANG:
return None
match = _TRAILING_RANGE_RE.search(text)
if match is None:
identity, range_part = text, None
else:
# ``match.start()`` is where the separator begins — the identity
# is everything before it (a multi-character separator, e.g. two
# spaces, is consumed whole).
identity, range_part = text[: match.start()], match.group("range")
# The identity is one whitespace-free ``source/path`` token, and it
# never ends in a dangling separator (``a.md#`` with no range is a
# typo, not a whole document).
if not identity or any(c.isspace() for c in identity):
return None
if identity.endswith(("#", ":", "-")):
return None
if "/" not in identity:
return None
if range_part is None:
return identity
bounds = [b.strip().lstrip(_LINE_PREFIX) for b in re.split(r"[-\u2013]", range_part)]
# Defensive: :data:`_RANGE` already guarantees digit bounds, so this
# only fires if the grammar above is ever loosened — it keeps
# ``canonical_locator`` total (never an ``int()`` ValueError to a
# caller that hands it model-written text).
if any(not b.isdigit() for b in bounds):
return None
start = int(bounds[0])
end = int(bounds[1]) if len(bounds) == 2 else start
if start < 1 or end < 1 or start > end:
return None
return f"{identity}#{start}-{end}"
def fences_in(answer: str) -> list[str]:
"""Every ``bor`` locator fence's RAW locator string, in answer order.
Rules (the same shape ``markdown.js`` applies, task 06):
* the opening line is a fence whose info string is exactly
`` ```bor `` + whitespace + a NON-empty remainder — a bare
`` ```bor `` with no path is an ordinary code block, not a locator,
and is not swallowed;
* the fence closes at the next line that is nothing but backticks;
* an **unclosed** fence contributes nothing (defensive: a half-streamed
answer must not resolve a locator the model never finished) and ends
the scan, since its body runs to the end of the answer;
* the body is read and DISCARDED — this module never looks at what the
model typed inside the fence (the bytes come from the row, D1);
* nested `` ``` `` inside a body is NOT supported: the fence ends at
the first closer (documented limitation, same rule as the renderer).
Duplicates collapse keeping the FIRST occurrence.
"""
lines = answer.splitlines()
locators: list[str] = []
index = 0
while index < len(lines):
match = _FENCE_OPEN_RE.match(lines[index].strip())
if match is None:
index += 1
continue
closer = _find_fence_close(lines, index + 1)
if closer is None:
# Unclosed: the body runs to the end of the answer, so there
# is nothing further to scan and this fence yields nothing.
break
locator = match.group("locator").strip()
if locator and locator not in locators:
locators.append(locator)
index = closer + 1
return locators
def _find_fence_close(lines: Sequence[str], start: int) -> int | None:
"""Index of the fence's closing line at or after ``start``, ``None`` if unclosed."""
for offset in range(start, len(lines)):
stripped = lines[offset].strip()
if stripped.startswith("```") and not stripped.strip("`"):
return offset
return None
def slice_lines(
content: str, offset: int | None = None, limit: int | None = None
) -> SliceResult:
"""Cut ``content`` to a 1-based line range with ``read``'s exact semantics.
Same indexing (``content.splitlines()``) and the same range arithmetic
as the ``read`` tool (``app/rag/agent.py``) and ``grep``, so a line
number the model got from ``grep`` names the same line here — the one
shared coordinate system the whole phase depends on:
* ``offset`` omitted → 1; ``limit`` omitted → read to the end;
* ``end = min(offset + limit - 1, total_lines)`` — an overshooting
``limit`` CLAMPS (``last_line`` is the document's last line);
* ``offset`` past the end, an empty document, or a ``limit`` below 1 →
``past_end=True`` with empty text and the TRUE ``total_lines`` (the
caller builds the teaching refusal).
Never raises for a range problem: the refusal is the caller's, the
total always travels with the result.
"""
lines = content.splitlines()
total_lines = len(lines)
first = 1 if offset is None else offset
end = total_lines if limit is None else min(first + limit - 1, total_lines)
if total_lines == 0 or first > total_lines or end < first:
return SliceResult(
text="",
first_line=first,
last_line=0,
total_lines=total_lines,
past_end=True,
)
return SliceResult(
text="\n".join(lines[first - 1 : end]),
first_line=first,
last_line=end,
total_lines=total_lines,
past_end=False,
)
def cap_slice(text: str, first_line: int, cap: int) -> tuple[str, int, bool]:
"""Apply the character budget to one slice AT A LINE boundary.
Returns ``(text_to_show, last_line_shown, truncated)``. The whole
point of cutting at a line boundary is that the caption's line range
stays true: ``last_line_shown`` is the last line the reader actually
sees, so a capped block never claims lines it did not show.
* ``cap <= 0`` raises :class:`ValueError` — a mis-wired caller must be
loud. The ``show_text_max_chars == 0`` kill switch is checked by
:func:`resolve_answer_blocks`, never by reaching here;
* ``len(text) <= cap`` → the text untouched, its own last line,
``truncated=False``;
* otherwise cut to the last WHOLE line that fits and append the shared
:data:`~app.rag.retriever.TRUNCATION_MARKER` on its own line — the
marker is not content and is not counted in ``last_line_shown`` (a
line counts as fitting only when its terminator is inside the
budget, or sits exactly at the budget's edge);
* a first line LONGER than ``cap`` cuts that one line mid-way (the only
sub-line cut, documented) and still ends with the marker.
"""
if cap <= 0:
raise ValueError(f"cap_slice needs a positive cap, got {cap!r}")
if len(text) <= cap:
if not text:
return text, first_line, False
# ``split("\\n")``, NOT ``splitlines()``: a slice that ends on a
# BLANK line legitimately ends with ``"\\n"`` (line 5 of
# ``"a\nb\n\n"`` is empty and present), and ``splitlines()`` drops
# that final empty element — which would report an extent one
# line short of the text actually shown (a caption that lies, and
# a canonical locator that no longer matches the same range
# resolved by another entry point).
return text, first_line + len(text.split("\n")) - 1, False
cut = text[:cap]
if text[cap] == "\n":
# The budget lands exactly on a line end — every line of the cut
# is whole, nothing to trim.
return (
f"{cut}\n{TRUNCATION_MARKER}",
first_line + len(cut.splitlines()) - 1,
True,
)
boundary = cut.rfind("\n")
if boundary <= 0:
# No whole line fits — the single documented mid-line cut.
return f"{cut}\n{TRUNCATION_MARKER}", first_line, True
whole = cut[:boundary]
return (
f"{whole}\n{TRUNCATION_MARKER}",
first_line + len(whole.splitlines()) - 1,
True,
)
def document_by_identity(db: Session, combined: str) -> Document | None:
"""The indexed row at the combined ``source/path`` identity, or ``None``.
Split at the FIRST ``'/'`` — source names are directory basenames and
can never contain one, so the part before is the source and everything
after is the path (the same identity contract as
``app.rag.agent._resolve_path``).
This is that contract **reimplemented as a bare lookup**, deliberately
NOT a call into ``agent.py``: the tool branch already has
``_resolve_path`` (with its did-you-mean suggestions), the fence
resolver only needs yes/no, and importing the agent from here would
cycle (``agent.py`` imports this module). A traversal string
(``../../etc/passwd``) is simply a row that does not exist — there is
no file access here to guard, only a ``WHERE`` clause.
"""
if "/" not in combined:
return None
source, _, path = combined.partition("/")
return db.scalar(
select(Document).where(Document.source == source, Document.path == path)
)
def block_for(
db: Session,
combined: str,
*,
offset: int | None = None,
limit: int | None = None,
cap: int,
doc: Document | None = None,
) -> VerbatimBlock | None:
"""Build the one block a locator names, or ``None`` (nothing displays).
Resolves ``combined`` (or takes the caller's already-resolved ``doc``)
and refuses ``None`` for: no such document (including every
traversal-shaped identity), an ``is_image`` document (image display
belongs to ``show_image`` — LOCKED A6), and an empty or past-the-end
slice. Otherwise ``slice_lines`` → ``cap_slice`` → the block, with the
canonical :attr:`~VerbatimBlock.locator` built here (``source/path``,
plus ``#first-last`` when the request named a range **or** the cap cut
the slice — the form task 07 matches on).
"""
if doc is None:
doc = document_by_identity(db, combined)
if doc is None or doc.is_image:
return None
sliced = slice_lines(doc.content, offset, limit)
if sliced.past_end or not sliced.text:
return None
text, last_line, truncated = cap_slice(sliced.text, sliced.first_line, cap)
locator = f"{doc.source}/{doc.path}"
if offset is not None or limit is not None or truncated:
locator = f"{locator}#{sliced.first_line}-{last_line}"
return VerbatimBlock(
source=doc.source,
path=doc.path,
title=doc.title,
start_line=sliced.first_line,
end_line=last_line,
total_lines=sliced.total_lines,
text=text,
truncated=truncated,
locator=locator,
)
def resolve_answer_blocks(
db: Session,
answer: str,
shown: Sequence[VerbatimBlock],
settings: Settings,
) -> list[VerbatimBlock]:
"""The blocks an answer displays, in DISPLAY ORDER (the done-time pass).
Fuses the two ways a block enters an answer — the `` ```bor `` locator
fences the model wrote (resolved here, from the row) and the blocks the
``show_text`` tool already showed during the turn (``shown``) — into
the ONE list task 05 puts on the ``done`` frame and persists with the
record.
**Kill switches first** (checked HERE so no caller has to remember
them): ``[]`` when ``settings.answer_text`` is off, or
``settings.show_text_max_blocks == 0``, or
``settings.show_text_max_chars == 0``.
**Ordering rule** (task 07's DOM order and the E2E assertions depend on
it): answer order first — one block per fence, in the order the fences
appear — then the ``shown`` blocks the answer never located, in their
original call order. Dedupe is by canonical locator, and the **fence
wins** over a same-locator entry from ``shown`` because the fence
controls placement. A fence that does not resolve (unknown path, image
document, past-the-end range, malformed locator) is simply absent: the
caller sees "not among the blocks", which is what task 07 renders as
the unresolved state — the answer's own fence text is left alone, so
nothing is silently deleted. The list is finally truncated to
``settings.show_text_max_blocks`` — and the fence scan STOPS once that
many blocks are in, because the same truncation makes every later
block unreachable (the bound keeps a fence-stuffed answer to at most
``show_text_max_blocks`` successful lookups instead of one ``SELECT``
per fence on the request's event loop).
"""
if not settings.answer_text:
return []
if settings.show_text_max_blocks == 0 or settings.show_text_max_chars == 0:
return []
cap = settings.show_text_max_chars
blocks: list[VerbatimBlock] = []
#: The locator strings already CLAIMED (canonical written forms) — a
#: repeated fence never resolves twice, and an unresolvable fence
#: still holds its slot against a same-locator ``shown`` block.
claimed: set[str] = set()
#: The locators of the blocks actually appended — a resolved block
#: can carry a NARROWER locator than the locator its fence wrote (the
#: char cap cut the slice), and the same range shown by the tool
#: carries exactly that narrower key, so both keys dedupe.
emitted: set[str] = set()
for raw in fences_in(answer):
canonical = canonical_locator(raw)
if canonical is None or canonical in claimed:
continue
# Reserve the key BEFORE resolving: an unresolvable fence must not
# let a same-locator ``shown`` block jump into the fence's slot.
claimed.add(canonical)
block = _block_for_canonical(db, canonical, cap)
if block is None or block.locator in emitted:
continue
claimed.add(block.locator)
emitted.add(block.locator)
blocks.append(block)
if len(blocks) >= settings.show_text_max_blocks:
# The display budget is already spent — the truncation below
# makes every later block unreachable, so stop LOOKING too:
# a fence-stuffed answer costs at most
# ``show_text_max_blocks`` successful lookups instead of one
# ``SELECT`` per fence on the request's event loop.
break
for block in shown:
if block.locator in emitted or block.locator in claimed:
continue
emitted.add(block.locator)
blocks.append(block)
return blocks[: settings.show_text_max_blocks]
def _block_for_canonical(db: Session, canonical: str, cap: int) -> VerbatimBlock | None:
"""Resolve one ALREADY-CANONICAL locator (the output of :func:`canonical_locator`)."""
identity, _, range_part = canonical.partition("#")
if not range_part:
return block_for(db, identity, cap=cap)
start_str, _, end_str = range_part.partition("-")
start = int(start_str)
end = int(end_str)
return block_for(db, identity, offset=start, limit=end - start + 1, cap=cap)