phase: 182_verbatim_show_text
**Phase 182 — final verification pass (all 8 tasks already complete; this was verification + docs + one defect fix).**
- Verified: `app/rag/verbatim.py` (locator grammar, `splitlines()` slice, line-boundary cap, `resolve_answer_blocks`), `SHOW_TEXT_TOOL` + `shown_texts`, `SHOW_TEXT_CLAUSE`, `done.answer_text` (9 fields, key dropped when empty) + `ChatMessageRecord.answer_text` (≤50, 600 000-char cap counted in the 8 M budget), `verbatim.js` + 4 `{verbatim:true}` renderers, `Containerfile` minify step, `.env.example` 3 vars.
- `uv run python -c "…answer_text, show_text_max_chars, show_text_max_blocks"` → `True 12000 5`; each negative refuses startup naming its field; `0` accepted as inert ✅
- `bash .agents/validate.sh` → **validation OK**: 4320 unit+integration tests green, `TOTAL 6849 17 → 99%` (gate app/ >90% OK), `ruff check .` "All checks passed!", `pyright` 0 errors (re-run after my edits, 08:28).
- E2E, each in isolation `--no-cov`: `test_verbatim_show_text.py` 13 passed (panel bytes == `documents.content`, server caption, clipboard == server string, `/document.html?source=…&path=…`, honest unresolved state, reload+share, `show_image` regression, flag-off app) ✅; green: `test_answer_images` `test_harness_aligned_tools` `test_agent_document_tools` `test_markdown_tables` `test_markdown_inline_links_images` `test_svg_document_preview` `test_smoke` `test_document_viewer` `test_markdown_blockquotes` `test_chat_persistence` `test_thinking_timeline` `test_chat_rag` `test_settings_mobile_value_width`.
- **Defect found & fixed (pre-existing, not phase 182):** `tests/e2e/test_admin_settings_panel.py::test_admin_sees_groups_and_source_badges` failed — its whole-page `assert "reese" not in page.content()` tripped on the shell's onboarding chips, which follow the newest saved chats in the one shared E2E DB (a leftover `reeselink.json` question from a tool suite). Scoped the strict sweep to `.settings-shell` and kept a whole-page `reese:reese` credential needle → 12 passed with the leftovers still present.
- Docs: README fixed 2 inconsistencies — Quick Start omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the app fail-loud refuses boot) and the broken anchor `#chat--` → `#chat-`; full re-read of README (1816 lines) + AGENTS confirmed the phase-182 feature/settings/log/UI-contract text matches the code, and every `.env.example` var + every `Settings` field is documented.
- Notable: kept `resolve_answer_blocks` synchronous on the SSE terminal path (bounded indexed SELECTs, matching the existing `query_log` precedent and the phase's documented fence-lookup bound); no live-infra changes made.
- Next pending phase: none — `todo/` holds only 182 (its 8 task files are in `complete/`; the harness commits + moves `00_phase.md`).
This commit is contained in:
@@ -34,6 +34,8 @@ uv sync
|
||||
# 2. Configure (copy and edit)
|
||||
cp .env.example .env
|
||||
# Set BOR_LLM_API_KEY — your aipi key (falls back to $AIPI_KEY)
|
||||
# Set BOR_ADMIN_PASSWORD and BOR_SESSION_SECRET — both are REQUIRED:
|
||||
# the app refuses to start while either is empty (see Admin & Sign-in)
|
||||
|
||||
# 3. Start the database
|
||||
podman compose up -d db
|
||||
@@ -162,6 +164,22 @@ collapses into body text (the UA default is 0.83em/0.67em) nor out-shouts
|
||||
the answer around it, in the bubble, the thinking block and the viewer
|
||||
alike.
|
||||
|
||||
**The renderer has one more pass, and it is opt-in** (phase 182):
|
||||
`renderMarkdown(md)` is now `renderMarkdown(md, opts)`, and the ONLY truthy
|
||||
option is `{ verbatim: true }`, passed by exactly the FOUR answer renderers —
|
||||
the live per-delta write, the `done`-time re-render, the chat page's restore
|
||||
and the shared page's record renderer. Every other call site omits it, so the
|
||||
feature is inert elsewhere. With it
|
||||
on, a ` ```bor <source/path> <range> ` locator fence is
|
||||
replaced — fence line, body and closer included — by the pending verbatim
|
||||
panel, so the model's re-typed copy of a file is never painted (see
|
||||
[Verbatim text display](#verbatim-text-display-phase-182)). Pass order with
|
||||
the option on is **verbatim → fences → tables → escape+inline → blockquotes →
|
||||
paragraphs → restore**; with it off — the document viewer, the thinking runs,
|
||||
and every question bubble on both pages — the output is byte-identical
|
||||
to pre-phase, so a document that legitimately contains a ` ```bor ` fence
|
||||
still renders its own text.
|
||||
|
||||
The document's meta row (the chat modal **and** the standalone viewer at
|
||||
`/document.html`) carries a **Copy path** button: one click copies the
|
||||
document's `source/path` (the exact form the `read` tool, the chips, and
|
||||
@@ -296,7 +314,15 @@ The overlay can be switched off for the whole process with
|
||||
### Shared chat (`/shared/<token>`)
|
||||
|
||||
A read-only snapshot of any conversation, shareable via a link. Open to
|
||||
everyone — no login or token required.
|
||||
everyone — no login or token required. The page still ships no composer, no
|
||||
Save/Share/Tune/Retry and no chips-as-buttons; the ONE interactive exception is
|
||||
phase 182's answer code: a shared verbatim panel and the answer's ordinary
|
||||
code blocks carry a **Copy** button and the panel a plain
|
||||
`/document.html?source=…&path=…` link, because a page that shows a slice of a
|
||||
file has to let the reader copy it and check it. Both come from the same
|
||||
`verbatim.js` the chat page calls (so a shared turn renders identically to the
|
||||
owner's), and a guest who follows the viewer link lands on the sign-in page —
|
||||
the documents API is admin-gated.
|
||||
|
||||
### Conversation payload limits
|
||||
|
||||
@@ -311,7 +337,9 @@ saving a conversation can never be stricter than continuing it (phase
|
||||
| One message (`text`, `thinking`) | 600 000 chars | a **full-length answer at the largest answer cap** — `BOR_MAX_OUTPUT_TOKENS` at 131 072 tokens ≈ 524 300 chars — as history and as a saved record, with margin (the old 32 000 ceiling broke even at the 32 768-token default) |
|
||||
| Turns sent as `history` | 500 | the whole browser conversation (the client sends everything it holds, minus the question being asked) |
|
||||
| Messages per saved chat | 500 | the same conversation, saved |
|
||||
| **Total chars per saved chat** | **8 000 000** | ~15 full-length answers; the one bound on the anonymous save surface, which stays bounded on purpose |
|
||||
| One verbatim block (`answer_text[].text`) | 600 000 chars | the same answer-text cap (phase 182) — comfortably above `BOR_SHOW_TEXT_MAX_CHARS`, so raising that budget never hits the wire limit |
|
||||
| Verbatim blocks per saved message | 50 | the `tools` / `segments` number (phase 182) — far past the `BOR_SHOW_TEXT_MAX_BLOCKS` default of 5 |
|
||||
| **Total chars per saved chat** | **8 000 000** | ~15 full-length answers; the one bound on the anonymous save surface, which stays bounded on purpose — and it counts the verbatim blocks' bytes too (phase 182), so the display blocks cannot become a second, unbounded byte channel |
|
||||
| One question (`message`) | 4 000 chars | unchanged; the composer counts it live |
|
||||
|
||||
These are sanity ceilings, not the prompt's budget: the real gate is
|
||||
@@ -675,8 +703,10 @@ server-side tools — on **grounded** (high-relevance) turns only:
|
||||
| **`read(path, offset?, limit?)`** | Appends one indexed document's text to context by its combined `source/path`; optional `offset` (1-based first line) + `limit` (line count) read just that part of a large document; very large reads are truncated at `BOR_READ_MAX_CHARS` with a notice that teaches the workflow — grep reports line numbers, then read that part with `offset` + `limit` |
|
||||
| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case- and accent-insensitive fixed substring — accents fold: `besta` matches `BESTÅ`, phase 162); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document |
|
||||
| **`show_image(path)`** | *(only when `BOR_ANSWER_IMAGES=1`)* shows one standalone image document as part of the answer — image documents only, at most one per answer |
|
||||
| **`show_text(path, offset?, limit?)`** | *(only when `BOR_ANSWER_TEXT=1`, the default)* displays the named range of a text document to the user **verbatim** — the server shows the bytes and keeps them byte-identical; the model describes them, never retypes them; image documents are refused (use `show_image`); per-answer display budgets enforced server-side |
|
||||
|
||||
(See [Answer images](#answer-images-phase-124) for the `show_image` tool.)
|
||||
(See [Answer images](#answer-images-phase-124) for the `show_image` tool and
|
||||
[Verbatim text display](#verbatim-text-display-phase-182) for `show_text`.)
|
||||
|
||||
**The prompt teaches the search discipline (phase 162).** A summary or an
|
||||
outline is a hint, never an inventory: the system prompt tells the model to run
|
||||
@@ -734,7 +764,7 @@ instead of an error frame (phase 160, the recorded revision of the
|
||||
phase-71 cap-path terminal). The `done` frame carries an additive
|
||||
`budget_exhausted` key (ABSENT on every non-cap turn) that drives the
|
||||
**Continue researching** button on the answer's meta row — see the
|
||||
[Chat (`/`)](#chat--) section.
|
||||
[Chat (`/`)](#chat-) section.
|
||||
|
||||
---
|
||||
|
||||
@@ -971,6 +1001,142 @@ of its answer:
|
||||
|
||||
---
|
||||
|
||||
## Verbatim text display (phase 182)
|
||||
|
||||
The mirror of the answer-image trust model, for words: when the user asks to
|
||||
see part of a document — a stanza, a passage, an exact range of lines — the
|
||||
model **names the location and the server shows the bytes, byte-for-byte**.
|
||||
The model never retypes quoted text, so a displayed quote cannot drift.
|
||||
|
||||
- **The model decides.** The grounded agent is offered a display tool,
|
||||
`show_text(path, offset?, limit?)`, and calls it when the user wants to
|
||||
SEE, COPY or RUN a file — a quadlet, a config, a script, a unit file —
|
||||
or read its exact wording. Optional
|
||||
`offset` (1-based first line) and `limit` (line count); an overshooting
|
||||
`limit` is clamped, an `offset` past the last line is refused with the
|
||||
true line count.
|
||||
- **What the prompt teaches.** With the switch on, the grounded `<tools>`
|
||||
section gains one clause (after the `show_image` clause when that rides
|
||||
too) saying the two things the tool description cannot: the user copies
|
||||
from **what is displayed**, not from the model's prose — so after a
|
||||
`show_text` call the model describes the lines instead of retyping them
|
||||
— and the inline locator fence for placing a displayed block mid-answer:
|
||||
a line of three backticks + `bor` + the combined `source/path` + an
|
||||
optional `12-40` range, with an **empty body** the app replaces with the
|
||||
stored text. Line numbers come from `grep`/`read`, never from a guess —
|
||||
when unsure, the range is omitted and the whole document is displayed.
|
||||
`BOR_ANSWER_TEXT=0` removes the clause too, so the prompt never teaches
|
||||
a tool the model was not given.
|
||||
- **Server-side budgets.** At most `BOR_SHOW_TEXT_MAX_BLOCKS` blocks per
|
||||
answer (a second identical show is refused — it is already on screen);
|
||||
each block is capped at `BOR_SHOW_TEXT_MAX_CHARS`, cut at a line boundary
|
||||
with a truncation marker. Image documents are refused with a pointer to
|
||||
`show_image` — the two display tools never convert into each other.
|
||||
- **The bytes ride the `done` frame.** A turn that displayed text ends
|
||||
with one extra, additive SSE key: `done.answer_text`, the list of blocks
|
||||
in **display order** — answer order first (one per ` ```bor ` fence, the
|
||||
fence controls placement), then the `show_text` blocks the answer never
|
||||
located. Each entry is
|
||||
`{source, path, title, start_line, end_line, total_lines, text, truncated, locator}`
|
||||
where `text` is the **server's** slice of `documents.content` (never
|
||||
model output) and `start_line`/`end_line` are the server's resolved
|
||||
1-based inclusive extent — the numbers a caption shows can never be the
|
||||
model's claim. `locator` is the canonical matching key
|
||||
(`source/path`, or `source/path#12-40` for a range). There is no new
|
||||
bytes route and no eighth SSE event type: the bytes travel inline, and a
|
||||
saved or shared chat renders its panels from its own record. The same
|
||||
array persists on the saved brain record (`bor.chat.v1` `answer_text`),
|
||||
so a reload and an anonymous shared page show the same blocks.
|
||||
- **Deflected turns resolve nothing.** A ` ```bor ` fence written into a
|
||||
deflected answer leaks no KB text — the disclaimer turn resolves no
|
||||
block and its frame carries no `answer_text` key.
|
||||
- **Off = pre-phase.** `BOR_ANSWER_TEXT=0` removes the tool from every
|
||||
offer **and** its `<tools>` clause (the grounded system prompt is
|
||||
byte-identical to pre-phase); answers return to composed text (the model
|
||||
quoting from what it read). A turn with no blocks serializes **without**
|
||||
the key — absent, never `null`, never `[]` — so every block-free frame
|
||||
and every pre-phase saved record keeps its exact pre-phase bytes. The
|
||||
SERVER surface is byte-pre-phase; the CLIENT rule is flag-independent, so
|
||||
a stray ` ```bor ` fence in a flag-off answer still cannot paint the
|
||||
model's copy — with no `answer_text` to hydrate it, it settles to the same
|
||||
honest sentence (the phase E2E's App B spec).
|
||||
|
||||
- **The client never paints the model's copy.** The one shared markdown
|
||||
renderer (`frontend/assets/markdown.js`) takes a SECOND, optional
|
||||
argument — `renderMarkdown(md, { verbatim: true })` — and with it ON a
|
||||
` ```bor ` locator fence is replaced, info line, body and closing fence
|
||||
included, by a pending panel:
|
||||
`<div class="verbatim verbatim-pending" data-bor-locator="…">` holding the
|
||||
`source/path` caption and the live-region line *Retrieving the exact
|
||||
text…*. The locator is normalized client-side by the exact mirror of the
|
||||
server's `canonical_locator` (`source/path`, or `source/path#12-40` —
|
||||
digits only, so the two sides match on a string compare), the model's
|
||||
bytes appear NOWHERE in the output, and a fence the model has not finished
|
||||
typing shows that same pending state instead of raw backticks. An
|
||||
unparseable locator degrades to an ordinary code block with the model's own
|
||||
body — no placeholder, no badge. With the argument OFF — every other
|
||||
surface in the app: the document viewer, the thinking blocks, the question
|
||||
bubbles — the renderer's output is byte-identical to pre-phase, so a
|
||||
document that legitimately contains a ` ```bor ` fence still renders its
|
||||
own text. Pass order with the option on:
|
||||
**verbatim → fences → tables → escape+inline → blockquotes → paragraphs →
|
||||
restore**.
|
||||
|
||||
- **Hydration paints the panel, and the panel is the file.** The client half
|
||||
is one classic script, `frontend/assets/verbatim.js`, loaded after
|
||||
`markdown.js` on the chat page and the shared page. At `done` (and on every
|
||||
restore) it matches each placeholder's normalized `data-bor-locator`
|
||||
against the frame's `answer_text[].locator` with a plain string compare —
|
||||
`markdown.js::borLocator` stays the ONLY locator grammar in the client, so
|
||||
the two sides cannot drift — and paints:
|
||||
|
||||
- `.verbatim-caption` — `source/path · lines 12–40 of 210`, or
|
||||
`source/path · all 210 lines` when the block is the whole document: the
|
||||
numbers are the **server's** extent, so a locator whose range the model got
|
||||
wrong still shows what was actually read (a panel matched on `source/path`
|
||||
alone is A2 made visible).
|
||||
- `.verbatim-badge` — the word `VERBATIM`: these bytes came from the
|
||||
knowledge base, not from the model.
|
||||
- `.verbatim-lines` — the server's line numbers, `user-select: none`, so a
|
||||
selection or a Copy never drags digits into the clipboard. A capped block's
|
||||
marker line gets a BLANK cell, because the marker is not a line of the file.
|
||||
- `.verbatim-text` — `answer_text[].text` through `textContent`, byte for
|
||||
byte (never re-wrapped, never trimmed, never joined).
|
||||
- `.verbatim-copy` — `Copy` → `Copied` (the state is TEXT), accessible name
|
||||
`Copy N lines from <source/path>`; the clipboard receives the server string
|
||||
exactly.
|
||||
- `.verbatim-open` — the document viewer: on the chat page the same same-page
|
||||
modal as a source chip (back → the chat), on the shared page a plain
|
||||
`/document.html?source=…&path=…` link.
|
||||
- `.verbatim-more` — on a capped block, one muted line saying the rest is in
|
||||
the document, plus that link.
|
||||
|
||||
A locator the server did not resolve renders the honest sentence *I could
|
||||
not verify this against the knowledge base — open `<source/path>`* with NO
|
||||
badge, NO body and none of the model's bytes; and every terminal path (the
|
||||
`error` frame, the 300 s guard, Stop, the failed-turn funnel) settles any
|
||||
leftover placeholder, so nothing ever sits at *Retrieving the exact text…*
|
||||
once a turn is over. A `show_text` block the answer never located is
|
||||
appended in the `answer_image` band — below the answer text, above the
|
||||
source clusters.
|
||||
|
||||
- **Every settled answer code block can be copied.** `enhanceAnswerCodeBlocks`
|
||||
wraps each `<pre>` inside a settled answer bubble that is not part of a
|
||||
panel in `.code-block` and adds a `.code-block-copy` pill: **no badge and no
|
||||
caption**, because a `VERBATIM` claim belongs only to a server-resolved
|
||||
block — an answer's code is either verbatim-and-badged or plainly the
|
||||
model's, never a silently-substituted hybrid. Buttons appear when the turn
|
||||
settles, never mid-stream. The share link's copy path delegates to the same
|
||||
`copyTextWithFallback`, so the http-origin inline select-to-copy degradation
|
||||
exists once (the owner-locked share strings and behavior are unchanged).
|
||||
- **Parity, and what stays untouched.** The live turn, a reload and an
|
||||
anonymous shared page call the same three globals in the same order, so one
|
||||
answer renders one way. Only the two answer renderers (plus the shared
|
||||
page's) opt in: a question, a thinking run and the document viewer stay
|
||||
one-argument calls, and `document.html` does not even load the panel client.
|
||||
|
||||
---
|
||||
|
||||
## PDF documents (phase 126)
|
||||
|
||||
PDFs are **opt-in**: add the `pdf` token to `BOR_IMPORT_EXTENSIONS`
|
||||
@@ -1339,6 +1505,9 @@ startup (the DB rows shadow the env, `.env` is never rewritten).
|
||||
|-----|---------|---------|
|
||||
| `BOR_AGENT_MAX_ROUNDS` | `10` | Hard cap on agent tool rounds per grounded turn (0 = no tools); every tool result carries the model's remaining count (phase 160) |
|
||||
| `BOR_READ_MAX_CHARS` | `128000` | Cap on a `read` tool result (chars); over-cap reads truncate + a notice that teaches the grep → `read(offset, limit)` workflow (phase 95, rewritten phase 131) |
|
||||
| `BOR_ANSWER_TEXT` | `1` | Verbatim file display (phase 182): the **master switch, ON by default** — an answer's code block is typed by the server from the indexed bytes, never retyped by the model. `0` = the escape hatch: the `show_text` tool is not offered, its `<tools>` teaching clause is absent, ` ```bor ` locator fences are not resolved, and the `done` frame carries no `answer_text` key (the server surface is exactly pre-phase; a stray fence on the client still settles to the honest unresolved sentence rather than the model's copy) |
|
||||
| `BOR_SHOW_TEXT_MAX_CHARS` | `12000` | Per-block character budget for ONE displayed verbatim slice (phase 182): the slice is cut at a **line boundary**, the shared `[…truncated…]` marker is appended and the caption's line range shrinks to what is actually shown. `0` = inert (nothing displays); a negative value refuses startup naming the field |
|
||||
| `BOR_SHOW_TEXT_MAX_BLOCKS` | `5` | Per-answer cap on displayed verbatim blocks (phase 182), counted in answer order. `0` = inert (nothing displays); a negative value refuses startup naming the field |
|
||||
|
||||
### Import & sources
|
||||
|
||||
@@ -1364,7 +1533,7 @@ startup (the DB rows shadow the env, `.env` is never rewritten).
|
||||
| `BOR_IMAGE_DIR` | `~/bor-sources/images` | Persistent home for the served image bytes (uploads are replaced, checkouts re-cloned) |
|
||||
| `BOR_CHAT_IMAGE_DIR` | `~/bor-sources/chat-images` | Where question images are stored (saved/shared chats carry the path, never base64) |
|
||||
| `BOR_CHAT_IMAGE_MAX_MB` | `10` | Question-image upload cap (MiB; must be > 0) |
|
||||
| `BOR_ANSWER_IMAGES` | `0` | Let the chat model show a standalone image document as part of its answer (phase 124): offers the agent the `show_image` tool and renders the shown image as an answer figure — **independent of `BOR_IMAGES`** (that toggle indexes image docs; this one shows them) |
|
||||
| `BOR_ANSWER_IMAGES` | `0` | Let the chat model show a standalone image document as part of its answer (phase 124): offers the agent the `show_image` tool and renders the shown image as an answer figure — **independent of `BOR_IMAGES`** (that toggle indexes image docs; this one shows them). The text-display sibling (`BOR_ANSWER_TEXT`, phase 182) lives in [Agent tools](#agent-tools) |
|
||||
|
||||
### Docs push
|
||||
|
||||
|
||||
Reference in New Issue
Block a user