From 76887639f3472a02496d7f691608ff5d77162d44 Mon Sep 17 00:00:00 2001 From: ducoterra Date: Wed, 14 Oct 2026 10:33:06 -0400 Subject: [PATCH] docs(phases): add phase 182 (verbatim file display) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Answers to "show me my pi.dev config" re-type file contents from context, so the code block in the answer can differ from the indexed file by a character and cannot be trusted for copy-paste — the owner opens the source bubble and ignores the answer. Phase 182 makes the server the only party that supplies bytes: - show_text(path, offset?, limit?), the display sibling of show_image, slices documents.content and hands the same bytes back to the model with a "do not retype" instruction - inline ```bor source/path 12-40 locator fences place a resolved block anywhere in the prose; the model's fence body is never painted, and an unresolvable locator says so instead of showing guessed text - the bytes ride inline in the done frame's additive answer_text array and persist in the saved chat, so restore and the shared page match - line numbers are always the server's resolved extent (overshoot clamps, past-the-end refuses to the model), which removes the "did the model invent those numbers?" problem entirely - a Copy button on every answer code block, a VERBATIM badge only on server-resolved ones - gated by BOR_ANSWER_TEXT with per-block/per-answer budgets; with the flag off every frame, record and byte budget is exactly pre-phase Overview plus eight tasks, each with its own test gate; the phase E2E runs in isolation and asserts the clipboard content equals the stored bytes. Run 180 first — both touch markdown.js. LLM-Generated: true --- .../todo/182_verbatim_show_text/00_phase.md | 79 +++++++++++++++++++ .../01_settings_and_kill_switch.md | 33 ++++++++ .../02_verbatim_resolver_module.md | 47 +++++++++++ .../03_show_text_tool.md | 57 +++++++++++++ .../04_prompt_teaching_clause.md | 36 +++++++++ .../05_done_frame_and_persistence.md | 34 ++++++++ .../06_markdown_locator_and_panel_css.md | 43 ++++++++++ .../07_hydration_and_copy_buttons.md | 43 ++++++++++ .../08_e2e_and_docs_pass.md | 50 ++++++++++++ 9 files changed, 422 insertions(+) create mode 100644 .agents/phases/todo/182_verbatim_show_text/00_phase.md create mode 100644 .agents/phases/todo/182_verbatim_show_text/01_settings_and_kill_switch.md create mode 100644 .agents/phases/todo/182_verbatim_show_text/02_verbatim_resolver_module.md create mode 100644 .agents/phases/todo/182_verbatim_show_text/03_show_text_tool.md create mode 100644 .agents/phases/todo/182_verbatim_show_text/04_prompt_teaching_clause.md create mode 100644 .agents/phases/todo/182_verbatim_show_text/05_done_frame_and_persistence.md create mode 100644 .agents/phases/todo/182_verbatim_show_text/06_markdown_locator_and_panel_css.md create mode 100644 .agents/phases/todo/182_verbatim_show_text/07_hydration_and_copy_buttons.md create mode 100644 .agents/phases/todo/182_verbatim_show_text/08_e2e_and_docs_pass.md diff --git a/.agents/phases/todo/182_verbatim_show_text/00_phase.md b/.agents/phases/todo/182_verbatim_show_text/00_phase.md new file mode 100644 index 0000000..3ec097d --- /dev/null +++ b/.agents/phases/todo/182_verbatim_show_text/00_phase.md @@ -0,0 +1,79 @@ +# Phase 182 — Verbatim file display: a `show_text` tool + `bor` locator fences the server resolves + +**Story:** n/a (owner chat 2026-10-14 — "show me my pi.dev config" / "show me the quadlet for gufo" answers are re-typed from context and silently add/remove characters, so the answer text cannot be copy-pasted; the owner has to open the source bubble and ignore the written code block) +**Context:** `app/rag/agent.py` — the harness-aligned tool surface `AGENT_TOOLS` (L332: `ls` / `read` / `grep`) plus the 4th, DISPLAY tool `SHOW_IMAGE_TOOL` (L501) composed by `agent_tools(settings)` (L538), the shared identity resolver `_resolve_path` (L1357), the line-range validator `_read_range_args` (L1474) and the `OFFSET_PAST_END` (L671) / `BAD_RANGE_ARGS` (L628) teachings, the refusal block L571–L690 (`MISSING_SHOW_ARGS` L571, `NOT_AN_IMAGE_DOC` L581, `ALREADY_SHOWING` L585), `AgentHolder` (L1600–L1680: `shown_images`, `read_coverage`, `read_truncations`), `_execute_tool` (L1681, the `show_image` branch L2023–L2068 is the shape this phase mirrors); `app/rag/prompts.py` — `TOOLS_SECTION` (L293) and the appended-only-when-offered `SHOW_IMAGE_CLAUSE` (L379) + `build_high_prompt(show_image=…)` (L759); `app/api/chat.py` — the prompt/tool call site `show_image=settings.answer_images` (L529), the per-turn visible-content counter `content_chars` (L934), the `done`-frame builder (L1320–L1365, `answer_ref`/`answer_image` L1343–L1355), the pinned per-turn log line (L1272–L1291); `app/schemas.py` — `SourceRef` (L169), `ChatToolResultEvent` (L279), `ChatDoneEvent.answer_image` + its omission serializer (L305–L380), `ChatMessageRecord.answer_image` + `_drop_image_when_absent` (L1463, L1478); `app/config.py` — `read_max_chars` (L296) + its validator (L686), `answer_images: bool = False` (L539), `log_large_response_bytes` (L76); `app/core/settings_catalog.py` — `read_max_chars` (L267) / `answer_images` (L296) rows (the completeness pin in `tests/unit/test_settings_catalog.py` makes a new `Settings` field a build failure until it has a row); `frontend/assets/markdown.js` — `renderMarkdown(md)` (L43), fence-protection pass (L44–L48: fences → tables → escape+inline → blockquotes → paragraphs → restore, escape-first, no CDN); `frontend/assets/app.js` — the per-delta whole-bubble re-render `renderMarkdown(acc)` (L4322), the `done` handler (L4325–L4460), `appendAnswerImage` (L2598), the restore renderer (L2890–L2960), `copyShareLinkWithFallback` (L3250, the http-origin clipboard degradation), `rememberBrainTurn` save point (L4390–L4420); `frontend/assets/shared.js` — `addAnswerImage` (L573) + the record renderer (L619–L660); `frontend/index.html` L1696–L1701 and `frontend/shared.html` L197–L199 (classic `brand.js` + `markdown.js`, then the module script); `frontend/assets/styles.css` — `.answer-image` (L1028–L1059), the bounded-scroll-panel and `.cell-clip` conventions (PLAN §7.1, phases 156/167); `tests/e2e/mock_llm.py` — the scripted-trigger registry (L1162–L1520, `SHOW_IMAGE_TRIGGER` L2606, `_SHOW_CONFIRM_MARKER`, the "verified: no existing E2E question or fixture file contains the phrase" discipline); `tests/e2e/test_answer_images.py` (the two-app ON/OFF pattern), `tests/unit/test_frontend_*.py` (the JS/CSS text-pin convention). + +## Objective + +When the owner asks *"show me the quadlet for gufo"*, the exact bytes of the indexed file reach the chat answer — **typed by the server, never by the model** — in a panel with a path + line-range caption, a `VERBATIM` badge, line numbers, a working Copy button and a link to the source. The model only *names* the location (`show_text(path, offset?, limit?)`, or an inline ` ```bor ` locator fence where it wants the block to sit); the server resolves the slice from Postgres and the client places it. This kills the class of bug where the answer's code block silently differs from the file, and it removes the user-facing line-number problem entirely: the numbers shown are the server's resolved extent, not the model's claim. + +## Design (shared by all tasks — the executor reads this, not the chat) + +- **D1 — the two mechanisms, one authority (A1).** (a) a 5th agent tool `show_text(path, offset?, limit?)`, the display sibling of `show_image`; (b) an **inline locator fence** the model may write in its answer — ` ```bor ` + the combined `source/path` + an optional range — so a resolved block can sit anywhere in the prose. **Both are resolution requests, never content requests**: the only party that supplies bytes is the server. `show_text` is the authority for teaching (a refusal teaches the model to self-correct before the user sees anything); the fence is placement sugar and ALSO resolves standalone (a fence with no prior `show_text` call still renders, because resolution is a plain `(source, path)` lookup, not a replay of a tool call). +- **D2 — server resolves, client places (A4/A7).** The client cannot resolve (no new route, A4), so: `app/api/chat.py` accumulates the turn's **clean visible answer text** (next to the existing `content_chars` counter, L934 — the post-scaffold-filter text only, never `thinking`), and at `done`-build time passes it through `app.rag.verbatim.resolve_answer_blocks(db, answer, holder.shown_texts, settings)`; the result rides the `done` frame as the ADDITIVE `answer_text` array. `markdown.js` turns a `bor` fence into `
` and **drops the model's fence body**; `app.js`/`shared.js` hydrate each placeholder by matching `data-bor-locator` against the record's `answer_text[].locator` (canonical-form match → unique path-only match → unresolved). A block the tool registered that the answer never locates renders APPENDED, at the `answer_image` position (below the answer, above the source clusters). +- **D3 — the model's bytes are NEVER painted (A7, the phase's core contract).** Inside a `bor` fence the model's body is discarded at render time — while streaming the placeholder reads `Retrieving the exact text…` (`role="status"`, polite by definition), and a fence the model has not finished typing shows that SAME pending state rather than raw backticks, at `done` it becomes the resolved panel or, when it cannot be resolved, the honest state `I could not verify this against the knowledge base — open ` (persona: radically honest, never a silent substitution and never the model's possibly-corrupt text). The E2E proves it with a mock that deliberately types a CORRUPTED body and asserts the panel carries the DB bytes. `renderMarkdown` gains a SECOND, OPTIONAL argument (`{ verbatim: true }`); default OFF keeps every existing call site — the document viewer (`document.js` L494), the thinking blocks, the user bubbles — byte-identical, and a document that legitimately contains a ` ```bor ` fence still renders its own text. +- **D4 — line ranges are the server's (A2).** `offset` (1-based) / `limit` are OPTIONAL and validated by the existing `_read_range_args` (L1474): omitted → the WHOLE document (the taught default for "show me X"); `offset` alone → to the end; `limit` overshoot → CLAMPED to the last line; `offset` past the last line → the `OFFSET_PAST_END`-style refusal naming the document's true line count (counts in nothing, consumes one round, invisible to the user). Slicing uses `content.splitlines()` with the SAME 1-based indexing as `read` (L1836) and `grep`, so every line number in the app agrees. The char cap (`settings.show_text_max_chars`) cuts the SLICE at a line boundary, appends the shared `TRUNCATION_MARKER` (`app/rag/retriever.py`) and shrinks `end_line` to the last fully-shown line — the caption never lies. +- **D5 — what the model gets back (A3).** The tool result is a confirmation + the truth + an instruction, then the exact slice: + `Text shown to the user — these lines are displayed verbatim; describe them, do NOT retype them.` / `Document {source}/{path}:` / `date: …` / `{origin lines}` / `lines: {first}-{end} of {total_lines}` / the slice. The `Document …` + `date:` + `lines:` header contract is the one `read` established (the E2E mock's `_READ_RESULT_PREFIX` pin keeps parsing it). Returning the text is deliberate (A21: LLM calls are free) — the model's commentary can then never disagree with the panel. +- **D6 — new module, no import cycle (D2).** `app/rag/verbatim.py` owns `VerbatimBlock` (frozen dataclass: `source, path, title, start_line, end_line, total_lines, text, truncated, locator`), the pure half (`canonical_locator`, `slice_lines`, `fences_in`, `cap_slice`) and one DB function (`document_by_identity(db, combined)`, a bare `select(Document).where(source==, path==)` — the same `(source, path)` identity contract as `agent._resolve_path`, WITHOUT its did-you-mean suggestions). `agent.py` imports the pure half, so `verbatim.py` must never import `agent.py` (documented at the module top — protocol D2). Path traversal is a non-issue by construction: a traversal string is simply a non-existent row (404-shaped refusal, no filesystem access anywhere). +- **D7 — canonical locator (the matching key).** `source/path` for a whole document, `source/path#12-40` for a range. The tolerant input grammar (both the fence parser and the tool's `path` argument) accepts `source/path`, `source/path 12-40`, `source/path:12-40`, `source/path#12-40`, `source/path#L12-L40` and `source/path 12` (single line); anything unparseable is not a `bor` locator at all and renders as an ordinary code block (the body is then the model's own, exactly as today, with a Copy button and no `VERBATIM` badge). Resolution normalizes to the canonical form, so the client's match is a string compare. +- **D8 — caps, kill switch, observability (A8).** `answer_text: bool = True` (`BOR_ANSWER_TEXT`, the master switch — `false` = the tool is not offered, the prompt clause is absent, resolution does not run, no `answer_text` key: every frame and every byte-identity contract is exactly pre-phase), `show_text_max_chars: int = 12_000` (per block; `0` = inert, negative fails startup naming the field — the `agent_max_rounds`/`read_max_chars` pattern), `show_text_max_blocks: int = 5` (per answer, in answer order; `0` = inert, negative fails startup). Worst-case frame growth 5 × 12 000 = 60 KB against the 256 KB budget, which the phase-167 byte-budget line (`api … bytes=…`, `grep ' :: api '`) makes visible. The §9 per-turn line gains ONE additive trailing field `verbatim_blocks=N` (the blocks actually sent, tool-registered + fence-resolved, deduped) — `tests/unit/test_chat_gate.py`'s pinned line is updated in the same task, never silently. +- **D9 — resolution scope (the honesty posture).** Resolution runs on GROUNDED turns only (the agent ran). A `bor` fence hallucinated into a DEFLECTED answer has no registered blocks and the gate said "I haven't done anything like that" — it renders the unresolved state rather than leaking KB text through a turn that just disclaimed it. `done.sources` / `related` semantics are NOT changed (no regression to the citation surface): the panel itself carries the path, the line range, a Copy button and a link to the document viewer (the git origin link stays the viewer's job, phase 145's `origin_for_source` — the panel does not duplicate it) — the citation affordance sits at the point of use. +- **NOT touched:** `show_image` / `SHOW_IMAGE_TOOL` / `answer_image` (image display is out of scope, A6 — verified, not modified), the `ls`/`read`/`grep` contracts, the SSE event-type LIST (A15 stays at seven types — this phase rides `done`, no 8th event), `documents`/`chunks` schema (no migration: `saved_chats.messages` is raw JSONB, the phase-50/160 precedent), `document.js` (the viewer renders documents, not answers), `app/core/response_budget.py`, `scripts/*`, `frontend/assets/router.js`. + +## Locked decisions (owner-confirmed 2026-10-14, from the phase-authoring interview) + +- **A1 — BOTH mechanisms in this phase:** the `show_text` tool AND the inline ` ```bor ` locator fence. (The owner chose "A + B in one phase" over the tool alone.) +- **A2 — whole-file default + server clamp:** `offset`/`limit` optional; omitted = the whole document; overshoot clamps; past-the-end refuses to the model with the true total; the caption always shows the SERVER's resolved extent. No `expect`-anchor snapping (considered and deferred — the clamp plus the model's own `grep`/`read` line numbers cover it). +- **A3 — the tool returns the verbatim slice** plus the "do not retype" instruction (not a terse confirmation, not the read_coverage-conditional variant). +- **A4 — the bytes ride INLINE** in `done.answer_text` and persist in the `bor.chat.v1` brain record: no new bytes route, no new SSE event type, no new anonymous surface (a shared page renders from the share's own record, the same trust model as the phase-123 chat image). +- **A5 — Copy buttons on EVERY fenced code block in an answer** (live, restored and shared), but the `VERBATIM` badge + caption belong ONLY to a server-resolved block. The clipboard goes through the phase-55 degradation (a non-secure origin gets the inline select-to-copy fallback) — never a bare `navigator.clipboard` assumption. +- **A6 — images are out of scope:** `show_image` keeps images; `show_text` on an `is_image` document is REFUSED with a pointer to `show_image`. The phase's E2E asserts the answer figure is unchanged. +- **A7 — the model's fence body never reaches the DOM** (see D3). Unresolvable = honest, visible, non-copyable-as-truth state. +- **A8 — settings:** `BOR_ANSWER_TEXT` default `true` (master switch), `BOR_SHOW_TEXT_MAX_CHARS` default `12000`, `BOR_SHOW_TEXT_MAX_BLOCKS` default `5`; `0` = inert, negative = loud startup failure. +- **A9 — additive wire only:** `answer_text` is dropped when empty (absent, never `null`) — every frame without blocks, every pre-phase record and the whole default-off configuration stay byte-identical. No PLAN §2/§4 anchor changes beyond recording these revisions. + +## Dependencies + +- `124_answer_images` (complete) — the display-tool precedent this phase mirrors end to end (`SHOW_IMAGE_TOOL` → `holder.shown_images` → `done.answer_image` → `appendAnswerImage` → persistence → shared page). +- `131_read_line_ranges` / `141_read_range_dedupe_fix` (complete) — `offset`/`limit` + `_read_range_args` + the `lines: N-M of T` header contract and `splitlines()` indexing. +- `145_git_source_links` (complete) — the document viewer's `origin_for_source` link, which the panel links TO rather than duplicating. +- `167_api_payload_diet` (complete) — the byte-budget line that will show the `done`-frame growth. +- `180_markdown_inline_links_images` (todo, already authored) — independent of this phase; both touch `markdown.js` and `styles.css`, so run it FIRST (or rebase) to avoid a renderer conflict. + +## Tasks + +1. `01_settings_and_kill_switch.md` — the three `Settings` fields + validators + the three `CATALOG` rows + `.env.example` (+ their unit/integration pins). Inert: nothing reads them yet. +2. `02_verbatim_resolver_module.md` — `app/rag/verbatim.py`: the locator grammar, the canonical normalizer, `splitlines()` slicing, the line-boundary cap and `resolve_answer_blocks` (+ the resolver unit suite). +3. `03_show_text_tool.md` — `SHOW_TEXT_TOOL`, the `agent_tools` gate, the `_execute_tool` branch, the refusal set, `AgentHolder.shown_texts` (+ agent unit suite). +4. `04_prompt_teaching_clause.md` — `SHOW_TEXT_CLAUSE` inside ``, `build_high_prompt(show_text=…)`, the `app/api/chat.py` call site (+ byte-identity pins when off). +5. `05_done_frame_and_persistence.md` — the `VerbatimBlockRef` schema, `ChatDoneEvent.answer_text`, `ChatMessageRecord.answer_text`, the visible-text accumulator + resolution + `verbatim_blocks=N` in the per-turn line (+ integration/frame tests). +6. `06_markdown_locator_and_panel_css.md` — `renderMarkdown(md, opts)`'s opt-in `verbatim` pass → `.verbatim[data-bor-locator]` (body dropped; ordinary fences keep rendering exactly as today), and the `styles.css` panel/copy-button CSS contract. Inert until task 07 opts in and wraps `
` nodes at the DOM level.
+7. `07_hydration_and_copy_buttons.md` — `frontend/assets/verbatim.js` (+ `index.html`/`shared.html` script tags): hydrate-at-`done`, restore, shared page, error/settled paths, `rememberBrainTurn` persistence, copy buttons on every answer code block.
+8. `08_e2e_and_docs_pass.md` — the mock triggers (`show_text` call, corrupt-body fence, fence-only, off-app), `tests/e2e/test_verbatim_show_text.py` in isolation, the regression set, the full gates, the README/AGENTS full consistency pass, the atomic commit.
+
+## Testing & Quality
+
+- **Unit:** `tests/unit/test_config.py` (the three defaults; env overrides; `0` accepted as inert; each negative value fails validation naming the field); `tests/unit/test_settings_catalog.py` (the completeness pin green with the three new rows); `tests/unit/test_verbatim_resolver.py` (NEW — the locator grammar incl. every tolerant form + rejects, canonical normalization, whole-file/offset/limit/clamp/past-end, the line-boundary cap + `TRUNCATION_MARKER` + shrunk `end_line`, `fences_in` ordering + dedupe, tool-then-fence dedupe, `max_blocks` cut, an `is_image` doc refused, an unknown path unresolved, a fenced `bor` line inside an ordinary fence, an unclosed fence, an empty answer); `tests/unit/test_agent.py` (the `show_text` branch: success shape + headers, `shown_texts`, refusal set — missing args, non-string, unknown path did-you-mean, image doc, bad range args, offset past end, block cap, duplicate identical show; every refusal counts in nothing and executes nothing; `agent_tools` offers 4 vs 5 tools by flag); `tests/unit/test_prompts.py` (the clause is INSIDE `` before ``, the off-build is byte-identical to pre-phase, `TOOLS_SECTION` itself unchanged); `tests/unit/test_chat_gate.py` (the per-turn line's new `verbatim_blocks=N`, and `0` on a turn with none); `tests/unit/test_frontend_verbatim_markdown.py` + `tests/unit/test_frontend_verbatim_blocks.py` + a CSS pin module (the `test_frontend_*` text-pin convention: the `verbatim` option, the dropped body, `data-bor-locator`, the `.verbatim`/`.code-block`/`.verbatim-copy` markers, the new `