docs(phases): add phase 182 (verbatim file display)
Answers to "show me my pi.dev config" re-type file contents from context, so the code block in the answer can differ from the indexed file by a character and cannot be trusted for copy-paste — the owner opens the source bubble and ignores the answer. Phase 182 makes the server the only party that supplies bytes: - show_text(path, offset?, limit?), the display sibling of show_image, slices documents.content and hands the same bytes back to the model with a "do not retype" instruction - inline ```bor source/path 12-40 locator fences place a resolved block anywhere in the prose; the model's fence body is never painted, and an unresolvable locator says so instead of showing guessed text - the bytes ride inline in the done frame's additive answer_text array and persist in the saved chat, so restore and the shared page match - line numbers are always the server's resolved extent (overshoot clamps, past-the-end refuses to the model), which removes the "did the model invent those numbers?" problem entirely - a Copy button on every answer code block, a VERBATIM badge only on server-resolved ones - gated by BOR_ANSWER_TEXT with per-block/per-answer budgets; with the flag off every frame, record and byte budget is exactly pre-phase Overview plus eight tasks, each with its own test gate; the phase E2E runs in isolation and asserts the clipboard content equals the stored bytes. Run 180 first — both touch markdown.js. LLM-Generated: true
This commit is contained in:
@@ -0,0 +1,79 @@
|
||||
# Phase 182 — Verbatim file display: a `show_text` tool + `bor` locator fences the server resolves
|
||||
|
||||
**Story:** n/a (owner chat 2026-10-14 — "show me my pi.dev config" / "show me the quadlet for gufo" answers are re-typed from context and silently add/remove characters, so the answer text cannot be copy-pasted; the owner has to open the source bubble and ignore the written code block)
|
||||
**Context:** `app/rag/agent.py` — the harness-aligned tool surface `AGENT_TOOLS` (L332: `ls` / `read` / `grep`) plus the 4th, DISPLAY tool `SHOW_IMAGE_TOOL` (L501) composed by `agent_tools(settings)` (L538), the shared identity resolver `_resolve_path` (L1357), the line-range validator `_read_range_args` (L1474) and the `OFFSET_PAST_END` (L671) / `BAD_RANGE_ARGS` (L628) teachings, the refusal block L571–L690 (`MISSING_SHOW_ARGS` L571, `NOT_AN_IMAGE_DOC` L581, `ALREADY_SHOWING` L585), `AgentHolder` (L1600–L1680: `shown_images`, `read_coverage`, `read_truncations`), `_execute_tool` (L1681, the `show_image` branch L2023–L2068 is the shape this phase mirrors); `app/rag/prompts.py` — `TOOLS_SECTION` (L293) and the appended-only-when-offered `SHOW_IMAGE_CLAUSE` (L379) + `build_high_prompt(show_image=…)` (L759); `app/api/chat.py` — the prompt/tool call site `show_image=settings.answer_images` (L529), the per-turn visible-content counter `content_chars` (L934), the `done`-frame builder (L1320–L1365, `answer_ref`/`answer_image` L1343–L1355), the pinned per-turn log line (L1272–L1291); `app/schemas.py` — `SourceRef` (L169), `ChatToolResultEvent` (L279), `ChatDoneEvent.answer_image` + its omission serializer (L305–L380), `ChatMessageRecord.answer_image` + `_drop_image_when_absent` (L1463, L1478); `app/config.py` — `read_max_chars` (L296) + its validator (L686), `answer_images: bool = False` (L539), `log_large_response_bytes` (L76); `app/core/settings_catalog.py` — `read_max_chars` (L267) / `answer_images` (L296) rows (the completeness pin in `tests/unit/test_settings_catalog.py` makes a new `Settings` field a build failure until it has a row); `frontend/assets/markdown.js` — `renderMarkdown(md)` (L43), fence-protection pass (L44–L48: fences → tables → escape+inline → blockquotes → paragraphs → restore, escape-first, no CDN); `frontend/assets/app.js` — the per-delta whole-bubble re-render `renderMarkdown(acc)` (L4322), the `done` handler (L4325–L4460), `appendAnswerImage` (L2598), the restore renderer (L2890–L2960), `copyShareLinkWithFallback` (L3250, the http-origin clipboard degradation), `rememberBrainTurn` save point (L4390–L4420); `frontend/assets/shared.js` — `addAnswerImage` (L573) + the record renderer (L619–L660); `frontend/index.html` L1696–L1701 and `frontend/shared.html` L197–L199 (classic `brand.js` + `markdown.js`, then the module script); `frontend/assets/styles.css` — `.answer-image` (L1028–L1059), the bounded-scroll-panel and `.cell-clip` conventions (PLAN §7.1, phases 156/167); `tests/e2e/mock_llm.py` — the scripted-trigger registry (L1162–L1520, `SHOW_IMAGE_TRIGGER` L2606, `_SHOW_CONFIRM_MARKER`, the "verified: no existing E2E question or fixture file contains the phrase" discipline); `tests/e2e/test_answer_images.py` (the two-app ON/OFF pattern), `tests/unit/test_frontend_*.py` (the JS/CSS text-pin convention).
|
||||
|
||||
## Objective
|
||||
|
||||
When the owner asks *"show me the quadlet for gufo"*, the exact bytes of the indexed file reach the chat answer — **typed by the server, never by the model** — in a panel with a path + line-range caption, a `VERBATIM` badge, line numbers, a working Copy button and a link to the source. The model only *names* the location (`show_text(path, offset?, limit?)`, or an inline ` ```bor <path> <range> ` locator fence where it wants the block to sit); the server resolves the slice from Postgres and the client places it. This kills the class of bug where the answer's code block silently differs from the file, and it removes the user-facing line-number problem entirely: the numbers shown are the server's resolved extent, not the model's claim.
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
|
||||
- **D1 — the two mechanisms, one authority (A1).** (a) a 5th agent tool `show_text(path, offset?, limit?)`, the display sibling of `show_image`; (b) an **inline locator fence** the model may write in its answer — ` ```bor ` + the combined `source/path` + an optional range — so a resolved block can sit anywhere in the prose. **Both are resolution requests, never content requests**: the only party that supplies bytes is the server. `show_text` is the authority for teaching (a refusal teaches the model to self-correct before the user sees anything); the fence is placement sugar and ALSO resolves standalone (a fence with no prior `show_text` call still renders, because resolution is a plain `(source, path)` lookup, not a replay of a tool call).
|
||||
- **D2 — server resolves, client places (A4/A7).** The client cannot resolve (no new route, A4), so: `app/api/chat.py` accumulates the turn's **clean visible answer text** (next to the existing `content_chars` counter, L934 — the post-scaffold-filter text only, never `thinking`), and at `done`-build time passes it through `app.rag.verbatim.resolve_answer_blocks(db, answer, holder.shown_texts, settings)`; the result rides the `done` frame as the ADDITIVE `answer_text` array. `markdown.js` turns a `bor` fence into `<div class="verbatim" data-bor-locator="<canonical>">` and **drops the model's fence body**; `app.js`/`shared.js` hydrate each placeholder by matching `data-bor-locator` against the record's `answer_text[].locator` (canonical-form match → unique path-only match → unresolved). A block the tool registered that the answer never locates renders APPENDED, at the `answer_image` position (below the answer, above the source clusters).
|
||||
- **D3 — the model's bytes are NEVER painted (A7, the phase's core contract).** Inside a `bor` fence the model's body is discarded at render time — while streaming the placeholder reads `Retrieving the exact text…` (`role="status"`, polite by definition), and a fence the model has not finished typing shows that SAME pending state rather than raw backticks, at `done` it becomes the resolved panel or, when it cannot be resolved, the honest state `I could not verify this against the knowledge base — open <source>` (persona: radically honest, never a silent substitution and never the model's possibly-corrupt text). The E2E proves it with a mock that deliberately types a CORRUPTED body and asserts the panel carries the DB bytes. `renderMarkdown` gains a SECOND, OPTIONAL argument (`{ verbatim: true }`); default OFF keeps every existing call site — the document viewer (`document.js` L494), the thinking blocks, the user bubbles — byte-identical, and a document that legitimately contains a ` ```bor ` fence still renders its own text.
|
||||
- **D4 — line ranges are the server's (A2).** `offset` (1-based) / `limit` are OPTIONAL and validated by the existing `_read_range_args` (L1474): omitted → the WHOLE document (the taught default for "show me X"); `offset` alone → to the end; `limit` overshoot → CLAMPED to the last line; `offset` past the last line → the `OFFSET_PAST_END`-style refusal naming the document's true line count (counts in nothing, consumes one round, invisible to the user). Slicing uses `content.splitlines()` with the SAME 1-based indexing as `read` (L1836) and `grep`, so every line number in the app agrees. The char cap (`settings.show_text_max_chars`) cuts the SLICE at a line boundary, appends the shared `TRUNCATION_MARKER` (`app/rag/retriever.py`) and shrinks `end_line` to the last fully-shown line — the caption never lies.
|
||||
- **D5 — what the model gets back (A3).** The tool result is a confirmation + the truth + an instruction, then the exact slice:
|
||||
`Text shown to the user — these lines are displayed verbatim; describe them, do NOT retype them.` / `Document {source}/{path}:` / `date: …` / `{origin lines}` / `lines: {first}-{end} of {total_lines}` / the slice. The `Document …` + `date:` + `lines:` header contract is the one `read` established (the E2E mock's `_READ_RESULT_PREFIX` pin keeps parsing it). Returning the text is deliberate (A21: LLM calls are free) — the model's commentary can then never disagree with the panel.
|
||||
- **D6 — new module, no import cycle (D2).** `app/rag/verbatim.py` owns `VerbatimBlock` (frozen dataclass: `source, path, title, start_line, end_line, total_lines, text, truncated, locator`), the pure half (`canonical_locator`, `slice_lines`, `fences_in`, `cap_slice`) and one DB function (`document_by_identity(db, combined)`, a bare `select(Document).where(source==, path==)` — the same `(source, path)` identity contract as `agent._resolve_path`, WITHOUT its did-you-mean suggestions). `agent.py` imports the pure half, so `verbatim.py` must never import `agent.py` (documented at the module top — protocol D2). Path traversal is a non-issue by construction: a traversal string is simply a non-existent row (404-shaped refusal, no filesystem access anywhere).
|
||||
- **D7 — canonical locator (the matching key).** `source/path` for a whole document, `source/path#12-40` for a range. The tolerant input grammar (both the fence parser and the tool's `path` argument) accepts `source/path`, `source/path 12-40`, `source/path:12-40`, `source/path#12-40`, `source/path#L12-L40` and `source/path 12` (single line); anything unparseable is not a `bor` locator at all and renders as an ordinary code block (the body is then the model's own, exactly as today, with a Copy button and no `VERBATIM` badge). Resolution normalizes to the canonical form, so the client's match is a string compare.
|
||||
- **D8 — caps, kill switch, observability (A8).** `answer_text: bool = True` (`BOR_ANSWER_TEXT`, the master switch — `false` = the tool is not offered, the prompt clause is absent, resolution does not run, no `answer_text` key: every frame and every byte-identity contract is exactly pre-phase), `show_text_max_chars: int = 12_000` (per block; `0` = inert, negative fails startup naming the field — the `agent_max_rounds`/`read_max_chars` pattern), `show_text_max_blocks: int = 5` (per answer, in answer order; `0` = inert, negative fails startup). Worst-case frame growth 5 × 12 000 = 60 KB against the 256 KB budget, which the phase-167 byte-budget line (`api … bytes=…`, `grep ' :: api '`) makes visible. The §9 per-turn line gains ONE additive trailing field `verbatim_blocks=N` (the blocks actually sent, tool-registered + fence-resolved, deduped) — `tests/unit/test_chat_gate.py`'s pinned line is updated in the same task, never silently.
|
||||
- **D9 — resolution scope (the honesty posture).** Resolution runs on GROUNDED turns only (the agent ran). A `bor` fence hallucinated into a DEFLECTED answer has no registered blocks and the gate said "I haven't done anything like that" — it renders the unresolved state rather than leaking KB text through a turn that just disclaimed it. `done.sources` / `related` semantics are NOT changed (no regression to the citation surface): the panel itself carries the path, the line range, a Copy button and a link to the document viewer (the git origin link stays the viewer's job, phase 145's `origin_for_source` — the panel does not duplicate it) — the citation affordance sits at the point of use.
|
||||
- **NOT touched:** `show_image` / `SHOW_IMAGE_TOOL` / `answer_image` (image display is out of scope, A6 — verified, not modified), the `ls`/`read`/`grep` contracts, the SSE event-type LIST (A15 stays at seven types — this phase rides `done`, no 8th event), `documents`/`chunks` schema (no migration: `saved_chats.messages` is raw JSONB, the phase-50/160 precedent), `document.js` (the viewer renders documents, not answers), `app/core/response_budget.py`, `scripts/*`, `frontend/assets/router.js`.
|
||||
|
||||
## Locked decisions (owner-confirmed 2026-10-14, from the phase-authoring interview)
|
||||
|
||||
- **A1 — BOTH mechanisms in this phase:** the `show_text` tool AND the inline ` ```bor ` locator fence. (The owner chose "A + B in one phase" over the tool alone.)
|
||||
- **A2 — whole-file default + server clamp:** `offset`/`limit` optional; omitted = the whole document; overshoot clamps; past-the-end refuses to the model with the true total; the caption always shows the SERVER's resolved extent. No `expect`-anchor snapping (considered and deferred — the clamp plus the model's own `grep`/`read` line numbers cover it).
|
||||
- **A3 — the tool returns the verbatim slice** plus the "do not retype" instruction (not a terse confirmation, not the read_coverage-conditional variant).
|
||||
- **A4 — the bytes ride INLINE** in `done.answer_text` and persist in the `bor.chat.v1` brain record: no new bytes route, no new SSE event type, no new anonymous surface (a shared page renders from the share's own record, the same trust model as the phase-123 chat image).
|
||||
- **A5 — Copy buttons on EVERY fenced code block in an answer** (live, restored and shared), but the `VERBATIM` badge + caption belong ONLY to a server-resolved block. The clipboard goes through the phase-55 degradation (a non-secure origin gets the inline select-to-copy fallback) — never a bare `navigator.clipboard` assumption.
|
||||
- **A6 — images are out of scope:** `show_image` keeps images; `show_text` on an `is_image` document is REFUSED with a pointer to `show_image`. The phase's E2E asserts the answer figure is unchanged.
|
||||
- **A7 — the model's fence body never reaches the DOM** (see D3). Unresolvable = honest, visible, non-copyable-as-truth state.
|
||||
- **A8 — settings:** `BOR_ANSWER_TEXT` default `true` (master switch), `BOR_SHOW_TEXT_MAX_CHARS` default `12000`, `BOR_SHOW_TEXT_MAX_BLOCKS` default `5`; `0` = inert, negative = loud startup failure.
|
||||
- **A9 — additive wire only:** `answer_text` is dropped when empty (absent, never `null`) — every frame without blocks, every pre-phase record and the whole default-off configuration stay byte-identical. No PLAN §2/§4 anchor changes beyond recording these revisions.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `124_answer_images` (complete) — the display-tool precedent this phase mirrors end to end (`SHOW_IMAGE_TOOL` → `holder.shown_images` → `done.answer_image` → `appendAnswerImage` → persistence → shared page).
|
||||
- `131_read_line_ranges` / `141_read_range_dedupe_fix` (complete) — `offset`/`limit` + `_read_range_args` + the `lines: N-M of T` header contract and `splitlines()` indexing.
|
||||
- `145_git_source_links` (complete) — the document viewer's `origin_for_source` link, which the panel links TO rather than duplicating.
|
||||
- `167_api_payload_diet` (complete) — the byte-budget line that will show the `done`-frame growth.
|
||||
- `180_markdown_inline_links_images` (todo, already authored) — independent of this phase; both touch `markdown.js` and `styles.css`, so run it FIRST (or rebase) to avoid a renderer conflict.
|
||||
|
||||
## Tasks
|
||||
|
||||
1. `01_settings_and_kill_switch.md` — the three `Settings` fields + validators + the three `CATALOG` rows + `.env.example` (+ their unit/integration pins). Inert: nothing reads them yet.
|
||||
2. `02_verbatim_resolver_module.md` — `app/rag/verbatim.py`: the locator grammar, the canonical normalizer, `splitlines()` slicing, the line-boundary cap and `resolve_answer_blocks` (+ the resolver unit suite).
|
||||
3. `03_show_text_tool.md` — `SHOW_TEXT_TOOL`, the `agent_tools` gate, the `_execute_tool` branch, the refusal set, `AgentHolder.shown_texts` (+ agent unit suite).
|
||||
4. `04_prompt_teaching_clause.md` — `SHOW_TEXT_CLAUSE` inside `<tools>`, `build_high_prompt(show_text=…)`, the `app/api/chat.py` call site (+ byte-identity pins when off).
|
||||
5. `05_done_frame_and_persistence.md` — the `VerbatimBlockRef` schema, `ChatDoneEvent.answer_text`, `ChatMessageRecord.answer_text`, the visible-text accumulator + resolution + `verbatim_blocks=N` in the per-turn line (+ integration/frame tests).
|
||||
6. `06_markdown_locator_and_panel_css.md` — `renderMarkdown(md, opts)`'s opt-in `verbatim` pass → `.verbatim[data-bor-locator]` (body dropped; ordinary fences keep rendering exactly as today), and the `styles.css` panel/copy-button CSS contract. Inert until task 07 opts in and wraps `<pre>` nodes at the DOM level.
|
||||
7. `07_hydration_and_copy_buttons.md` — `frontend/assets/verbatim.js` (+ `index.html`/`shared.html` script tags): hydrate-at-`done`, restore, shared page, error/settled paths, `rememberBrainTurn` persistence, copy buttons on every answer code block.
|
||||
8. `08_e2e_and_docs_pass.md` — the mock triggers (`show_text` call, corrupt-body fence, fence-only, off-app), `tests/e2e/test_verbatim_show_text.py` in isolation, the regression set, the full gates, the README/AGENTS full consistency pass, the atomic commit.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **Unit:** `tests/unit/test_config.py` (the three defaults; env overrides; `0` accepted as inert; each negative value fails validation naming the field); `tests/unit/test_settings_catalog.py` (the completeness pin green with the three new rows); `tests/unit/test_verbatim_resolver.py` (NEW — the locator grammar incl. every tolerant form + rejects, canonical normalization, whole-file/offset/limit/clamp/past-end, the line-boundary cap + `TRUNCATION_MARKER` + shrunk `end_line`, `fences_in` ordering + dedupe, tool-then-fence dedupe, `max_blocks` cut, an `is_image` doc refused, an unknown path unresolved, a fenced `bor` line inside an ordinary fence, an unclosed fence, an empty answer); `tests/unit/test_agent.py` (the `show_text` branch: success shape + headers, `shown_texts`, refusal set — missing args, non-string, unknown path did-you-mean, image doc, bad range args, offset past end, block cap, duplicate identical show; every refusal counts in nothing and executes nothing; `agent_tools` offers 4 vs 5 tools by flag); `tests/unit/test_prompts.py` (the clause is INSIDE `<tools>` before `</tools>`, the off-build is byte-identical to pre-phase, `TOOLS_SECTION` itself unchanged); `tests/unit/test_chat_gate.py` (the per-turn line's new `verbatim_blocks=N`, and `0` on a turn with none); `tests/unit/test_frontend_verbatim_markdown.py` + `tests/unit/test_frontend_verbatim_blocks.py` + a CSS pin module (the `test_frontend_*` text-pin convention: the `verbatim` option, the dropped body, `data-bor-locator`, the `.verbatim`/`.code-block`/`.verbatim-copy` markers, the new `<script>` tags in `index.html` + `shared.html` (plain relative `src` — no `cache-busting` token in the path, the `?v=` rewrite is the middleware's job, phase 160 A7), `shared.js` parity, no inline `onclick` — CSP `default-src 'self'`).
|
||||
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` (all three rows: `bool` + two `int`s, correct groups/labels, live apply without a restart, reset-to-env); the chat API suite — a grounded turn with a `show_text` call and/or a fence yields `done.answer_text` with the SERVER's text and extent; a turn with none has NO `answer_text` key (byte-identity); `answer_text=false` yields the pre-phase frame byte-identically; the saved-chat round trip preserves `answer_text` (`extra="forbid"` accepts it — the phase-113 lesson) and `GET /api/chats/{id}` returns it.
|
||||
- **E2E (AGENTS rule 4, task 08):** `tests/e2e/test_verbatim_show_text.py`, run **in isolation** — the panel's text equals the fixture file's bytes (the mock's corrupted fence body never appears anywhere in the DOM), the caption reads `source/path · lines 12–40 of N` from the SERVER's extent, `VERBATIM` badge + line numbers + Copy button copy EXACTLY the panel's bytes (clipboard read with granted permissions, or the fallback field), the panel's open link targets `/document?source=…&path=…`, an unresolvable locator renders the honest state with no model bytes, restored-after-reload and the anonymous shared page render the same panel, every ordinary code block has a Copy button and no badge, and the `show_image` figure still renders (A6 regression).
|
||||
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
|
||||
- **Lint/types:** `uv run ruff check . && uv run pyright`.
|
||||
|
||||
## Documentation & Consistency
|
||||
|
||||
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
|
||||
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (setup, commands, configuration, features, conventions). This phase's user-visible additions: the `show_text` capability + the `bor` locator syntax in the feature list, the three `BOR_*` settings in the env table, and (in `AGENTS.md` rule 5's UI list) the verbatim-panel contract "an answer's code block is either verbatim-and-badged or plainly the model's — never a silently-substituted hybrid".
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `ask "show me the quadlet for <x>"` on a running app (real or mock LLM) renders a panel whose text is byte-identical to `documents.content`'s resolved slice, with a caption whose numbers came from the server, and a Copy button whose clipboard content equals that slice exactly.
|
||||
- [ ] `uv run python -c "from app.config import get_settings as g; s=g(); print(s.answer_text, s.show_text_max_chars, s.show_text_max_blocks)"` prints `True 12000 5`; each negative value refuses startup naming its field.
|
||||
- [ ] `GET /api/admin/settings` lists `answer_text`, `show_text_max_chars`, `show_text_max_blocks`; `PUT`/`DELETE` round-trip per the overlay contract.
|
||||
- [ ] A grounded turn's `done` frame carries `answer_text[]` with `source,path,title,start_line,end_line,total_lines,text,truncated,locator`; a turn without verbatim content carries NO key; with `BOR_ANSWER_TEXT=false` the frame is byte-identical to pre-phase and only FOUR tools are offered.
|
||||
- [ ] The per-turn log line ends with `verbatim_blocks=N` (`grep` it on a shown turn and on a plain turn).
|
||||
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
|
||||
- [ ] `uv run pytest tests/e2e/test_verbatim_show_text.py -v --no-cov` green **in isolation**; `test_answer_images.py`, `test_harness_aligned_tools.py`, `test_agent_document_tools.py`, `test_markdown_tables.py` (and the phase-180 suite if it landed first) green in isolation.
|
||||
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass).
|
||||
- [ ] no behavior change in completed phases (image answers, `read`/`grep`/`ls`, deflected byte-identity, the document viewer, saved-chat byte-identity).
|
||||
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `feat(agent): display file contents verbatim with a show_text tool and server-resolved locator fences`); the phase dir moves to `.agents/phases/complete/`.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Task 01 — Three settings and the kill switch (inert)
|
||||
|
||||
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
|
||||
|
||||
## Objective
|
||||
|
||||
Add the three `Settings` fields the phase runs on — the master `answer_text` switch and the two budgets — with their admin-panel rows and `.env.example` entries. Nothing reads them yet: after this task the app behaves byte-identically to pre-phase.
|
||||
|
||||
## Work
|
||||
|
||||
1. `app/config.py` — in the agent-tools neighborhood of `read_max_chars` (L296), add:
|
||||
- `answer_text: bool = True` — the master switch for verbatim display (`BOR_ANSWER_TEXT`). Docstring: names the env var, the default `True`, and what `False` means (the `show_text` tool is not offered, the `<tools>` clause is absent, locator resolution does not run, the `done` frame carries no `answer_text` key — every byte-identity contract is exactly pre-phase). Model the docstring on `answer_images` (L539) but note the DEFAULT IS TRUE here (the owner chose the fix on by default; the switch is an escape hatch, not an opt-in).
|
||||
- `show_text_max_chars: int = 12_000` (`BOR_SHOW_TEXT_MAX_CHARS`) — the per-block character budget for one displayed slice; `0` = inert (no block displays), a negative value is a typo and fails startup naming the field (the `_read_max_chars_non_negative` validator at L686 is the exact pattern).
|
||||
- `show_text_max_blocks: int = 5` (`BOR_SHOW_TEXT_MAX_BLOCKS`) — the per-answer cap on displayed blocks; `0` = inert, negative fails startup (the `agent_max_rounds` validator at L678 is the pattern).
|
||||
Add the two `@field_validator`s next to `_read_max_chars_non_negative`, each raising a `ValueError` that names the field (house rule: fail loud, never half-configured).
|
||||
2. `app/core/settings_catalog.py` — three `SettingMeta` rows beside `read_max_chars` (L267): `answer_text` → `"Answer text"`, `show_text_max_chars` → `"Show text max chars"`, `show_text_max_blocks` → `"Show text max blocks"`, all in the `Agent tools` group (the `read_max_chars` group). The completeness pin in `tests/unit/test_settings_catalog.py` (`set(CATALOG) == set(Settings.model_fields)`) is the enforcement — without the rows the build fails.
|
||||
3. `.env.example` — three lines in the agent-tools block, each with a one-line comment: `BOR_ANSWER_TEXT=true`, `BOR_SHOW_TEXT_MAX_CHARS=12000`, `BOR_SHOW_TEXT_MAX_BLOCKS=5`. State the `0`/negative semantics in the comments (the surrounding file's style).
|
||||
4. No consumer changes. Do NOT touch `app/rag/agent.py`, `app/rag/prompts.py`, `app/api/chat.py`, `app/schemas.py` or any frontend file in this task.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **Unit:** `tests/unit/test_config.py` — defaults (`True`, `12000`, `5`); each env override applies; `BOR_SHOW_TEXT_MAX_CHARS=0` and `BOR_SHOW_TEXT_MAX_BLOCKS=0` are accepted (inert, not an error); each negative value raises a validation error whose message names the field (the `read_max_chars` test is the template). `tests/unit/test_settings_catalog.py` — the completeness pin green, the three rows carry the right type/group/label.
|
||||
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` — the three rows appear in `GET /api/admin/settings` (`bool` + two `int`, group `Agent tools`); `PUT` stores each raw value and it resolves live without a restart; `DELETE` resets each to the env default; the `0` values are storable.
|
||||
- **Coverage:** **>90%** on this task's new/modified code (the validators are the only new logic — cover both branches of each).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run python -c "from app.config import get_settings as g; s=g(); print(s.answer_text, s.show_text_max_chars, s.show_text_max_blocks)"` → `True 12000 5`.
|
||||
- [ ] `BOR_SHOW_TEXT_MAX_CHARS=-1 uv run python -c "import app.config"` refuses with a message naming `show_text_max_chars` (same for `show_text_max_blocks`).
|
||||
- [ ] The three rows are visible in the admin Settings panel's `Agent tools` group (or asserted by the integration test if the browser is not running).
|
||||
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
|
||||
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the env table) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] no behavior change in completed work (no consumer reads the new fields yet).
|
||||
@@ -0,0 +1,47 @@
|
||||
# Task 02 — `app/rag/verbatim.py`: the locator grammar, the slicer, the resolver
|
||||
|
||||
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
|
||||
|
||||
## Objective
|
||||
|
||||
The single source of truth for "which exact lines of which indexed document, and what are their bytes". Pure functions for the locator grammar and the slicing, one bare DB lookup, and `resolve_answer_blocks` which the API layer calls at `done` time. Inert: nothing calls it yet (tasks 03 and 05 wire it in).
|
||||
|
||||
## Work
|
||||
|
||||
1. Create `app/rag/verbatim.py`. Module docstring states the contract (PLAN §6's tool surface extension, phase 182 D1–D7): *the model names a location, this module supplies the bytes*; and the import rule — **this module must never import `app.rag.agent`** (agent.py imports the pure half of this module; a reverse import cycles). Allowed imports: `sqlalchemy`, `app.models.Document`, `app.rag.retriever.TRUNCATION_MARKER`, `app.config.Settings`, stdlib.
|
||||
2. `LOCATOR_LANG = "bor"` and `@dataclass(frozen=True) class VerbatimBlock` — `source: str`, `path: str`, `title: str`, `start_line: int`, `end_line: int`, `total_lines: int`, `text: str`, `truncated: bool`, `locator: str` (the canonical form, built by the factory, never passed in by a caller). The dataclass docstring pins: `start_line`/`end_line` are 1-based and INCLUSIVE and describe exactly the lines `text` contains after any cap; `text` never carries a trailing newline (lines are joined with `"\n"`).
|
||||
3. `canonical_locator(raw: str) -> str | None` — the tolerant parser of D7. Accepts, in order: a combined `source/path` (the first `/` splits it, the `read`/`grep` identity contract) followed by an optional separator + range — separators ` `, `:`, `#`, `#L`; ranges `12-40`, `12–40` (en dash), `L12-L40`, or a bare `12` (one line, end == start). Returns `source/path` for a whole document and `source/path#12-40` for a range (always the canonical `#` + plain digits, never the en dash, never the `L`). Returns `None` (NOT a locator) for: an empty/whitespace input, a value with no `/` in the path part, a non-numeric or non-positive bound, `start > end`, trailing junk after the range, or a leading `bor` token left in by the caller. No DB, no normalization of case (paths are case-sensitive row values).
|
||||
4. `fences_in(answer: str) -> list[str]` — every `bor` fence's RAW locator string, in document order, deduped keeping the first occurrence. Rules: the opening line is a fence whose info string is exactly `` ```bor `` + whitespace + a non-empty remainder (a bare `` ```bor `` with no path is NOT a locator — it is an ordinary code block and must not be swallowed); the fence closes at the next line that is `` ``` `` (trimmed); an UNCLOSED fence contributes nothing (defensive — a half-streamed answer); the body is read and DISCARDED (this module never looks at what the model typed); nested `` ``` `` inside the body is not supported and is a documented limitation (the fence ends at the first closer, the same rule `markdown.js` applies).
|
||||
5. `@dataclass(frozen=True) class SliceResult` — `text: str`, `first_line: int`, `last_line: int`, `total_lines: int`, `past_end: bool` — and `slice_lines(content: str, offset: int | None = None, limit: int | None = None) -> SliceResult`. Indexing is `content.splitlines()` with the SAME 1-based semantics as `read` (`app/rag/agent.py` L1836) and `grep`: `offset` omitted → 1; `limit` omitted → to the end; `end = min(offset + limit - 1, total_lines)` (an overshooting `limit` CLAMPS); `offset > total_lines` → `past_end=True` with empty text and the true `total_lines` (the caller builds the teaching refusal); `total_lines == 0` (an empty document) → `past_end=True`. Never raises for a range problem — the refusal is the caller's, the total always travels with the result.
|
||||
6. `cap_slice(text: str, first_line: int, cap: int) -> tuple[str, int, bool]` — the character budget applied at a LINE boundary, returning `(text_to_show, last_line_shown, truncated)`. `cap <= 0` raises `ValueError` (a mis-wired caller must be loud — the `show_text_max_chars == 0` kill switch is checked by `resolve_answer_blocks`, never by reaching here). When `len(text) <= cap`: `(text, last_line_of(text), False)`. Otherwise cut to the last whole line that fits within `cap` and return `(cut + "\n" + TRUNCATION_MARKER, first_line + n_lines_in_cut - 1, True)`; a first line LONGER than `cap` cuts that one line mid-way (the only sub-line cut, documented) and still ends with the marker. The marker is the shared `app.rag.retriever.TRUNCATION_MARKER` — never a second copy of the text.
|
||||
7. `document_by_identity(db: Session, combined: str) -> Document | None` — split at the FIRST `/` (source names are directory basenames and never contain one); `select(Document).where(Document.source == source, Document.path == path)` → the row or `None`. Docstring: this is the same `(source, path)` identity contract as `app.rag.agent._resolve_path` REIMPLEMENTED as a bare lookup — no did-you-mean suggestions here (the tool branch already has `_resolve_path` for teaching; the fence resolver only needs yes/no), and the duplication exists to keep the import direction one-way. A traversal string is simply a row that does not exist.
|
||||
8. `block_for(db, combined: str, *, offset: int | None = None, limit: int | None = None, cap: int, doc: Document | None = None) -> VerbatimBlock | None` — resolve (or take the caller's already-resolved `doc`), refuse `None` for: no document, `doc.is_image` (image display belongs to `show_image`, locked A6), an empty/past-end slice; otherwise `slice_lines` → `cap_slice` → the block with `locator = f"{source}/{path}"` (+ `f"#{first}-{last}"` when the request named a range or the cap cut it — the canonical form task 07 matches on). `title` is the row's title.
|
||||
9. `resolve_answer_blocks(db, answer: str, shown: Sequence[VerbatimBlock], settings: Settings) -> list[VerbatimBlock]`:
|
||||
- returns `[]` when `not settings.answer_text`, or `settings.show_text_max_blocks == 0`, or `settings.show_text_max_chars == 0` (the kill switches, checked HERE so no caller has to remember);
|
||||
- for each `fences_in(answer)` raw string in order: `canonical_locator` → skip when `None`; `block_for(...)` → skip when `None` (unknown path, image doc, past end) — the caller sees simply "not among the blocks", which is what task 07 renders as the unresolved state;
|
||||
- dedupe by canonical locator, the FENCE winning over a same-locator entry from *`shown`* (the fence controls placement);
|
||||
- append the `shown` blocks the answer never located, in their original call order;
|
||||
- truncate the list to `settings.show_text_max_blocks`.
|
||||
Docstring pins the ordering rule (answer order first, then the unlocated tool blocks) — task 07's DOM order and the E2E assertions depend on it.
|
||||
10. Update the module-level `__all__` and keep every public name docstring'd in the house style (contract + phase reference).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **Unit:** create `tests/unit/test_verbatim_resolver.py`.
|
||||
- `canonical_locator`: `homelab/a.md` → itself; `homelab/a.md 12-40` / `:12-40` / `#12-40` / `#L12-L40` / `12–40` (en dash) → `homelab/a.md#12-40`; `homelab/a.md 12` → `#12-12`; `None` for `a.md` (no source), `""`, `" "`, `homelab/a.md 0-5`, `homelab/a.md 40-12`, `homelab/a.md x-y`, `homelab/a.md 12-40 extra`, `"bor homelab/a.md"`.
|
||||
- `slice_lines`: whole file, offset-only, limit-only, both, overshooting limit clamps `last_line` to the total, `offset` past the end → `past_end` + true total, empty content → `past_end`, and the `splitlines()` agreement check (a fixture whose line `N` is asserted by content, mirroring what `grep` would report).
|
||||
- `cap_slice`: under cap (untouched, `truncated=False`), cut at a line boundary (text ends with the shared `TRUNCATION_MARKER`, `last` = the last whole line shown, and the marker line is not counted as content), a single over-long first line, `cap=0` raises `ValueError`.
|
||||
- `document_by_identity`: hit, miss, traversal-shaped miss, first-slash splitting when the path itself contains slashes.
|
||||
- `block_for`: whole-document block, ranged block, canonical locator with/without a range, image doc → `None`, unknown path → `None`, past-end → `None`, capped block's `end_line` equals the lines actually shown.
|
||||
- `resolve_answer_blocks`: no fences + no shown → `[]`; fence-only (no tool call) resolves; tool-only appends; fence + identical shown block → ONE block (fence wins, appears at the fence position); two fences keep answer order; `show_text_max_blocks=1` truncates; every kill-switch combination returns `[]`; an unresolvable fence is simply absent; a duplicated fence appears once.
|
||||
- Session-fixture style follows the existing agent/resolver unit suites (the same in-memory/session fixture the `tests/unit/test_agent.py` helpers use).
|
||||
- **Coverage:** **>90%** on this task's new/modified code (every branch above is a test).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/unit/test_verbatim_resolver.py -v` green.
|
||||
- [ ] `uv run python -c "import app.rag.verbatim as v; print(v.canonical_locator('homelab/a.md#L12-L40'))"` prints `homelab/a.md#12-40`.
|
||||
- [ ] `grep -n "import" app/rag/verbatim.py` shows NO import of `app.rag.agent` (the one-way rule).
|
||||
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
|
||||
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (nothing user-visible yet — the phase's final pass re-checks both project-wide).
|
||||
- [ ] no behavior change in completed work (nothing calls this module yet).
|
||||
@@ -0,0 +1,57 @@
|
||||
# Task 03 — The `show_text` agent tool: name a location, the server shows the bytes
|
||||
|
||||
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
|
||||
|
||||
## Objective
|
||||
|
||||
The 5th agent tool: `show_text(path, offset?, limit?)` displays an indexed document (or a line range of one) to the user verbatim. The handler resolves the slice through task 02, records it for the `done` frame, and hands the SAME bytes back to the model with an explicit "do not retype" instruction.
|
||||
|
||||
## Work
|
||||
|
||||
1. `app/rag/agent.py` — add `SHOW_TEXT_TOOL` right after `SHOW_IMAGE_TOOL` (L501), same OpenAI function shape:
|
||||
- `name: "show_text"`.
|
||||
- `description` (this IS the behavior contract — the phase-124 discipline): "Display part of a document to the user EXACTLY as stored — the user sees the server's bytes and can copy them. Call it when the user wants to see, copy or run a file (a quadlet, a config, a script): they copy from what is displayed, so do NOT retype the content yourself after calling it — describe it instead. `path` is the combined `source/path` string exactly as shown in `ls`. `offset` (1-based first line) and `limit` (line count) are optional — omit them to display the whole document; use them when `grep`/`read` told you which lines matter. The displayed line numbers come from the server, not from your arguments: an overshooting `limit` is clamped and an `offset` past the end is refused with the true line count. Only text documents — for an image use `show_image`. Call one tool at a time — wait for this result before your next call."
|
||||
- `parameters`: `path` (string, required, the `read` "a bare document path will not resolve" teaching), `offset` and `limit` (integers, optional, the `read` wording reused).
|
||||
2. `agent_tools(settings)` (L538) — append `SHOW_TEXT_TOOL` when `settings.answer_text`, keeping `AGENT_TOOLS` byte-identical and the list fresh per call (the `show_image` precedent; the offer order is `ls`, `read`, `grep`, `show_image`, `show_text`). Update the docstring (four base + two display tools, each gated by its own flag).
|
||||
3. `AgentHolder` (L1600–L1680) — add `shown_texts: list[VerbatimBlock] = field(default_factory=list)` with a docstring paragraph in the `shown_images` style: the blocks in CALL order, capped by `settings.show_text_max_blocks`, read by `app.api.chat` after the stream for the `done` frame's `answer_text` (the `shown_images`/`read_docs` read-after-stream precedent); refused shows append nothing.
|
||||
4. Refusal constants beside the `show_image` ones (L571–L590), each single-braced and pinned in the house style:
|
||||
- `MISSING_SHOW_TEXT_ARGS = "show_text requires a string argument 'path'."`
|
||||
- `SHOW_TEXT_NOT_A_TEXT_DOC` — one `{arg}` field: the doc is a standalone image; `show_image` displays images. (LOCKED A6 — never silently converted.)
|
||||
- `SHOW_TEXT_BLOCK_CAP` — one `{max}` field: that many blocks are already displayed this answer — describe them instead of showing more.
|
||||
- `SHOW_TEXT_ALREADY_SHOWN` — one `{locator}` field: that exact range is already on screen — say so instead of showing it again.
|
||||
Reuse `BAD_RANGE_ARGS` (L628) and `OFFSET_PAST_END` (L671) verbatim for the range problems (the same teaching, the same echoed values — one copy of each string in the codebase).
|
||||
5. `_execute_tool` (L1681) — a `show_text` branch placed next to the `show_image` branch (L2023), in this order:
|
||||
1. `raw_path = call.arguments.get("path")`; non-string/blank → `MISSING_SHOW_ARGS`-shaped refusal (`MISSING_SHOW_TEXT_ARGS`).
|
||||
2. `settings.answer_text` false (defensive — the tool was not offered) → the same refusal copy as the round-cap-off posture: state that the tool is unavailable; counts in nothing.
|
||||
3. `doc, _source, _path = _resolve_path(db, arg)`; `doc is None` → `_no_document_refusal(db, arg)` (the phase-72 did-you-mean teaching, reused unchanged).
|
||||
4. `doc.is_image` → `SHOW_TEXT_NOT_A_TEXT_DOC`.
|
||||
5. `ok, err = _read_range_args(offset, limit)` — reuse the existing validator unchanged; `not ok` → `err`.
|
||||
6. `len(holder.shown_texts) >= settings.show_text_max_blocks` → `SHOW_TEXT_BLOCK_CAP` (checked BEFORE the work; `0` therefore disables the tool's effect while keeping it offered — pinned in the tests).
|
||||
7. `block_for(db, arg, offset=offset, limit=limit, cap=settings.show_text_max_chars, doc=doc)` → `None` (empty document / `offset` past the end) → `OFFSET_PAST_END.format(total_lines=...)` (compute the total with `len(doc.content.splitlines())`, the same expression `read` uses); a returned block whose canonical `locator` already appears in `holder.shown_texts` → `SHOW_TEXT_ALREADY_SHOWN`.
|
||||
8. Success: `holder.shown_texts.append(block)`; `holder.tool_calls += 1` (executed-only counting, the house rule); return the result string in this EXACT shape (the `read`/`show_image` header contract the E2E mock's `_READ_RESULT_PREFIX` parses):
|
||||
```
|
||||
Text shown to the user — these lines are displayed verbatim; describe them, do NOT retype them.
|
||||
Document {source}/{path}:
|
||||
date: {YYYY-MM-DD}
|
||||
{origin lines — the same _origin_lines(origin) helper the read branch uses}
|
||||
lines: {first}-{last} of {total_lines}
|
||||
{block.text}
|
||||
```
|
||||
`block.text` already ends with `TRUNCATION_MARKER` when the char cap cut it; when `block.truncated` is true append one line, in the `READ_TRUNCATION_NOTICE` spirit, naming that the rest is NOT displayed and that the user can open the document (no `offset`/`limit` re-show loop is implied — the full text is a click away, and only the displayed slice is what the user copies).
|
||||
- Every refusal path appends nothing, increments nothing, and consumes a round exactly like the existing refusals.
|
||||
6. Module docstring — extend the numbered loop contract (the `show_image` paragraph's neighborhood) with the `show_text` contract and the locked rules A1–A3/A6: the model names a location, the server supplies the bytes, the displayed extent is the server's, images belong to `show_image`, the fence form is resolved at the API layer (task 05), not here.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **Unit (`tests/unit/test_agent.py`, the `show_image` family is the template):** `agent_tools` offers 4 tools with `answer_text=False` (byte-identical to pre-phase, `show_image` last) and 5 with it on (`show_text` present, correct order); a successful whole-document show (result headers in order, `lines: 1-N of N`, the slice verbatim, `shown_texts` length 1, `tool_calls` 1, `read_docs` unchanged); a ranged show (clamped `limit` reflected in the `lines:` header); a capped show (`TRUNCATION_MARKER` present + the extra notice line); every refusal — missing `path`, non-string `path`, unknown path (did-you-mean copy, no `shown_texts`), image doc, bad range args, `offset` past the end (the true total named), block cap (`max_blocks=1`, second call refused), duplicate identical show; each refusal asserts `tool_calls` and `shown_texts` untouched; `max_blocks=0` → the first call is refused with the cap copy.
|
||||
- **Integration:** none required in this task (the frame wiring is task 05) — but `tests/integration/` must stay green, and any test that pins the exact tool-offer array is updated here, deliberately, with a comment.
|
||||
- **Coverage:** **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/unit/test_agent.py -v -k show_text` green, including all nine refusal cases and the counting assertions.
|
||||
- [ ] With `answer_text=True`, a scripted agent turn returns a result whose 2nd line starts `Document ` and whose `lines:` header matches the slice actually returned.
|
||||
- [ ] `grep -n "SHOW_TEXT_" app/rag/agent.py` shows the four new constants, each used by exactly one branch.
|
||||
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
|
||||
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the tool surface list in README's agent-tools section) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] no behavior change in completed work (`ls`/`read`/`grep`/`show_image` results byte-identical; the pre-phase four-tool offer byte-identical when the flag is off).
|
||||
@@ -0,0 +1,36 @@
|
||||
# Task 04 — Teach the model to point instead of type: the `<tools>` clause
|
||||
|
||||
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
|
||||
|
||||
## Objective
|
||||
|
||||
The HIGH prompt teaches two things it has never said: **the user copies from what is displayed, not from your prose**, and the exact ` ```bor ` locator-fence syntax for placing a displayed block. Gated by `settings.answer_text`, byte-identical when off.
|
||||
|
||||
## Work
|
||||
|
||||
1. `app/rag/prompts.py` — add `SHOW_TEXT_CLAUSE: str` right after `SHOW_IMAGE_CLAUSE` (L379), with a docstring comment in the same style (why it exists, what each sentence pins, and that `TOOLS_SECTION` stays byte-identical). The clause text must contain, in this order:
|
||||
- the display contract: "`show_text` displays part of a document to the user EXACTLY as stored — the user copies from what is displayed, not from your prose.";
|
||||
- when to call it: "Call it whenever the user wants to SEE, COPY or RUN a file — a quadlet, a config, a script, a unit file — instead of writing the content into your answer yourself.";
|
||||
- the do-not-retype rule (the phase's reason for existing): "After calling it, do NOT repeat the displayed lines in your answer — describe them, explain them, point at them. Anything you type yourself may differ from the file by a character, and the user cannot tell.";
|
||||
- the placement syntax: "To put a displayed block exactly where you want it in your answer, write a locator line on its own: three backticks followed by `bor`, a space, the combined `source/path`, and optionally the line range (` 12-40`). Leave its body EMPTY — the app replaces it with the exact stored text. Never type the file's content inside a `bor` block.";
|
||||
- the honesty-about-numbers rule: "Never guess line numbers — use the numbers `grep` and `read` reported. When you are not sure which lines, OMIT the range and display the whole document: the numbers you display are the server's, never yours.";
|
||||
- the failure rule: "If `show_text` refuses, say what you could not display — do not reconstruct it from memory.";
|
||||
- the image pointer: "For an image use `show_image` instead."
|
||||
2. `build_high_prompt(...)` (L759) — add `show_text: bool = False` after `show_image`; when true, the clause rides INSIDE `<tools>` AFTER the `SHOW_IMAGE_CLAUSE` (when that is present) and before the closing `</tools>` tag, using the existing "section text minus the final `</tools>` + clause + closing tag" mechanism (L782–L791 documents it). Both clauses compose independently: `show_image` only → exactly the phase-124 output; `show_text` only → the three-tool copy + the text clause; both → image clause then text clause; neither → byte-identical to pre-phase. Update the docstring (a *show_text* paragraph mirroring the *show_image* one, incl. the note that the deflection prompt NEVER carries it — a deflected turn has no tools and D9 forbids resolving locators on it).
|
||||
3. `app/api/chat.py` (the grounded `TurnPlan` builder, L516–L529) — pass `show_text=settings.answer_text` next to `show_image=settings.answer_images`, with the same style of phase comment.
|
||||
4. Do NOT change `TOOLS_SECTION`, `build_deflect_prompt`, or any mock in `tests/e2e/mock_llm.py` (the new scripted triggers belong to task 08).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **Unit (`tests/unit/test_prompts.py` + the `tests/unit/test_prompt_lock.py` pins):** the off-build (`show_text=False`, the default) is byte-identical to the pre-phase string for every combination of the other arguments; the on-build contains the clause INSIDE `<tools>` (index of the clause text > index of `<tools>`, `< ` index of `</tools>`); `show_image=True, show_text=True` yields the image clause before the text clause; `TOOLS_SECTION` itself unchanged (the existing lock test still passes untouched); the clause names `show_text`, `bor`, the do-not-retype rule and the "omit the range" rule (the E2E and the persona depend on those exact ideas); the deflect prompt never contains `show_text` or `bor`.
|
||||
- **Integration:** the grounded-turn integration tests stay green (the prompt string is not pinned byte-for-byte there beyond the `<tools>` marker); add one assertion that a grounded plan built with `answer_text=True` carries the clause and one that `answer_text=False` does not.
|
||||
- **Coverage:** **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_prompt_lock.py -v` green.
|
||||
- [ ] `uv run python -c "from app.rag.prompts import build_high_prompt as b; print('show_text' in b([], show_text=True))"` prints `True`, and the same call without the flag prints `False`.
|
||||
- [ ] A grounded request against a running app shows the clause in the model request (log/assert via the E2E mock's request capture in task 08 — here, the unit proof is enough).
|
||||
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
|
||||
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the prompt/tool teaching summary) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] no behavior change in completed phases (deflected prompt byte-identical; the phase-124 clause position untouched).
|
||||
@@ -0,0 +1,34 @@
|
||||
# Task 05 — The `done` frame's `answer_text`, the visible-text accumulator, the log field
|
||||
|
||||
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
|
||||
|
||||
## Objective
|
||||
|
||||
Put the resolved bytes on the wire and in the persisted record: the turn's clean visible answer text is accumulated during streaming, resolved through `app.rag.verbatim` at `done`-build time, and sent as the ADDITIVE `answer_text` array; the per-turn log line gains `verbatim_blocks=N`.
|
||||
|
||||
## Work
|
||||
|
||||
1. `app/schemas.py` — `class VerbatimBlockRef(BaseModel)` beside `SourceRef` (L169): `source: str = Field(max_length=120)`, `path: str = Field(max_length=1000)`, `title: str = Field(max_length=500)` (the `SourceRef` caps, the same identity), `start_line: int = Field(ge=0)`, `end_line: int = Field(ge=0)`, `total_lines: int = Field(ge=0)`, `text: str = Field(max_length=600_000)` (the phase-83 answer-text cap — generous above the 12 000 default, so an operator raising `BOR_SHOW_TEXT_MAX_CHARS` does not hit the wire limit), `truncated: bool = False`, `locator: str = Field(max_length=2100)`. Docstring: the phase-182 display block; `text` is SERVER-supplied bytes from `documents.content`, never model output; 1-based inclusive lines.
|
||||
2. `ChatDoneEvent` (L305) — `answer_text: list[VerbatimBlockRef] | None = None` + docstring in the `answer_image` style, and extend its `model_serializer` to POP the key when `None` **or empty** (absent, never `null`, never `[]`: every frame without blocks stays byte-identical to pre-phase — the pin that proves it lives in the integration suite). Note the two deliberate differences from `answer_image` in the docstring: it is a LIST (several blocks per answer, D1), and it carries the TEXT itself (A4 — no bytes route, no new anonymous surface, a shared page renders from its own record).
|
||||
3. `ChatMessageRecord` (L1400–L1510) — `answer_text: list[VerbatimBlockRef] | None = None` on the BRAIN record, popped when `None`/empty in the existing `_drop_image_when_absent` serializer (extend its docstring). Required so `extra="forbid"` does not 422 the done-time auto-save of a turn with blocks (the phase-113 lesson, applied up front) and so the restore path + the shared page re-render the panels. No migration (`saved_chats.messages` is raw JSONB, the phase-50/160 precedent).
|
||||
4. `app/api/chat.py`:
|
||||
1. Next to the visible-content counter (`content_chars`, L934) add `visible_parts: list[str]` and append EVERY visible (post-scaffold-filter) content piece in the piece loop — the same pieces counted into `content_chars`, never `thinking` (D3: locators only ever come from the answer). Bounded by the model's output-token budget; note that in the comment.
|
||||
2. In the `done` builder (L1320–L1365), after `answer_ref`: `blocks = resolve_answer_blocks(db, "".join(visible_parts), holder.shown_texts, settings) if not plan.deflected else []` (D9 — a deflected turn resolves NOTHING: no fence leaks KB text through a turn that just disclaimed it; the deflected path has no `holder`, so the guard is load-bearing). Convert with `VerbatimBlockRef(**dataclasses.asdict(b))` in order, pass `answer_text=blocks or None`.
|
||||
3. Per-turn log line (L1272–L1291): append `verbatim_blocks=%d` after `scaffold_stripped=%d` (the count of blocks SENT, i.e. after dedupe + `show_text_max_blocks`; `0` on every turn without them, including deflected ones) and the module docstring's line-format block with the new field. This is the §9 additive revision — update `tests/unit/test_chat_gate.py`'s pinned line in the same commit, never silently.
|
||||
4. `holder.shown_texts` is read AFTER the stream (the `shown_images` precedent) — do not reorder the generator's finally/cancel semantics (a cancelled turn still writes neither line nor `query_log` row).
|
||||
5. Nothing in `app/schemas.py`'s existing validators changes; no new SSE event type (A15 stays at seven).
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **Unit:** `tests/unit/test_chat_gate.py` (or the module that owns the per-turn line pins) — the line ends `… scaffold_stripped=N verbatim_blocks=0` on a plain grounded turn, `verbatim_blocks=2` on a turn whose holder has two blocks, `verbatim_blocks=0` on a deflected turn (and the deflected frame still carries no `answer_text`).
|
||||
- **Integration:** the chat API/streaming suite — a scripted grounded turn that calls `show_text` yields a `done` frame whose `answer_text[0]` has the SERVER's `text`, the clamped extent, the canonical `locator`, and `truncated` false under the cap; a turn whose ANSWER contains a ` ```bor ` fence (no tool call) resolves the same block from the fence; fence + identical tool call produce exactly ONE entry; three blocks with `show_text_max_blocks=2` send two in answer order; a turn with no blocks serializes WITHOUT the `answer_text` key (byte-identity pin, exact-dict compare); `answer_text=False` yields the pre-phase frame byte-for-byte AND `verbatim_blocks=0`; a deflected frame carries no `answer_text` even when its (mocked) text contains a fence. The saved-chat suite — `POST /api/chats` with a brain record carrying `answer_text` is accepted (`extra="forbid"`), `GET /api/chats/{id}` returns it unchanged, and a record without it round-trips byte-identically to pre-phase.
|
||||
- **Coverage:** **>90%** on this task's new/modified code.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] A scripted SSE turn with a `show_text` call returns `answer_text: [{source,path,title,start_line,end_line,total_lines,text,truncated,locator}]` and the text equals `documents.content` for that range exactly.
|
||||
- [ ] `grep ' :: api ' ` on that request shows the `done`-bearing response's byte line (the phase-167 audit) — record the observed size in the phase's task notes if it approaches the 256 KB budget.
|
||||
- [ ] The per-turn log line shows `verbatim_blocks=N`; a plain turn shows `0`.
|
||||
- [ ] `uv run pytest` green (incl. the updated line pins), coverage >90%, ruff + pyright clean.
|
||||
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the SSE `done` frame contract in README's API section; AGENTS rule 10's log-line list) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] no behavior change in completed phases (`answer_image`, `sources`, `related`, `budget_exhausted`, the deflected byte-identity, the cancelled-turn contract).
|
||||
@@ -0,0 +1,43 @@
|
||||
# Task 06 — `markdown.js`: the opt-in `bor` locator pass (model bytes never painted) + the panel CSS
|
||||
|
||||
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
|
||||
|
||||
## Objective
|
||||
|
||||
Teach the shared renderer to recognize a ` ```bor ` locator fence and replace it — body and all — with a placeholder node, and write the panel's CSS. Inert: the pass is opt-in through a second argument that NO call site passes yet (task 07 opts the answer renderers in), so every existing render path stays byte-identical.
|
||||
|
||||
## Work
|
||||
|
||||
1. `frontend/assets/markdown.js` — `renderMarkdown(md)` becomes `renderMarkdown(md, opts)` (a SECOND, OPTIONAL argument; `opts && opts.verbatim === true` is the only truthy path). Header comment: the phase-182 pass, why the model's body is discarded, and the byte-identical-when-off contract.
|
||||
2. Add the **verbatim pass BEFORE the existing fence pass** (order: verbatim → fences → tables → escape+inline → blockquotes → paragraphs → restore):
|
||||
- it matches a fence whose info string is exactly `bor` + whitespace + a non-empty remainder: `` /^```bor[ \t]+\S[^\n]*\n(?:[\s\S]*?^```[ \t]*$|(?![\s\S]))/gm `` — the second alternative matches a fence left OPEN at the very end of the input, so a half-streamed locator shows the pending placeholder instead of raw backticks (a bare ` ```bor ` with no path still does NOT match: it is an ordinary code block and must survive to the fence pass untouched); note the deliberate asymmetry with the server's stricter `fences_in` (task 02 ignores unclosed fences — the server only ever sees a finished answer, the client sees a partial one);
|
||||
- the matched locator text is run through the new local `borLocator(raw)` normalizer; if it returns `""` (unparseable), the match is put BACK byte-for-byte and left to the existing fence pass — a malformed locator degrades to whatever that pass does with this text today (no placeholder, no badge, the model's own body): this phase neither improves nor worsens it;
|
||||
- a parsed one is replaced with `"\u0000VERBATIM" + n + "\u0000"` and recorded, and restored in the final restore step as:
|
||||
`<div class="verbatim verbatim-pending" data-bor-locator="<escaped canonical>"><span class="verbatim-caption"><escaped source/path></span><span class="verbatim-state" role="status">Retrieving the exact text…</span></div>`
|
||||
— every interpolated value goes through the existing `escapeHtml`; the model's body appears NOWHERE in the output (the phase's core contract, A7).
|
||||
3. `borLocator(raw)` (module-local, docstring'd): trims; splits the path at the FIRST `/` (no `/` → invalid); accepts the separators ` `, `:`, `#` and `#L` before the range and the ranges `12-40`, `12–40` (en dash), `L12-L40`, `12`; returns `source/path` or `source/path#12-40` (canonical digits only), `""` for anything else (non-numeric, `< 1`, `start > end`, trailing junk). Documented as the CLIENT MIRROR of Python `app.rag.verbatim.canonical_locator` — the parity is pinned by the phase's E2E (both sides must agree or hydration misses), not by a unit test.
|
||||
4. Do NOT wrap ordinary fences in markdown: task 07 wraps `<pre>` nodes at the DOM level (so the document viewer, the thinking blocks and every other caller keep byte-identical HTML).
|
||||
5. `frontend/assets/styles.css` — a clearly-labelled phase-182 block. Palette variables ONLY (`--surface`, `--line`, `--ink`, `--ink-soft`, `--accent-*`, `--ok-*`; the phase-92 zero-literal rule — a monochrome theme must keep every state readable, WCAG "text + color, never color alone"):
|
||||
- `.verbatim` — block, `margin` in the bubble's rhythm, `border: 1px solid var(--line)`, `border-radius: var(--radius-sm)`, `background: var(--surface)`, `min-width: 0`, `overflow: hidden`.
|
||||
- `.verbatim-head` (row) + `.verbatim-caption` — the `source/path · lines 12–40 of 210` line, mono, `--ink-soft` (≥4.5:1), clipped with the existing `.cell-clip` ellipsis convention (phase 156) while the control's `aria-label`/`title` keeps the full text; `.verbatim-head` STICKS to the top of the scroll panel (`position: sticky; top: 0` + a matching background — PLAN §7.1's sticky-header rule for bounded panels).
|
||||
- `.verbatim-badge` — a small TEXT badge reading `VERBATIM` (text, never an icon alone), `--ok` family, `text-transform: none` (no uppercase surprises for screen readers), and a `title` explaining "these bytes are copied from the knowledge base, not written by the model".
|
||||
- `.verbatim-body` — the bounded scroll panel: `max-height: min(40rem, 70vh)`, `overflow: auto`, monospace, `white-space: pre`, `tab-size: 2`, `overflow-wrap: normal` (code scrolls horizontally, prose never does — phase 165's `word-wrap` split), `min-width: 0`.
|
||||
- `.verbatim-lines` — the line-number gutter (`<ol>`/`<li>` or a `data-line` column, the implementer's choice, but numbers must be `user-select: none` so a Copy/select never drags line numbers into the clipboard), right-aligned, `--ink-soft`, `position: sticky; left: 0` optional.
|
||||
- `.verbatim-pending` + `.verbatim-state` — the streaming/pending look (muted, no animation beyond what `prefers-reduced-motion` allows); `.verbatim-unresolved` — the honest failure state, `--err`/`--ink-soft`, no `VERBATIM` badge.
|
||||
- `.code-block` (the task-07 DOM wrapper) + `.code-block-copy` / `.verbatim-copy` — a small button, `min-height: 44px` target on `@media (max-width: 640px)`, `:focus-visible` 3px ring like the rest of the app, `text-overflow: ellipsis` on its label, and a `.copied` state that changes TEXT (`Copy` → `Copied`), not color alone.
|
||||
- The panel is a code surface, so it must stay readable in a light theme too: no hard-coded `#fff`/`#000`, no new hue.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **Unit (`tests/unit/test_frontend_verbatim_markdown.js` is not possible — use the project's text-pin convention, `tests/unit/test_frontend_verbatim_markdown.py`):** the `renderMarkdown(md, opts)` signature and the `opts.verbatim` guard exist; the verbatim pass runs BEFORE the fence pass; `\u0000VERBATIM` placeholders + the restore step exist; the output template contains `verbatim-pending`, `data-bor-locator`, `verbatim-caption`, `role="status"` and the pinned pending copy `Retrieving the exact text…`; `borLocator` exists and the separator/range tokens it handles are all present in its body; the ordinary-fence `<pre><code>` output string is unchanged in the source.
|
||||
- **Unit (CSS, `tests/unit/test_frontend_verbatim_css.py`, the `test_frontend_cluster_css.py` pattern):** the phase-182 selectors exist; `.verbatim-body` has a `max-height` in the `min(…, 70vh)` form and `overflow: auto`; `.verbatim-head` is `position: sticky`; line numbers are `user-select: none`; no new hex/rgb literals in the phase-182 block (parse the block and assert every color is a `var(--…)`); the mobile block keeps a ≥44px control height; `prefers-reduced-motion` is honored by anything that animates.
|
||||
- **Coverage:** **>90%** on the task's new/modified code (Python-side: the pin modules execute the assertions; there is no new `app/` code in this task, so `app/` coverage is unchanged — it must stay >90%).
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] `uv run pytest tests/unit/test_frontend_verbatim_markdown.py tests/unit/test_frontend_verbatim_css.py -v` green.
|
||||
- [ ] Serving the app and calling `renderMarkdown("```bor homelab/a.md 12-40\nGARBAGE\n```", { verbatim: true })` in the browser console yields the placeholder DOM with `data-bor-locator="homelab/a.md#12-40"` and NO `GARBAGE` anywhere in the returned HTML; the same call WITHOUT the second argument returns the ordinary `<pre><code>` block.
|
||||
- [ ] `renderMarkdown` output for the phase-44 table fixtures, the phase-138 blockquote fixtures and the phase-180 image fixtures is unchanged (the existing suites stay green).
|
||||
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
|
||||
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the renderer pass list in README's frontend section) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] no behavior change in completed work (document viewer, thinking blocks, user bubbles, tables, blockquotes, inline images).
|
||||
@@ -0,0 +1,43 @@
|
||||
# Task 07 — Hydration, panels, and a Copy button on every answer code block
|
||||
|
||||
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
|
||||
|
||||
## Objective
|
||||
|
||||
Wire the client: resolve each placeholder against the frame's `answer_text`, render the panel (caption, `VERBATIM` badge, line numbers, Copy, viewer link), persist the blocks with the turn, and give every settled answer code block a Copy button. Live turn, restore-after-reload and the anonymous shared page all render identically.
|
||||
|
||||
## Work
|
||||
|
||||
1. Create `frontend/assets/verbatim.js` (classic script, like `markdown.js`) and include it AFTER `markdown.js` in `frontend/index.html` (L1696–L1697 block) and `frontend/shared.html` (L197–L198 block) — plain relative/absolute `src` exactly as the neighbors, the `?v=` token rewrite is the caching middleware's job (A19). Expose globals (the same pattern `markdown.js` uses), all DOM built with `createElement` + `textContent` (the frontend house rule — never an HTML string), all handlers via `addEventListener` (CSP `default-src 'self'`, no inline handlers):
|
||||
- NO locator normalizer here — `markdown.js::borLocator` is the single implementation (task 06), and hydration compares the server's canonical `locator` against the already-normalized `data-bor-locator` attribute with a plain string compare. Re-implementing the grammar in a second file is the phase's main drift risk; do not.
|
||||
- `buildVerbatimPanel(block, opts)` → the `.verbatim` node: `.verbatim-head` with `.verbatim-caption` (`source/path · lines 12–40 of 210`, or `source/path · all 210 lines` when the block is the whole document), `.verbatim-badge` (`VERBATIM`, `title` = "these bytes are copied from the knowledge base, not written by the model"), `.verbatim-copy` (`Copy`, `aria-label` = `Copy N lines from <source/path>` — the phase-156 clipped-label rule), an optional `.verbatim-open` link to the document (`opts.onOpen` hook: `app.js` passes the modal opener built from `documentUrl` + `openDocumentModal` (back → the chat, phase 153); `shared.js` passes a plain `/document?source=&path=` href — the shared page has no modal); `.verbatim-body` with the line-number gutter (`user-select: none`) and the text lines; the root gets `role="group"` + `aria-label` = `Verbatim text from <source/path><, lines N–M>`, and `data-bor-locator`. A truncated block (`block.truncated`) appends one muted line naming that the rest is in the document, plus the same open link. The git ORIGIN link stays the document viewer's job (phase 145) — the panel links to the viewer, it does not duplicate the origin.
|
||||
- `hydrateVerbatim(root, blocks)` — for every `.verbatim[data-bor-locator]` in *root*: exact canonical-locator match in *blocks* → replace with the panel; else exactly ONE block sharing the same `source/path` → use it (the model's range was wrong; the panel shows the SERVER's extent — this is A2 made visible); else → the unresolved state: `.verbatim verbatim-unresolved` with the text `I could not verify this against the knowledge base — open <source/path>` (the link only when the path itself parsed) and NO badge, NO body, and none of the model's bytes. Never throws on a malformed/missing `blocks`.
|
||||
- `appendUnlocatedVerbatim(wrap, blocks)` — the blocks no placeholder matched, appended in order at the `answer_image` position (below the answer text, above the source clusters, the phase-124/130 DOM order).
|
||||
- `settleVerbatimPlaceholders(root, kind)` — turns leftover `.verbatim-pending` into the unresolved state (used by the error/stop paths so no panel ever sits at "Retrieving…" after a terminal frame).
|
||||
- `enhanceAnswerCodeBlocks(root)` — every `<pre>` inside an answer bubble that is NOT inside a `.verbatim`: wrap in `.code-block` (once — idempotent on re-render) and append `.code-block-copy` with the same clipboard path. NO badge, NO caption — a `VERBATIM` badge belongs only to a server-resolved block (locked A5).
|
||||
- `copyTextWithFallback(text, statusEl)` — the phase-55 degradation generalized from `app.js::copyShareLinkWithFallback` (L3250): try `navigator.clipboard.writeText`, on rejection render/return the inline select-to-copy fallback; returns `true` when the clipboard took it. `app.js`'s share path then DELEGATES to it (single implementation) — the owner-locked copy strings stay exactly as they are, `test_history_copy.py` / the share suites are the guard.
|
||||
2. `frontend/assets/app.js`:
|
||||
- the answer renderers pass the opt-in: the per-delta `renderMarkdown(acc)` (L4322), the `done`-time re-render (L4371) and the brain restore renderer (L2909) become `renderMarkdown(text, { verbatim: true })`. The USER bubble renderers (L1272, L1347, L2890) and every thinking-block render (L2841, L2939) stay as they are — user text and the scratchpad are not locators.
|
||||
- the `done` handler (the `answer_image` neighborhood, L4340–L4360): `if (ev.answer_image) appendAnswerImage(...)` then `hydrateVerbatim(bubble, ev.answer_text || [])` then `appendUnlocatedVerbatim(wrap, ev.answer_text || [])` then `appendSources(...)`; `enhanceAnswerCodeBlocks(wrap)` runs once the turn settles (buttons appear with the settled answer, never mid-stream — pinned copy in the tests).
|
||||
- persistence (the `rememberBrainTurn(...)` save point, L4390–L4420): `answer_text: ev.answer_text || undefined` (undefined drops the key — the `answer_image` pattern, so a pre-phase record stays byte-identical).
|
||||
- the restore path re-renders + hydrates from `m.answer_text` (the `answer_image` restore precedent at L3001); a restored record with no `answer_text` renders its fences as the unresolved state (the record is what it is — never resurrect model bytes).
|
||||
- the terminal failure paths (the `error` frame handler, the 300 s `TURN_TIMEOUT_MS` settle, the Stop action) call `settleVerbatimPlaceholders(bubble, "unverified")`.
|
||||
- the Retry / edit-and-resend paths: nothing extra — they re-run the turn, the new turn hydrates on its own (verify the phase-120/159 suites stay green).
|
||||
3. `frontend/assets/shared.js` — the record renderer (L619–L660, the `addAnswerImage` neighborhood) passes `{ verbatim: true }`, hydrates from `m.answer_text`, appends the unlocated blocks, enhances the settled code blocks, and uses the plain-href `onOpen` (no modal on the shared page). Parity is a hard requirement: the same blocks render the same panel on both surfaces.
|
||||
4. `frontend/assets/styles.css` — extend task 06's block only where the real DOM needs it (states the JS adds: `.copied`, `.verbatim-unresolved`); palette variables only, no new literals.
|
||||
|
||||
## Testing & Quality
|
||||
|
||||
- **Unit (`tests/unit/test_frontend_verbatim_blocks.py`, the `test_frontend_*` text-pin convention):** `verbatim.js` is included after `markdown.js` in BOTH `index.html` and `shared.html`; `hydrateVerbatim` / `appendUnlocatedVerbatim` / `settleVerbatimPlaceholders` / `enhanceAnswerCodeBlocks` / `copyTextWithFallback` / `buildVerbatimPanel` exist; `app.js` passes `{ verbatim: true }` at the three brain renderers and does NOT pass it in the user/thinking renderers (assert by scoping the regex to the right call sites); the `done` handler hydrates BEFORE `appendSources`; `rememberBrainTurn` persists `answer_text` with `|| undefined`; `shared.js` reaches the same three globals (parity); no `onclick=`/`oncopy=` attribute strings anywhere in the new code (CSP); no `innerHTML += ` in the new code; the pinned unresolved copy `I could not verify this against the knowledge base`; the badge text `VERBATIM`; the copy button's `aria-label` template names the full path.
|
||||
- **Unit (CSS):** the new state selectors exist and stay literal-free (task 06's module extends).
|
||||
- **Coverage:** no new `app/` code; `app/` coverage must stay **>90%**, and the new pin module runs in the unit suite.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
- [ ] Live turn against a mock or real LLM: the panel shows, with the caption's numbers, the `VERBATIM` badge, line numbers, and a Copy button whose clipboard content equals the server text exactly; every ordinary code block in the same answer has a Copy button and NO badge.
|
||||
- [ ] Reload the page: the restored bubble renders the identical panel. Open the Share link in an anonymous window: the shared page renders the identical panel.
|
||||
- [ ] An answer with a ` ```bor ` fence whose path does not resolve shows the honest unresolved state with no model bytes; nothing sits at "Retrieving the exact text…" after a terminal frame (force it with the `slow_llm`/always-fail trigger).
|
||||
- [ ] A `show_text` call whose block the answer never locates renders appended above the source clusters.
|
||||
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean; keyboard-only check: the copy button and the viewer link are reachable with a 3px `:focus-visible` ring, and the reduced-motion setting changes nothing jumpy.
|
||||
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the answer-panel feature list, AGENTS rule 5's UI contract line) — the phase's final pass re-checks both project-wide.
|
||||
- [ ] no behavior change in completed phases (share-link copy strings, answer image, source clusters, related row, edit/resend, thinking timeline).
|
||||
@@ -0,0 +1,50 @@
|
||||
# Task 08 — Phase E2E suite, regression sweep, docs pass, commit
|
||||
|
||||
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
|
||||
|
||||
## Objective
|
||||
|
||||
One dedicated Playwright file proving verbatim display end to end (including clipboard truth), the named regression suites green in isolation, the docs/README/AGENTS consistency pass, and the single atomic commit.
|
||||
|
||||
## Work
|
||||
|
||||
1. `tests/e2e/mock_llm.py` — three new scripted branches (module docstring updated; keep every existing branch byte-identical, and re-verify the phase-116 discipline: no new trigger phrase is a substring of, or contained in, any existing trigger or fixture question):
|
||||
- `_SHOW_TEXT_TRIGGER = "show me the exact text of"` → request 1: a `show_text` tool call whose `path` is the parsed target (whole document, no `offset`/`limit`); the follow-up (request 2 carries the tool result starting with the new module constant `_SHOW_TEXT_CONFIRM_MARKER`, the `_READ_RESULT_PREFIX` precedent) → the final answer: prose + a ` ```bor ` fence carrying the target locator AND a DELIBERATELY CORRUPTED body (one fixture line with a character flipped and a trailing space) + a second fence for a second locator if the fixture has one. The corruption is the test: it must never reach the DOM.
|
||||
- `_FENCE_ONLY_TRIGGER = "put the block inline"` → NO tool call; prose + one ` ```bor ` fence (resolved purely from the answer text — proves the standalone half of the mechanism).
|
||||
- `_BAD_LOCATOR_TRIGGER = "show me a block I cannot find"` → NO tool call; prose + a ` ```bor ` fence pointing at a path that does not exist in the KB.
|
||||
2. Fixture: seed (module-scoped truncate + `import_directory`-style insert, the `test_answer_images.py` idiom) TWO small text documents with distinctive, punctuation-rich content — one with a short line-count and obvious per-line identity (so a clamped range and its line numbers are checkable), one whose content includes a URL, backticks and an em dash (the escaping and the "no retype" proof). Fixture questions (asked by the happy-path spec, disjoint from every other trigger's question): *"show me the exact text of `baselab/quadlets/gufo.container`"*, *"put the block inline for `baselab/notes/verbatim-b.md 2-3`"*.
|
||||
3. Create `tests/e2e/test_verbatim_show_text.py` (fixtures + style per `tests/e2e/README.md`; App A runs with the phase defaults, App B explicitly `BOR_ANSWER_TEXT=false`):
|
||||
- **happy path (App A):** ask the show-me question → assert exactly FIVE tools were offered (assert on the request capture, the `test_agent_document_tools.py` idiom); a `.verbatim` panel exists whose caption shows `lines 1-N of N`; the panel's body text equals the fixture's content line-for-line; the badge `VERBATIM` is present; the answer's prose does NOT contain the corrupted line; `.verbatim-pending` is gone at settle; the panel's open link targets `/document?source=…&path=…` (the viewer, where the git origin link lives); the appended-block path renders above `.sources`.
|
||||
- **clipboard truth:** grant `clipboard-read`/`clipboard-write` for the app origin (skip with a clear reason if the runner refuses), click the panel's Copy, read `navigator.clipboard.readText()`, assert byte equality with the fixture text fetched from the DB/API (no trailing newline, no line numbers); then click a plain code block's Copy in the same answer and assert IT copies exactly the model's block content (no line numbers, no badge).
|
||||
- **fence only (App A):** the inline locator case resolves with no tool call in the request log, the model's body text is absent from `page.content()`, and no literal ` ```bor ` text remains in the DOM at settle (a half-typed fence must have become the pending state, then the panel).
|
||||
- **unresolved (App A):** the bad-locator case renders the pinned unresolved copy, no `.verbatim-body`, no `VERBATIM`, and the model's bytes are absent.
|
||||
- **wrong range corrected (App A):** drive the tool + a fence with an out-of-range `offset` (reuse the show-me trigger with a range suffix) → the panel shows the server's extent, not the model's claim; if the resolver refused the whole show, the honest unresolved state appears and the model's claimed numbers appear nowhere.
|
||||
- **persistence:** after the happy path, reload `/history` and the open chat → the panel re-renders from the saved record; open the Share link in a second context (or `page.context().new_page()` in an incognito-style context) → the anonymous shared page shows the same panel; assert `/api/chat` for the same turn never emitted a second SSE event type (collect event names, assert `<= 7` distinct and `answer_text` came inside `done`).
|
||||
- **line-number honesty:** assert the caption's numbers exist as a DOM range (`lines 1-… of …`) and that with a fixture of exactly N lines the `of N` matches — no invented numbers.
|
||||
- **image regression (this phase's image check, per the owner):** a turn with `answer_image=True` still shows exactly one image bubble and NO `.verbatim` panel; `show_text` pointed at the image document (drive it via the tool branch if the mock can be pointed at the fixture image, else assert at unit level in task 03 and record it here) → the refusal copy naming `show_image`.
|
||||
- **App B (`BOR_ANSWER_TEXT=false`):** exactly FOUR tools offered; asking the same show-me question yields no `.verbatim` element at all; a ` ```bor ` fence in the answer renders as an ordinary `<pre><code>` block; the `done` frame has no `answer_text` key; the Copy button on ordinary code blocks STILL works (A5 is flag-independent).
|
||||
- **mobile:** at 390 px the panel scrolls horizontally without stretching the bubble and the Copy control is ≥44 px tall.
|
||||
- **stale-UI check (owner's standing rule):** the pending → resolved transition never leaves a second panel behind (assert exactly one `.verbatim` per locator).
|
||||
4. **Regression sweep (run in isolation, all green, no edits to the existing specs):** `test_answer_images.py`, `test_harness_aligned_tools.py`, `test_agent_document_tools.py`, `test_read_range_tool_lines.py`, `test_read_truncation_cap.py`, `test_tool_path_teaching.py`, `test_tool_scaffolding_guardrails.py`, `test_markdown_tables.py`, `test_markdown_blockquotes.py`, `test_long_answers.py`, `test_history_copy.py`, `test_chat_persistence.py`, `test_stale_ui_copy.py`. If `phase 180_markdown_inline_links_images` is still in `todo/`, run ITS E2E too (it shares `markdown.js`) and record the result.
|
||||
5. **Docs consistency pass:**
|
||||
- `README.md` — the feature (what the user sees: panel, `VERBATIM` badge, Copy on every code block), the `show_text` tool + the ` ```bor ` fence in the agent-tools/API sections, the `done.answer_text` field, the three env vars, the "the model points, the server supplies the bytes" principle stated once in plain words.
|
||||
- `AGENTS.md` — rule 5's UI contract line gains the phase-182 clause (an answer code block always has a Copy button; only a server-resolved block carries `VERBATIM`; a `bor` block's model bytes are never painted); rule 10's log-line list mentions `verbatim_blocks=N`.
|
||||
- `.agents/PLAN.md` — the tool surface (`show_text` as the 5th tool, gated like `show_image`), the `done` frame's additive field, the new settings; state explicitly that A1–A6 and A15 are otherwise UNCHANGED.
|
||||
- `git status` must show nothing stale; every file this phase touched is in the commit.
|
||||
6. **Commit** (mandatory, one atomic commit for the phase — Conventional Commits, `--no-gpg-sign` per repo default):
|
||||
```bash
|
||||
git add -A && git commit --no-gpg-sign -m "feat(rag): show file contents verbatim with a show_text tool and locator blocks
|
||||
|
||||
- add show_text(path, offset?, limit?): the server slices documents.content
|
||||
and displays it, so the user copies the KB's bytes instead of a retyped
|
||||
copy that may differ by a character
|
||||
- resolve \`\`\`bor source/path 12-40 locator fences in answers server-side;
|
||||
the model's fence body is never painted, unknown locators say so honestly
|
||||
- carried in the done frame's additive answer_text array and persisted with
|
||||
the saved chat, so restore and the shared page re-render identically
|
||||
- Copy button on every answer code block, VERBATIM badge only on
|
||||
server-resolved ones; line numbers always come from the server
|
||||
- gated by BOR_ANSWER_TEXT with per-block/per-answer budgets; with the flag
|
||||
off every frame, record and byte budget is exactly pre-phase"
|
||||
```
|
||||
Message in English (repo convention).
|
||||
Reference in New Issue
Block a user