Compare commits

..
3 Commits
Author SHA1 Message Date
ducoterra 76887639f3 docs(phases): add phase 182 (verbatim file display)
Build and Push Containers / build-and-push-app (push) Successful in 1m49s
Build and Push Containers / build-and-push-db (push) Successful in 13s
Answers to "show me my pi.dev config" re-type file contents from context,
so the code block in the answer can differ from the indexed file by a
character and cannot be trusted for copy-paste — the owner opens the
source bubble and ignores the answer.

Phase 182 makes the server the only party that supplies bytes:

- show_text(path, offset?, limit?), the display sibling of show_image,
  slices documents.content and hands the same bytes back to the model
  with a "do not retype" instruction
- inline ```bor source/path 12-40 locator fences place a resolved block
  anywhere in the prose; the model's fence body is never painted, and an
  unresolvable locator says so instead of showing guessed text
- the bytes ride inline in the done frame's additive answer_text array
  and persist in the saved chat, so restore and the shared page match
- line numbers are always the server's resolved extent (overshoot
  clamps, past-the-end refuses to the model), which removes the
  "did the model invent those numbers?" problem entirely
- a Copy button on every answer code block, a VERBATIM badge only on
  server-resolved ones
- gated by BOR_ANSWER_TEXT with per-block/per-answer budgets; with the
  flag off every frame, record and byte budget is exactly pre-phase

Overview plus eight tasks, each with its own test gate; the phase E2E
runs in isolation and asserts the clipboard content equals the stored
bytes. Run 180 first — both touch markdown.js.

LLM-Generated: true
2026-10-14 10:33:06 -04:00
ducoterra 50658580e9 chore(agent): phase roadmap from TODO.md, 2 phases
Convert the 3 open TODO.md items into sequential phases appended to the
existing pipeline (Protocol B, numbered after 179):

- 180_markdown_inline_links_images: the shared escape-first renderer
  gains the [text](url) link transform, the ![alt](src) image
  transform, and the ####-###### heading levels, with CSS and the
  dedicated Playwright suite.
- 181_svg_document_preview: SVGs index as vector image documents
  (text extracted from title/desc/text, no vision call), are served as
  sandboxed image/svg+xml, and preview as graphics instead of raw XML.

TODO.md is cleared; its items now live in .agents/phases/.

LLM-Generated: true
2026-10-14 09:54:01 -04:00
ducoterra dad54a2854 chore(agent): phase roadmap from TODO.md (179 git clone server freeze)
TODO L3 (a long git clone seems to freeze the server) converted to phase 179:
deterministic in-process repro + verdict, the asyncio.to_thread fix for the
sync's git ops, and the dedicated Playwright E2E suite. TODO.md cleared
(items now live in .agents/phases/).

LLM-Generated: true
2026-10-12 09:27:58 -04:00
26 changed files with 971 additions and 2 deletions
@@ -0,0 +1,41 @@
# Phase 179 — Git clone server freeze: investigate the owner's suspicion, then fix it
**Source:** `TODO.md` L3 — "A long git clone seems to freeze the server? Investigate to find out if this is true."
**Story:** n/a (TODO-derived, owner roadmap confirmation 2026-10-11)
**Context:** `app/api/sync.py` — `start_sync` (L421–438, `asyncio.create_task(_run_sync())` L438), `_run_sync` (L482, **async**, runs on the app's single uvicorn event loop), `await check_models(llm)` (L533 — already async HTTP probes), the short-lived `SessionLocal()` + `effective_sources(db)` (L538–540), the per-source `_check_cancelled()` boundary (L565), **`root = clone_or_pull(…)` (L577 — synchronous)**, **`file_commit_dates(root)` (L589 — synchronous)**, `sync_status` (L353–354 — "the UI polls this every 2 s"), `cancel_sync` (L442–459); `scripts/git_sync.py` — `clone_or_pull` (L96), `file_commit_dates` (L270), `run_git` (L320 — `subprocess.run` L342, **blocking**); `app/api/health.py` — `GET /api/health` (L13, mounted at `/api/health` via `app/main.py` L211); `scripts/import_docs.py` — the standalone script path (a SEPARATE process, never on the app's event loop — NOT a freeze vector); `app/api/doc_drafts.py` — `push_doc_draft` (a sync `def` → FastAPI threadpool — NOT a freeze vector); `app/api/sync.py` L233 (`from scripts.git_sync import GitSyncError, clone_or_pull, file_commit_dates` — the names to monkeypatch); `tests/integration/test_sync_api.py` — the `sync_client` context-managed `TestClient` pattern (L176–179 — one app event loop across requests) + the env-fallback `BOR_GIT_SOURCES` driving (L64–65 docstring); `tests/e2e/test_sync_cancel.py` — the dedicated module-app E2E pattern (dedicated port, `file://` fixture repo via the `_git` helper, the `_wall_guard` module fixture); `tests/e2e/slow_llm.py` (not needed here — the mock LLM import is fast; the fixture's slowness is in the CLONE, not the import).
## Objective
Verify the owner's suspicion with a deterministic reproduction (a long `git clone` during a sync blocks the app's event loop, stalling the entire HTTP surface — including the 2 s sync-status poll the UI relies on), record the verdict, and — the suspicion being confirmed — make the sync's git operations run OFF the event loop so the server stays fully responsive for the whole duration of a long clone (and the phase-134 cooperative cancel keeps its contract).
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the repro is in-process and deterministic (task 01).** A new `tests/integration/test_sync_git_freeze.py` follows `test_sync_api.py`'s `sync_client` pattern (context-managed `TestClient` — one app event loop across requests). `app.api.sync.clone_or_pull` and `app.api.sync.file_commit_dates` are monkeypatched (the module-level names imported at L233) with stubs that record a `time.monotonic()` start/end and sleep 3 s. The probe test measures the wall-clock latency of a `GET /api/sync/status` issued while the stub is in its sleep. The probe asserts the **PRE-fix** behavior (the probe is blocked — latency ≥ 2.5 s of the 3 s stub) → the test is GREEN before the fix and IS the demonstration of the freeze; the verdict (**CONFIRMED** / not) + mechanism + the measured latency are recorded in the test module docstring. No app/ code is touched in task 01.
- **D2 — the fix is `asyncio.to_thread`, nothing else (task 02).** Inside `_run_sync`: `root = await asyncio.to_thread(clone_or_pull, clone_url_for(row), sources_root / repo_name(row.url), row.branch)` and `doc_dates_by_root[str(root)] = await asyncio.to_thread(file_commit_dates, root)`. The module docstring and the phase-134 boundary comment (L565–576) update the semantics: a cancel registered while a git op is in flight no longer parks the EVENT LOOP — the git **worker thread** runs to completion (Python threads are not interruptible), the loop stays free (every other request, status poll included, is served), and the run still stops at the NEXT `_check_cancelled()` boundary (the phase-134 contract, unchanged). The task-01 probe flips to the post-fix assertion (probe latency < 1.5 s during the 3 s stub clone). The short-lived `SessionLocal()` / `effective_sources` DB calls stay on the loop (A3 — millisecond-scale, out of scope, noted in the docstring). `check_models` is already async HTTP — untouched.
- **D3 — the E2E pins the real pipeline (task 03).** `tests/e2e/test_sync_git_freeze.py` (the phase's dedicated Playwright suite, AGENTS rule 4): the dedicated module app on a dedicated port (the `test_sync_cancel` pattern — the session app is never started in the isolated run), mock LLM, a dedicated `BOR_SOURCES_DIR`, `BOR_GIT_SOURCES=file://<big repo>`, the shared E2E Postgres truncated before AND after every test. The `big_git_repo` fixture is a GENERATED local repo — N commits × ~1 MiB of near-incompressible (seeded pseudo-random) `.md` files — sized at authoring so a FRESH `git clone` of it takes ≥ 5 s on the test host (the pinned constants + the calibration note live in the module; `git` on PATH is the documented prerequisite, phase 28). Contract 1: while the run is in the clone phase, `GET /api/sync/status` polls (via `page.request`, ~2 s apart like the UI) each complete in < 1.5 s and the Sources page keeps updating with no network-error state; the run lands `success` and the sentinel file is in the KB. Contract 2 (phase 134 under to_thread): a cancel registered during the clone — the checkout dir is deleted first so the run re-clones (a long git phase again) — the status endpoint stays responsive (< 1.5 s) while the in-flight git thread finishes, and the run lands the `cancelled` terminal (error null, the UI's "Sync cancelled" neutral state). The `_wall_guard` module fixture pins the budget (the `test_sync_cancel` pattern — cut the fixture size if it drifts, never the guard). `E2E_REAL_LLM=1` is skipped (mock-only — the clone, not the import, is the timing fixture).
- **NOT touched:** `scripts/import_docs.py` (standalone process), `scripts/git_sync.py` (the git layer is unchanged — only its SCHEDULING changes), `app/core/docs_push.py` / `app/api/doc_drafts.py` (sync `def` → threadpool), `check_models`, the async LLM import path, the frontend, any other endpoint.
## Dependencies
— (none: the app is fully functional at the head of `complete/`; this phase investigates + fixes the in-process sync path)
## Tasks
1. `01_repro_verdict.md` — the deterministic in-process repro (3 s stubbed clone, status-poll latency probe) that demonstrates the event-loop freeze and records the verdict in the test module docstring.
2. `02_offloop_git_ops.md` — the fix: `clone_or_pull` / `file_commit_dates` off the event loop via `asyncio.to_thread`, the docstring/boundary-note update, the probe flipped to the post-fix assertion.
3. `03_e2e_freeze_contract.md` — `tests/e2e/test_sync_git_freeze.py` (the phase's dedicated Playwright suite, AGENTS rule 4): a real long `file://` clone, the responsive-status-poll contract, the phase-134 cancel-during-clone contract, the full gates, the atomic commit.
## Testing & Quality
- **Integration:** `tests/integration/test_sync_git_freeze.py` — the probe: pre-fix (task 01) the status poll issued during the 3 s stubbed clone is blocked (≥ 2.5 s); post-fix (task 02) it completes in < 1.5 s; the run reaches a terminal state in both (fixture cleanliness). `test_sync_api.py` + `test_import_docs_git.py` stay green (no behavior change beyond loop-freeness).
- **E2E (AGENTS rule 4, task 03):** `tests/e2e/test_sync_git_freeze.py`, run **in isolation** — the clone-phase status polls responsive (< 1.5 s each, ~2 s apart), the Sources page updating, the `success` terminal + sentinel in the KB; the cancel-during-clone → `cancelled` terminal with the status endpoint responsive while the git thread finishes.
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing` — the only `app/` change is two `await asyncio.to_thread(…)` lines + docstrings in `app/api/sync.py`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_sync_git_freeze.py -v` green with the post-fix assertion (probe < 1.5 s during the 3 s stub clone) and the verdict ("CONFIRMED" + mechanism + measured latency) recorded in the module docstring.
- [ ] `app/api/sync.py::_run_sync` runs the per-source git ops off the event loop (`asyncio.to_thread`); the phase-134 boundary note documents the worker-thread-runs-to-completion semantics; the DB calls stay on the loop (the A3 note).
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` green **in isolation** (the responsive-poll contract + the cancel-during-clone contract, wall-time guard holds).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `fix(sync): run the sync's git clone/pull off the event loop so long clones no longer freeze the server`); the phase dir moves to `.agents/phases/complete/`.
## Locked decisions (owner-confirmed 2026-10-11, roadmap confirmation)
- **A1 — scope: investigate + fix.** The TODO item is an investigation ("find out if this is true"); the owner confirmed the phase goes on to the minimal fix. If the repro in task 01 had landed NOT-confirmed, task 02 would document the no-fix decision instead — the code-path analysis (the synchronous `subprocess.run` inside the async `_run_sync`) and the repro together confirm the freeze.
- **A2 — fix mechanism: `asyncio.to_thread`.** No new packages; the single-event-loop uvicorn is unchanged; a cancel at the git phase leaves the in-flight git worker thread running to completion (Python threads are not interruptible) while the event loop is free — the phase-134 cooperative-cancel contract (stop at the next boundary, `cancelled` terminal, no corruption) is preserved.
- **A3 — out of scope: the sync's short-lived DB calls.** `SessionLocal()` / `effective_sources` (L538–540) are millisecond-scale SQLAlchemy calls — not the reported symptom; they stay on the loop and are noted in the docstring as a known, bounded, non-fix.
- **A4 — the E2E slow clone is a generated local `file://` repo.** N commits × ~1 MiB seeded pseudo-random `.md` files, sized at authoring so a fresh clone takes ≥ 5 s on the test host; deterministic, no network, `git` on PATH is the documented prerequisite (phase 28).
@@ -0,0 +1,27 @@
# Task 01 — Reproduce the freeze in-process and record the verdict
**Phase:** `179_git_clone_server_freeze` · **Source:** `TODO.md:3` — "A long git clone seems to freeze the server? Investigate to find out if this is true."
**Story:** n/a
## Objective
A deterministic, in-process reproduction that proves (or disproves) the owner's suspicion: while a sync's git clone is in flight, the app's event loop is blocked and the whole HTTP surface — including `GET /api/sync/status`, the endpoint the UI polls every 2 s — is stalled. The verdict (CONFIRMED / not, the mechanism, the measured latency) is recorded in the test module docstring.
## Work
1. `tests/integration/test_sync_git_freeze.py` — a NEW file following `tests/integration/test_sync_api.py`'s house patterns:
- The `sync_client` fixture (L176–179 pattern): a **context-managed** `TestClient(fastapi_app)` — one app event loop across requests (the docstring L64–65 explains why the context manager is required). Admin login via the house helper (the `_login` pattern, `test_sync_api.py` L260–265).
- The git source via the **env fallback**: drive `BOR_GIT_SOURCES=file://<tmp repo>` the way `test_sync_api.py` does (a fresh `Settings` on the resolver's module; the dev `.env` never leaks in). The tmp repo is a real one-commit git repo built with the house `_git` helper pattern (`tests/e2e/test_sync_cancel.py` L156–195 shape: `git init -q`, `add -A`, `-c user.email=… -c user.name=… -c commit.gpgsign=false commit -qm`); the stub never actually clones it, so the repo's content is irrelevant — one small `.md` file suffices.
- **The stubs (the timing fixture):** `monkeypatch.setattr("app.api.sync.clone_or_pull", …)` and `monkeypatch.setattr("app.api.sync.file_commit_dates", …)` — the module-level names imported at `app/api/sync.py` L233 (patching the `scripts.git_sync` originals would not affect the runner). The `clone_or_pull` stub: on entry sets a `threading.Event` (`clone_started`) and records `time.monotonic()`, sleeps **3.0 s** (simulating a long clone), records the end, and returns the tmp repo's `Path`; the `file_commit_dates` stub returns `{}`. Both are plain sync functions — the point is that PRE-fix they run ON the event loop.
- **The probe test** `test_long_git_clone_freezes_the_event_loop`: `POST /api/sync` (admin) → wait for `clone_started` (bounded wait, ~10 s) → `t0 = time.monotonic()`, `client.get("/api/sync/status")`, `elapsed = time.monotonic() - t0` → **assert `elapsed >= 2.5`** (the probe request, issued into the middle of the 3 s stubbed clone, is BLOCKED — it cannot be served until the stubbed clone returns and the event loop is free again). That assertion IS the freeze demonstration and it is GREEN pre-fix. Then poll `GET /api/sync/status` to the terminal state (the stub returns, so the run completes: `success` with the import, or a clean terminal — whichever the stubbed root yields; pin whichever the suite shows and document it) so the fixture is left clean for the next test.
- **The module docstring (the verdict record):** the owner's TODO item quoted (L3), the code path (`POST /api/sync` → `asyncio.create_task(_run_sync())` — `app/api/sync.py` L438/L482 → the synchronous `clone_or_pull` L577 → `run_git`'s blocking `subprocess.run` — `scripts/git_sync.py` L342 — with no `to_thread`/`create_subprocess_exec` anywhere in `app/` or `scripts/`), the measured probe latency (e.g. "the status probe took ≈3.0 s of the 3.0 s stubbed clone — the event loop was stalled for the full git op"), and the verdict line: **CONFIRMED** (or NOT — if the probe came back fast, stop and flag it in the docstring + the phase report rather than proceeding to a fix).
- `GET /api/health` (`app/api/health.py` L13, mounted at `/api/health`) may be used as a SECOND probe in the same test (the same assertion — it is blocked too: any endpoint on the loop stalls); keep the status probe primary (it is the endpoint the UI's 2 s poll depends on).
2. Verify by running: `uv run pytest tests/integration/test_sync_git_freeze.py -v` — green (the probe is green with the PRE-fix assertion: it documents the freeze, it is not a failing test), and `uv run pytest` — the full suite green (no other test is affected: the stubs live inside this one test's monkeypatch).
## Testing & Quality
- Integration: the probe family above (the blocked-latency assertion, the terminal-state assertion, the admin/anonymous gating falls out of the house `sync_client` pattern — add the anonymous-403 pins if the house pattern does not already carry them in this file).
- Coverage: **>90%** on `app/` (this task is test-only — `app/` code is untouched, so the suite's coverage must simply stay green).
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_sync_git_freeze.py -v` green with the pre-fix assertion (`elapsed >= 2.5` for the status probe issued mid-clone).
- [ ] The module docstring records: the TODO item, the code path, the measured latency, and the verdict (**CONFIRMED** — expected, per the code-path analysis).
- [ ] `uv run pytest` green.
- [ ] No `app/` code change in this task.
@@ -0,0 +1,28 @@
# Task 02 — The fix: the sync's git operations run off the event loop
**Phase:** `179_git_clone_server_freeze` · **Source:** `TODO.md:3` — "A long git clone seems to freeze the server? Investigate to find out if this is true."
**Story:** n/a
## Objective
With the freeze confirmed (task 01's verdict), make the server stay fully responsive for the whole duration of a long `git clone` / `git pull` (and the `file_commit_dates` walk): the sync's per-source git operations run in a worker thread via `asyncio.to_thread`, the phase-134 cooperative-cancel contract is preserved, and the task-01 probe flips to pin the post-fix behavior.
## Work
1. `app/api/sync.py` — `_run_sync` (L482), the per-source git block (L566–589):
- L577: `root = clone_or_pull(…)` → `root = await asyncio.to_thread(clone_or_pull, clone_url_for(row), sources_root / repo_name(row.url), row.branch)` (the three-argument call, phase 121/135, unchanged — only its scheduling changes).
- L589: `doc_dates_by_root[str(root)] = file_commit_dates(root)` → `doc_dates_by_root[str(root)] = await asyncio.to_thread(file_commit_dates, root)`.
- `import asyncio` if not already imported at module level (check first — the module already uses `asyncio.create_task` at L438, so `asyncio` is imported; verify, do not duplicate).
- **The phase-134 boundary note (L565–576)** — update the comment: a cancel registered while a git op is in flight no longer parks the EVENT LOOP; the git **worker thread** runs its blocking `subprocess` to completion (Python threads are not interruptible — `_task.cancel()` unblocks the run at the `to_thread` await point / the run stops at the NEXT `_check_cancelled()` boundary), while every other request — status polls included — is served immediately. The observable contract is unchanged (phase 134: stop at the next boundary, `cancelled` terminal, no corruption).
- **The module docstring** — add the freeze note: pre-phase-179 the synchronous `clone_or_pull` / `file_commit_dates` (their `subprocess.run` calls — `scripts/git_sync.py` L342) ran ON the app's event loop, stalling the whole HTTP surface (including the 2 s `sync_status` poll) for the duration of a long clone (the TODO L3 suspicion — confirmed by `tests/integration/test_sync_git_freeze.py`); phase 179 runs them off the loop via `asyncio.to_thread`. Include the **A3 note**: the short-lived `SessionLocal()` / `effective_sources` DB calls (L538–540) remain on the loop — millisecond-scale, a known bounded non-fix, deliberately out of scope.
- `check_models` (L533) is untouched (already async HTTP probes). The kind-local branch (L590+) is untouched (no subprocess in it).
2. `tests/integration/test_sync_git_freeze.py` — flip the task-01 probe to the post-fix assertion: the status probe issued into the middle of the 3 s stubbed clone now completes in **< 1.5 s** (the event loop is free while the stubbed clone runs in the worker thread); keep the terminal-state assertion. Update the module docstring: the verdict section now reads CONFIRMED (task 01) → FIXED (phase 179, `asyncio.to_thread`), with the post-fix measured latency recorded (the probe ≈ instant, e.g. "the status probe took ≈0.01 s while the 3.0 s stubbed clone ran in the worker thread").
3. Verify by running: `uv run pytest tests/integration/test_sync_git_freeze.py tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -v` — all green (the flipped probe green; the sync families show no behavior change beyond loop-freeness), then `uv run pytest` — the full suite green.
## Testing & Quality
- Integration: the flipped probe (post-fix: status probe < 1.5 s during the 3 s stubbed clone) + the `test_sync_api.py` / `test_import_docs_git.py` regression set.
- Coverage: **>90%** on `app/` (the only `app/` change is two `await asyncio.to_thread(…)` lines + docstring/comment in `app/api/sync.py` — the suite's coverage must stay green).
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_sync_git_freeze.py -v` green with the POST-fix assertion (probe < 1.5 s during the 3 s stub clone) and the docstring verdict updated (CONFIRMED → FIXED, post-fix latency recorded).
- [ ] `app/api/sync.py::_run_sync` awaits `asyncio.to_thread` for BOTH git ops; the phase-134 boundary note + the module docstring document the worker-thread semantics and the A3 DB note; `check_models` untouched.
- [ ] `uv run pytest tests/integration/test_sync_git_freeze.py tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -v` green; `uv run pytest` green.
- [ ] No behavior change in completed work beyond loop-freeness (the sync pipeline's sequence, errors, and cancel terminals are unchanged).
@@ -0,0 +1,32 @@
# Task 03 — E2E: the status poll stays responsive through a real long clone (the phase's dedicated Playwright suite)
**Phase:** `179_git_clone_server_freeze` · **Source:** `TODO.md:3` — "A long git clone seems to freeze the server? Investigate to find out if this is true."
**Story:** n/a
## Objective
Pin the fix against the REAL pipeline (the owner's exact suspicion, end to end): a sync whose first action is a genuinely long `git clone` (a generated local `file://` repo) must keep the whole HTTP surface — the 2 s `GET /api/sync/status` poll and the Sources page — responsive for the entire duration of the clone, the run must land `success`, and the phase-134 cancel-during-clone contract must hold under the off-loop scheduling. Then run the full gates and commit the phase.
## Work
1. `tests/e2e/test_sync_git_freeze.py` — a NEW module, mock-only (`E2E_REAL_LLM=1` skipped — the CLONE, not the import, is the timing fixture), following `tests/e2e/test_sync_cancel.py`'s house pattern point for point:
- **Isolation:** a per-module app on a **dedicated port** (the conftest session app is never started in this isolated run), the mock LLM on the conftest port, a dedicated `BOR_SOURCES_DIR` for the fixture repo's checkout, `BOR_GIT_SOURCES=file://<big repo>` (the env-fallback row — the `test_sync_cancel` L256 shape), the shared E2E Postgres truncated before AND after every test.
- **The `big_git_repo` fixture (the timing fixture — A4):** a module-scoped helper (the `_git` pattern, `test_sync_cancel.py` L156–195) that generates N commits, each adding one ~1 MiB `notes/note-<k>.md` whose body is **seeded pseudo-random** text (deterministic — a fixed-seed generator, e.g. `random.Random(k)` over a word list — so the pack is near-incompressible and the clone time is stable; plain repeated text would compress away and the clone would be fast). **Calibrate at authoring time**: pick N (start ~200) so a FRESH `git clone file://<repo>` takes **≥ 5 s** on the test host (measure it during authoring, record the measured clone time in the module docstring, and pin N + the file size as module constants with the calibration note). The last file carries the sentinel marker (the `test_sync_cancel` sentinel shape).
- **Test 1 — `test_status_polls_responsive_during_long_clone`:** admin login (the house helper) → trigger the sync (the UI's sync button, or the API via `page.request` — the `test_sync_cancel` L100-ms `page.request` poll shape) → while the run reports `running` and is still in the clone phase (the status's `current_file` is null / no file positions — the import has not started), poll `GET /api/sync/status` via `page.request` at ~2 s intervals (the UI's poll cadence): **every poll must complete in < 1.5 s** (the UI's 2 s poll budget — pre-fix the polls issued during the clone would stall for the whole clone and this is the owner's frozen server, observable as hung status polls / a stalled Sources page) → the Sources page keeps updating (no network-error state, the status label keeps advancing) → the run lands the `success` terminal → the sentinel file is in the KB (the house drill-down/`GET /api/docs` assertion shape).
- **Test 2 — `test_cancel_during_clone_reaches_terminal`:** first DELETE the fixture repo's checkout dir under `BOR_SOURCES_DIR` (the run re-clones — a long git phase again), trigger a second sync, and while it is in the clone phase register the cancel (the UI's Cancel control or `POST /api/sync/cancel` via `page.request`) → **the status endpoint stays responsive** (the `page.request` polls during the in-flight git worker thread all complete in < 1.5 s — the A2 observable: the git thread runs to completion in the background while the loop serves requests) → the run lands the `cancelled` terminal (the phase-134 contract: `error: null`, the in-flight positions cleared, the UI's neutral "Sync cancelled" state — no error banner, the main button retry-ready).
- **The `_wall_guard` module fixture** (the `test_sync_cancel` pattern): budget the module (two syncs, the first a full clone + mock import of N files, the second a re-clone + cancel) — budget it at the measured clone time + import span + setup, pin the guard; **if the suite drifts, cut N, not the guard.**
- The module docstring: the owner's TODO item quoted, the fix (task 02's `asyncio.to_thread`), the fixture calibration (N, the measured clone time), the timing paragraph, the run-in-isolation command (`uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov`), the environment prerequisites (DB up: `podman compose up -d db`; `git` on PATH — the phase-28 documented prerequisite), the test→observable mapping (the house shape).
2. **The full gates** (the phase's final pass):
- `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` — green **in isolation**.
- `uv run pytest --cov=app --cov-report=term-missing` — the full suite green, `app/` coverage **>90%**.
- `uv run ruff check . && uv run pyright` — clean.
- The regression spot-check in isolation: `uv run pytest tests/e2e/test_sync_cancel.py -v --no-cov` (the phase-134 cancel E2E — the contract this phase must not have broken; run it AFTER the new suite so the DB truncate ordering is clean).
3. **Commit + move:** one atomic `--no-gpg-sign` Conventional Commit covering `app/api/sync.py`, the two test files, and the `.agents/` tree — e.g. `fix(sync): run the sync's git clone/pull off the event loop so long clones no longer freeze the server` (the commit body names the TODO L3 investigation + the verdict). Move the phase dir to `.agents/phases/complete/179_git_clone_server_freeze/`.
## Testing & Quality
- E2E: the dedicated suite above (the responsive-poll contract + the cancel-during-clone contract, the `test_sync_cancel` regression spot-check).
- Coverage: **>90%** on `app/`; ruff + pyright clean.
## Completion Criteria
- [ ] `uv run pytest tests/e2e/test_sync_git_freeze.py -v --no-cov` green **in isolation** (every status poll < 1.5 s during the real clone, `success` terminal + sentinel in the KB, cancel-during-clone → `cancelled` terminal with the status endpoint responsive, wall-time guard holds).
- [ ] `uv run pytest tests/e2e/test_sync_cancel.py -v --no-cov` green in isolation (the phase-134 contract intact).
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green with `app/` coverage >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One atomic `--no-gpg-sign` Conventional Commit; the phase dir moved to `.agents/phases/complete/`.
@@ -0,0 +1,54 @@
# Phase 180 — Markdown inline links, inline images, and the full heading ladder
**Source:** `TODO.md` L4 ("Markdown [links](some-address) aren't being render correctly and show up as plain text"), L5 ("Quad-level headers (####) aren't being rendered and show up as plain text") and the in-answer half of L3 ("Need SVG render support on the front end when citing … source")
**Story:** n/a (TODO-derived phase, Protocol B append — owner roadmap confirmation 2026-10-14)
**Context:** `frontend/assets/markdown.js` — the SHARED escape-first renderer used by all three pages (`index.html` chat bubbles + thinking, `document.html` viewer, `shared.html`) and by `document-modal.js` through `document.js`'s `renderDocument`. Pipeline order: step 1 fence protection (`\u0000CODE n\u0000`) → step 1b GFM pipe-table protection (`\u0000TABLE n\u0000`, phase 44) → step 2 `text = inline(escapeHtml(text))` → step 2b blockquotes (phase 138) → step 3 the paragraph pass (split on `\n{2,}`, block guard `/^<(h\d|ul|ol|pre|li|blockquote)/`) → step 4 restore. The `inline(s)` chain (the single source of every inline transform, also applied per table cell) is today, in order: code span → `**strong**` → `*em*` → `^### `→`<h4>` → `^## `→`<h3>` → `^# `→`<h3>` → `^\s*[-*] `→`<li>` → `<ul>` grouping → `^\d+\. `→`<li>`. Three verified gaps: **no link transform exists at all** (`renderMarkdown("[a](https://x)")` returns the text verbatim inside a `<p>`), **no image transform exists at all** (`![alt](src)` prints literally), and the heading ladder stops at `###` — `####`/`#####`/`######` match no rule and print as plain text (each rule requires a space after its hashes, so no rule partially eats a deeper level). `frontend/assets/styles.css` — the `:root` palette (`--ink`, `--ink-soft`, `--line`, `--surface`, `--bg`, the `--brand-*` family, `--radius*`) plus the precedent rule families: `.md-table`/`.md-table-wrap` (L539-557), `.md-quote` + the `.msg.user .bubble .md-quote` inversion (L561-588), `.source-image-img` / `.doc-image-img` (L981-1027, L5611-5640) for image treatment. `app/core/security_headers.py::CSP` (L67-68) — `default-src 'self'; base-uri 'none'; frame-ancestors 'none'; img-src 'self' data:`: served-path and `data:` images are ALREADY permitted, no CSP change is needed or allowed here. Test patterns: `tests/unit/test_markdown_blockquotes.py` (source pins + a `HAVE_NODE`-skipped node behaviour harness + a byte-identity fixture string) and `tests/e2e/test_markdown_blockquotes.py` (dedicated fixture tree `tests/fixtures/quote_docs/`, mock LLM trigger, chat + viewer + localStorage-restore assertions).
## Objective
`[text](url)` renders as a real, safe `<a>`, `![alt](src)` renders as a lazy `<img>`, and `####`–`######` render as `<h5>`/`<h6>` — in chat answers, the thinking block, saved/shared chats, the document viewer and the document modal, from the ONE shared renderer. Every input that contains none of the three constructs renders byte-identical to today, so the phase-44 table suite, the phase-138 blockquote suite and every renderer-dependent unit suite stay green unedited.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — one chain, one place (tasks 01–03).** All three fixes are edits to the same `inline(s)` chain in `frontend/assets/markdown.js`. Nothing else in the pipeline moves: fences (step 1) and tables (step 1b) keep their precedence, and because a table cell goes through `inline(escapeHtml(cell))`, links/images/deep headings inside a cell work for free (pinned in task 05).
- **D2 — escape-first is the security boundary, so NEVER unescape.** `inline()` always receives HTML-escaped text. The captured URL may therefore contain `&amp;`/`&#39;`/`&quot;` — that is *valid* in an `href`/`src` and must be passed through verbatim (never decoded, never re-escaped). Any attempt to decode would re-open the injection surface the renderer exists to close.
- **D3 — links (task 01).** `/\[([^\]]*)\]\(([^)\s]+)\)/g` (empty link text allowed, no whitespace inside the URL), applied AFTER the `*em*` rule and BEFORE the heading ladder. The URL is validated by an allowlist *function*, not a regex-in-replacement: allow `^https?://`, `^mailto:`, `^/`, `^#`, `^\.{1,2}/`; anything else (`javascript:`, `data:`, `vbscript:`, `file:`, bare `javascript&colon;` variants) returns the ORIGINAL matched text unchanged — it stays visible as inert escaped text, never a live element. Absolute `http(s)://` links get `target="_blank" rel="noopener noreferrer"`; `mailto:`, root-relative and `#fragment` links get neither (a same-origin target in the same tab is the correct behaviour). Output marker: `<a class="md-link" …>`.
- **D4 — images (task 02), matched BEFORE the link rule.** `/!\[([^\]]*)\]\(([^)\s]+)\)/g` → `<img class="md-img" src="…" alt="…" loading="lazy">`. The image rule runs FIRST so the `!` is never orphaned into `!<a>`. Same allowlist discipline: `/`-rooted served paths, `http(s)://` and `data:image/*` render; anything else stays inert text. `alt` is ALWAYS emitted (empty allowed) — an `<img>` without `alt` fails WCAG. Linked images (`[![alt](src)](url)`) get their OWN rule ahead of both (locked A6): `<a class="md-link" …><img class="md-img" …></a>` — otherwise the image rule leaves a stray `[` inside the link text.
- **D5 — the heading ladder (task 03, locked A4).** Insert `/^#### (.*)$/gm → <h5>$1</h5>` and `/^#{5,6} (.*)$/gm → <h6>$1</h6>` immediately BEFORE the existing `/^### /` rule (deeper-first is the readable order; the rules cannot cross-match because every rule requires a space after its hashes). `#` and `##` keep mapping to `<h3>` — existing behaviour, deliberately unchanged. The step-3 guard already matches `h\d`, so no paragraph-guard change is needed (pinned, not assumed).
- **D6 — CSS (task 04).** `.md-link` (underlined; `color: var(--brand-ink)` — 9.0:1 on `--surface`, the file's documented AA-safe brand text token — with `--brand-hover` on `:hover`, the shared 3px `--brand` `:focus-visible` ring already used at L163, and the `.msg.user .bubble .md-link { color: var(--bg) }` inversion following the `.md-quote` precedent at L588, 5.2:1 on the brand bubble), `.md-img` (`max-width:100%; height:auto; object-fit:contain` + the `.source-image-img` border/radius/surface language), and the `.doc-md` / `.bubble` / `.thinking-text` `h5`/`h6` type scale. Palette variables ONLY, no new hue (the monochrome invariant), every non-obvious color carrying its `/* n.n:1 on --x (AA) */` comment.
- **D7 — byte-identity is the regression contract (task 05).** A checked-in fixture string with no links, no image refs and no `####`+ heading, plus its byte-exact pre-phase renderer output (the phase-138 `NO_QUOTE_INPUT`/`NO_QUOTE_PRE_PHASE_OUTPUT` shape), pinned in the node behaviour harness.
- **Known, documented limits (recorded in the module docstring, not fixed):** inline code spans are NOT placeholder-protected (pre-existing: `**bold**` inside backticks already transforms), so markdown syntax inside an inline code span transforms too; a URL containing a literal `)` or whitespace is a miss; a size-less SVG (`viewBox` only, no `width`/`height`) in `.md-img` paints at the replaced-element default box — the vector sizing contract belongs to phase 181.
- **NOT touched:** `app/**` (no server change in this phase), the CSP string, `document.js`'s `renderDocument` structure, `app.js`'s chip/figure builders, `styles.css` families outside `.md-*` + the `h5`/`h6` scale.
## Dependencies
- `179_git_clone_server_freeze` (todo) — numbering/pipeline order only. The code dependency is the shared renderer at the head of `complete/` (phases 44 tables, 138 blockquotes).
## Tasks
1. `01_inline_links.md` — the `[text](url)` → `<a class="md-link">` transform with the protocol allowlist in `markdown.js`.
2. `02_inline_images.md` — the `![alt](src)` → `<img class="md-img">` transform (ahead of the link rule) + the linked-image rule.
3. `03_heading_ladder.md` — `####` → `<h5>`, `#####`/`######` → `<h6>` in the ladder.
4. `04_styles_markdown_inline.md` — `.md-link`, `.md-img` and the `h5`/`h6` type scale in `styles.css`.
5. `05_unit_and_e2e.md` — `tests/unit/test_markdown_inline_links_images.py` + the phase's dedicated Playwright suite `tests/e2e/test_markdown_inline_links_images.py`, the full gates, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_markdown_inline_links_images.py` (new, the phase-138 pattern): source pins (the chain order code → strong → em → **image → linked-image → link** → headings → lists; the allowlist function; the exact `md-link`/`md-img` markers; no `unescape`/`decodeURIComponent` anywhere in the file; the `.md-link`/`.md-img` CSS rules reference palette vars only) + node behaviour (link renders with `target`/`rel`; `javascript:`/`data:` inert; relative + `#fragment` + `mailto:` link without `target`; `&amp;` in a query string survives verbatim; image renders with `alt` (empty alt allowed), disallowed src inert; linked image wraps without a stray `[`; `####`→`<h5>`, `#####`/`######`→`<h6>`, `###`→`<h4>` unchanged; a link inside a fence stays code; a link/image inside a table cell renders; byte-identity fixture).
- **E2E (AGENTS rule 4):** `tests/e2e/test_markdown_inline_links_images.py`, run **in isolation** (`uv run pytest tests/e2e/test_markdown_inline_links_images.py -v --no-cov`, DB up).
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`) — this phase adds no `app/` code, so the full suite staying green IS the gate.
- **Lint/types:** `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] `[text](url)` renders as `<a class="md-link">` in a chat answer, the thinking block, a restored shared chat, the viewer and the document modal; `javascript:`/`data:` URLs stay visible as inert text
- [ ] `![alt](src)` renders as `<img class="md-img" loading="lazy">` with an `alt` present; a non-allowlisted src stays inert text
- [ ] `####` renders `<h5>`, `#####`/`######` render `<h6>`; `#`/`##`/`###` are unchanged
- [ ] a link/image/`####` inside a fenced code block is not transformed; a cell-level link renders in a `.md-table`
- [ ] an input with none of the three constructs renders byte-identical (phase-44 + phase-138 suites pass unedited)
- [ ] `uv run pytest` green; coverage TOTAL >90%; the phase's E2E green in isolation; `uv run ruff check . && uv run pyright` clean
- [ ] one `--no-gpg-sign` Conventional Commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate
## Locked decisions
- **A4 — `####` → `<h5>`; `#####` and `######` both → `<h6>`; `#`/`##` keep rendering as `<h3>` (owner-confirmed 2026-10-14, roadmap confirmation).**
- **A5 — link/image URL allowlist = `http:`, `https:`, `mailto:` (links), `/`-rooted + `./`/`../` relative paths, `#fragment`, and `data:image/*` (images). Everything else renders as inert text, never a live element (owner-confirmed 2026-10-14, roadmap confirmation).**
- **A6 — `[![alt](src)](url)` renders as a link wrapping the image (an explicit rule ahead of the two others), not as text with a stray bracket (assistant decision, surfaced here so the executor does not re-decide it).**
## Commit
```bash
git add frontend/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(ui): render markdown links, inline images, and h5/h6 headings in the shared renderer"
```
@@ -0,0 +1,27 @@
# Task 01 — The link transform in the shared renderer
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:4` — "Markdown [links](some-address) aren't being render correctly and show up as plain text"
**Story:** n/a
## Objective
`renderMarkdown` turns `[text](url)` into `<a class="md-link" …>text</a>` in the shared `inline()` chain — through an explicit URL allowlist, escape-first, with every existing transform's output byte-identical.
## Work
1. `frontend/assets/markdown.js` — in `const inline = (s) => …`, insert one link `.replace()` **after** the `*em*` rule and **before** the `^### ` heading rule: `/(\[)([^\]]*)(\]\()([^)\s]+)(\))/g` written plainly as `/\[([^\]]*)\]\(([^)\s]+)\)/g` (empty link text allowed, no whitespace in the URL). The replacement is a FUNCTION, not a string, so the URL passes the allowlist check:
- `allowLinkUrl(url)` — returns the url for `^https?://`, `^mailto:`, `/`-rooted (`^/`), `^\.{1,2}/` relative, and `^#` fragment URLs; returns `null` for anything else (`javascript:`, `data:`, `vbscript:`, `file:`, `blob:`) — LOCKED A5;
- allowed → `<a class="md-link" href="<url>"[ target="_blank" rel="noopener noreferrer"]>text</a>`; the `target`/`rel` pair is added ONLY for `^https?://` (a `mailto:`/relative/fragment link stays in the same tab);
- not allowed → return the ORIGINAL matched substring untouched, so it renders as the inert escaped text it is today (never a live element, never dropped).
2. `frontend/assets/markdown.js` — pass the captured URL through VERBATIM: the chain's input is already `escapeHtml`-ed, so `&amp;`/`&#39;`/`&quot;` in a query string are already attribute-safe. No `decodeURIComponent`, no manual unescape, no re-escape (design D2 — this is the security boundary).
3. `frontend/assets/markdown.js` — module docstring: add the phase-180 paragraph (the chain order is now code → strong → em → **linked-image → image → link** (tasks 02 adds the first two ahead of this one) → headings → lists), the allowlist (LOCKED A5), the `target`/`rel` rule, and the documented limits (a URL containing a literal `)` or a space is a miss; inline code spans are not placeholder-protected, so markdown syntax inside `` `…` `` transforms — the pre-existing `**strong**` behaviour, unchanged here).
- ASSUMPTION: link text may itself contain inline markup (`**bold**`, `*em*`, `` `code` ``) — those rules run FIRST in the chain, so the text is already transformed when the link rule wraps it; no extra handling.
- ASSUMPTION: autolinks (`<https://x>`) and reference-style links (`[text][ref]`) stay out of scope — the TODO names the inline `[text](url)` form only.
## Testing & Quality
- Unit: pinned by task 05's file (source pins: the link rule sits between the `em` and heading rules, the marker is exactly `<a class="md-link"`, the file contains no `decodeURIComponent`/`unescape`; node behaviour: absolute link with `target="_blank" rel="noopener noreferrer"`, `mailto:`/relative/`#` link without them, `javascript:` + `data:` inert, `?a=1&b=2` keeping `&amp;`, an empty link text, a link inside a table cell).
- Coverage: n/a (frontend) — the suite + the pins are the gate.
## Completion Criteria
- [ ] node repro: `renderMarkdown("see [the docs](https://example.com/a?x=1&y=2)")` → a `<p>` containing `<a class="md-link" href="https://example.com/a?x=1&amp;y=2" target="_blank" rel="noopener noreferrer">the docs</a>`
- [ ] node repro: `renderMarkdown("[click](javascript:alert(1))")` contains no `<a` and no `href`
- [ ] every other existing construct (`strong`, `em`, `code`, `#`–`###`, lists, tables, fences, blockquotes) renders byte-identical to the pre-task renderer
- [ ] `uv run pytest` green (the renderer-dependent unit suites unedited); `uv run ruff check .` clean
@@ -0,0 +1,27 @@
# Task 02 — The image transform (and the linked-image rule) in the shared renderer
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:3` — "Need SVG render support on the front end when citing … Also would be nice to click an svg and see a preview in the RAG tab rather than the xml" (the in-answer rendering half; the document-preview half is phase `181_svg_document_preview`)
**Story:** n/a
## Objective
`renderMarkdown` turns `![alt](src)` into a lazy `<img class="md-img">` (and `[![alt](src)](url)` into that image wrapped in a link), through the same URL allowlist as task 01 — so a diagram referenced inside markdown paints instead of printing its reference text.
## Work
1. `frontend/assets/markdown.js` — in the `inline()` chain, insert TWO rules **ahead of** task 01's link rule, in this order:
1. **linked image first:** `/\[!\[([^\]]]*)\]\(([^)\s]+)\)\]\(([^)\s]+)\)/g` → `<a class="md-link" href="url"[ target="_blank" rel="noopener noreferrer"]><img class="md-img" src="src" alt="alt" loading="lazy"></a>` (LOCKED A6 — without this rule the image rule consumes the inner `![…](…)` and the link rule then leaves a stray `[` in the link text). Both URLs are allowlist-checked (`allowLinkUrl` for the href, `allowImageUrl` for the src); either failing → the original text is returned untouched.
2. **image second:** `/!\[([^\]]*)\]\(([^)\s]+)\)/g` → `<img class="md-img" src="src" alt="alt" loading="lazy">`.
2. `frontend/assets/markdown.js` — add `allowImageUrl(url)` next to task 01's `allowLinkUrl`: allows `/`-rooted served paths (`/api/documents/<id>/image`, `/assets/…`), `^.{0,2}/` relative paths, `^https?://` and `^data:image/` — the `data:` case exists because the app's own CSP (`app/core/security_headers.py::CSP`, `img-src 'self' data:`) already permits `data:` images; everything else (`javascript:`, `file:`, non-image `data:`) returns `null` → the original escaped text stays visible and inert (LOCKED A5).
3. `frontend/assets/markdown.js` — `alt` is ALWAYS emitted, escaped-passthrough like the URL, empty allowed (`alt=""` for a decorative image) — an `<img>` with no `alt` attribute fails WCAG 1.1.
4. `frontend/assets/markdown.js` — docstring: record the final chain order (code → strong → em → **linked-image → image → link** → headings → lists), the image allowlist, and the documented limit that a size-less SVG (`viewBox` only) in `.md-img` paints at the replaced-element default box (vector sizing is phase 181's `04_svg_preview_render.md` contract).
- ASSUMPTION: no `onerror` handler is added — the renderer emits an HTML STRING into `innerHTML`-style surfaces and the no-CDN/CSP posture forbids inline handlers; a broken src degrades to the browser's alt text (the house pattern for the shared renderer). The JS `onerror` fallbacks live in `document.js`/`app.js`, which do not route through this function.
- ASSUMPTION: title syntax (`![alt](src "title")`) is a miss (the URL capture forbids whitespace) — out of scope.
## Testing & Quality
- Unit: pinned by task 05 (chain order pins the two image rules before the link rule; markers `<img class="md-img"` / `loading="lazy"`; `alt=` always present; node behaviour: served-path image, `data:image/*` image, disallowed-protocol image inert, linked image produces `<a class="md-link">` wrapping `<img class="md-img">` with no stray bracket, empty alt allowed, an image inside a table cell renders).
- Coverage: n/a (frontend).
## Completion Criteria
- [ ] node repro: `renderMarkdown('![topology](/api/documents/aaaa/image)')` → `<p><img class="md-img" src="/api/documents/aaaa/image" alt="topology" loading="lazy"></p>`
- [ ] node repro: `renderMarkdown('[![d](/a.svg)](https://example.com)')` → one `<a class="md-link"` containing one `<img class="md-img"`, and the output contains no stray `[` or `]`
- [ ] node repro: `renderMarkdown('![x](javascript:alert(1))')` contains no `<img` and no `src=`
- [ ] input with no image reference renders byte-identical; `uv run pytest` green
@@ -0,0 +1,27 @@
# Task 03 — Extend the heading ladder to `####`, `#####`, `######`
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:5` — "Quad-level headers (####) aren't being rendered and show up as plain text"
**Story:** n/a
## Objective
The ladder renders `#### x` as `<h5>x</h5>` and `##### x` / `###### x` as `<h6>x</h6>`; `#`, `##` and `###` keep their existing output exactly.
## Work
1. `frontend/assets/markdown.js` — in the `inline()` chain, insert TWO rules immediately BEFORE the existing `/^### (.*)$/gm → "<h4>$1</h4>"` rule (deeper-first, so the ladder reads top-down by depth):
- `/^#### (.*)$/gm` → `"<h5>$1</h5>"`
- `/^#{5,6} (.*)$/gm` → `"<h6>$1</h6>"` (LOCKED A4: both `#####` and `######` map to `<h6>` — the app's type scale stops at two levels below `###`, no `<h7>` exists in HTML)
The rules cannot cross-match each other or the existing ones because every heading rule requires a space after its hash run (`####` never matches `/^### /`).
2. `frontend/assets/markdown.js` — leave `/^## /` and `/^# /` mapping to `<h3>` UNTOUCHED (LOCKED A4 — the existing flattening of `#`/`##` to `<h3>` is deliberate; the phase adds levels below `###`, it does not re-rank the existing ones).
3. `frontend/assets/markdown.js` — the step-3 paragraph guard `/^<(h\d|ul|ol|pre|li|blockquote)/` already matches `h5`/`h6`, so NO change is needed there: pin it with a unit assertion rather than editing it.
4. `frontend/assets/markdown.js` — docstring: record the ladder (`#`/`##` → `h3`, `###` → `h4`, `####` → `h5`, `#####`/`######` → `h6`) so the deliberate flattening is documented, not re-discovered.
- ASSUMPTION: a heading line must start at column 0 of its line (the `m` flag, the existing rule shape) — an indented `#### x` stays plain text, matching `^### ` today.
## Testing & Quality
- Unit: pinned by task 05 (source pins: the two new rules sit before `/^### /`; the guard regex unchanged; node behaviour: `#### A` → `<h5>A</h5>`, `##### A`/`###### A` → `<h6>A</h6>`, `### A` → `<h4>A</h4>` unchanged, `## A`/`# A` → `<h3>` unchanged, a `####` line inside a fence stays code, a `####` heading immediately followed by a paragraph is not wrapped in `<p>`).
- Coverage: n/a (frontend).
## Completion Criteria
- [ ] node repro: `renderMarkdown("#### Deep")` → `<h5>Deep</h5>` (no `<p>` wrapper, no literal `####`)
- [ ] node repro: `renderMarkdown("##### Deeper\n###### Deepest")` → two `<h6>` elements
- [ ] node repro: a `####` line inside a ``` fence renders inside `<pre><code>` unchanged
- [ ] existing heading output byte-identical; `uv run pytest` green (phase-44/138 suites unedited)
@@ -0,0 +1,26 @@
# Task 04 — Styling for `.md-link`, `.md-img` and the `h5`/`h6` type scale
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:4–5` (link + heading rendering), `TODO.md:3` (inline image rendering)
**Story:** n/a
## Objective
The three new elements are legible and theme-correct in every renderer context (chat bubble — brain and user, the thinking block, the shared page, the viewer, the document modal), built from the existing `:root` palette only.
## Work
1. `frontend/assets/styles.css` — `.md-link` next to the `.md-quote` family (L561-588): `color: var(--brand-ink)` (the file's documented AA-safe brand text token, 9.0:1 on `--surface`), `text-decoration: underline` (a link must not rely on colour alone — WCAG 1.4.1), `text-underline-offset: 2px`, `:hover { color: var(--brand-hover) }`. The global `:focus-visible { outline: 3px solid var(--brand) }` (L162-166) already covers the link — do NOT add a competing focus rule, and do not reset the outline.
2. `frontend/assets/styles.css` — `.msg.user .bubble .md-link { color: var(--bg); }` — the exact inversion the `.md-quote` precedent applies at L588 (the user bubble is a `--brand` fill: `--bg` on `--brand` is 5.2:1, AA). Add the same inversion for `:hover` if the default `--brand-hover` fails contrast on the brand fill — a colour-mix toward `--bg` is acceptable, a new hue is not.
3. `frontend/assets/styles.css` — `.md-img`: `display: inline-block; max-width: 100%; height: auto; object-fit: contain;` plus the `.source-image-img` card language (`background: var(--surface)`, `border: 1px solid var(--line)`, `border-radius: var(--radius-sm)`) and `margin: 0.4rem 0`. No fixed width/height — an oversized screenshot must never widen the page.
4. `frontend/assets/styles.css` — the deep-heading type scale, ONE rule set for all renderer contexts (no chat-specific duplication): `.doc-md h5, .msg .bubble h5, .thinking-text h5 { margin: 0.8rem 0 0.3rem; font-size: 0.95rem; }` and the `h6` sibling at `0.88rem`, following the existing `.doc-md h3` / `.doc-md h4` rhythm (L5282-5283) — the scale stays monotonic (`h3` 1.15 → `h4` 1.0 → `h5` 0.95 → `h6` 0.88) and every level keeps a visible size difference from body text.
5. `frontend/assets/styles.css` — comment block above each rule in the house style: the phase number, the TODO source line, the contrast math for every non-obvious colour, and the "static content, no animation" note (the background-motion invariant).
- ASSUMPTION: `h5`/`h6` in the chat bubble get explicit sizing (they inherit UA defaults today, which are larger than `0.95rem`) so a deep heading in an answer cannot out-shout the answer text.
- ASSUMPTION: no `.md-img--vector` modifier here — vector (viewBox-only SVG) sizing belongs to `181_svg_document_preview/04_svg_preview_render.md`, which knows the document's format.
## Testing & Quality
- Unit: pinned by task 05 — the `.md-link`, `.md-img`, `h5`/`h6` rules exist and reference ONLY `var(--…)` palette tokens (no hex literals, no `rgb(`) — the established `:root`-drift pin pattern used by the blockquote/table CSS tests; the `.msg.user .bubble .md-link` inversion is present.
- Coverage: n/a (frontend).
## Completion Criteria
- [ ] a link in a brain bubble, a user bubble, the thinking block, the shared page, the viewer and the document modal is underlined and passes ≥4.5:1 against its surface
- [ ] Tab focus on a link shows the app's 3px `--brand` ring
- [ ] a wide image in an answer does not produce a horizontal scrollbar on the chat page at 375px and at 1440px
- [ ] `uv run pytest` green (the CSS-pinning unit suites unedited)
@@ -0,0 +1,36 @@
# Task 05 — Unit pins, the dedicated Playwright suite, the gates, the commit
**Phase:** `180_markdown_inline_links_images` · **Source:** `TODO.md:3–5` (the acceptance evidence for tasks 01–04)
**Story:** n/a
## Objective
Lock the three renderer fixes with a source-pin + node-behaviour unit file and the phase's own Playwright E2E suite (AGENTS rules 4/9), run every gate, and land the single atomic commit.
## Work
1. `tests/unit/test_markdown_inline_links_images.py` (new — the `tests/unit/test_markdown_blockquotes.py` shape, copy its `HAVE_NODE = shutil.which("node") is not None` skip and its `subprocess` node harness):
- **source pins on `frontend/assets/markdown.js`:** the chain order is code → strong → em → linked-image → image → link → `^#### ` → `^#{5,6} ` → `^### ` → `^## ` → `^# ` → lists (assert the index of each rule's literal in the file is strictly increasing); the exact markers `<a class="md-link"`, `<img class="md-img"`, `loading="lazy"`, `<h5>`, `<h6>`; `allowLinkUrl` / `allowImageUrl` named functions exist; the file contains NO `decodeURIComponent` / `unescape` / `innerHTML`; the step-3 guard regex is still exactly `/^<(h\d|ul|ol|pre|li|blockquote)/`.
- **CSS pins on `frontend/assets/styles.css`:** `.md-link`, `.md-img`, the `h5` and `h6` rules exist and their colour values reference `var(--…)` only (no hex, no `rgb(`); `.msg.user .bubble .md-link` exists (the inversion).
- **node behaviour (skipped without node):** every repro listed in tasks 01–04's Completion Criteria, PLUS the byte-identity contract — a checked-in `NO_INLINE_INPUT` / `NO_INLINE_PRE_PHASE_OUTPUT` fixture pair (a paragraph + a list + a pipe table + a fence + a blockquote, i.e. everything the renderer already supported) asserted byte-equal.
2. `tests/fixtures/links_docs/` (new dedicated fixture tree — the phase-138 `quote_docs` precedent: a NEW tree, never the shared 13-doc fixture set, whose import count many suites pin). One document `homelab/inline-markdown.md` whose body carries: a `[absolute link](https://example.com/a?x=1&y=2)`, a `[relative link](/sources)`, a `mailto:` link, a `#anchor`, an inert `[oops](javascript:alert(1))` probe, `![topology](data:image/svg+xml;base64,…)` (a tiny inline SVG — CSP `img-src 'self' data:` already allows it, no bytes fixture needed), an inert `![bad](javascript:alert(1))`, a `[![linked](data:…)](https://example.com)`, and an `#### Deep header` + `##### Deeper` line, plus a pipe table with a `[cell link](https://example.com)` inside a cell.
3. `tests/e2e/test_markdown_inline_links_images.py` (new — the phase-138 E2E skeleton verbatim in structure: module-level fixture constants, `_import_fixtures` + `_run_in_thread` + `_reset_db` (truncate `chunks, documents, query_log, steering_notes, kb_overview` then re-import, `summary.added == 1`), `login(page, app_url, next="/")`, the mock LLM trigger answering with the byte-stable inline-markdown answer):
- `test_chat_inline_markdown_renders` — the brain bubble contains `a.md-link` elements for the absolute (with `target="_blank"`, `rel` containing `noopener noreferrer`, `href` preserving `&amp;`), relative and `mailto:` links; NO `a.md-link` for the `javascript:` probe (its text stays visible); `img.md-img` for the data-URL image with a non-empty `alt` and `loading="lazy"`; the linked image is an `a.md-link > img.md-img` with no stray bracket in its text; `h5`/`h6` elements exist; `table.md-table a.md-link` exists (cell-level transform); the turn is NOT deflected.
- `test_viewer_inline_markdown_renders` — the fixture document opens from the Sources page in the document modal (admin, `openDocumentModal`) and renders the same `a.md-link` / `img.md-img` / `h5` / `h6` shapes (the shared renderer contract across surfaces).
- `test_restore_keeps_inline_markdown` — reload the chat page: the localStorage restore path re-runs `renderMarkdown` and the restored brain bubble keeps its links, image and headings.
- `test_no_horizontal_overflow_at_mobile` — at `viewport={"width": 375, "height": 720}`, `document.documentElement.scrollWidth <= clientWidth` with the wide image rendered (the `.md-img` `max-width: 100%` contract).
4. Run the gates, in order, and fix what fails (do not baseline, do not skip):
- `uv run pytest`
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL >90%
- `uv run pytest tests/e2e/test_markdown_inline_links_images.py -v --no-cov` (in isolation, DB up: `podman compose up -d db`)
- `uv run ruff check . && uv run pyright`
5. Move the phase directory to `.agents/phases/complete/` and commit — see the overview's Commit block.
## Testing & Quality
- This task IS the phase's test gate. The unit file must fail if any of tasks 01–03's rules are reordered or removed, and must fail if task 04's CSS introduces a hard-coded colour.
- Coverage: **>90%** on `app/` (no `app/` code changes in this phase — the number must not move).
## Completion Criteria
- [ ] `uv run pytest` green, including the new unit file (and green with node absent — the skip path)
- [ ] `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%
- [ ] `uv run pytest tests/e2e/test_markdown_inline_links_images.py -v --no-cov` green in isolation
- [ ] `uv run ruff check . && uv run pyright` clean
- [ ] one atomic commit, `--no-gpg-sign`: `feat(ui): render markdown links, inline images, and h5/h6 headings in the shared renderer`, containing `frontend/`, `tests/` and `.agents/`
@@ -0,0 +1,56 @@
# Phase 181 — SVG documents render as pictures, not as XML
**Source:** `TODO.md` L3 — "Need SVG render support on the front end when citing and svg source. Also would be nice to click an svg and see a preview in the RAG tab rather than the xml"
**Story:** n/a (TODO-derived phase, Protocol B append — owner roadmap confirmation 2026-10-14)
**Context:** **The root cause is ingestion, not the viewer.** `app/config.py::image_extensions` (L436) defaults to `png,jpg,jpeg,webp,gif,bmp` and `.env` mirrors it (`BOR_IMAGE_EXTENSIONS`), so an `.svg` can only enter the KB through `BOR_IMPORT_EXTENSIONS` — i.e. as a **text** document whose `content` is the raw XML, which `frontend/assets/document.js::renderDocument` then paints in `<pre class="doc-raw">`. That is exactly the "xml instead of a preview" the owner sees. The rest of the image machinery is already there and needs no change: `app/rag/retriever.py::source_ref_with_image` (L1010-1022) puts `image_url = /api/documents/<id>/image` on a cited image doc, `app/schemas.py::SourceRef.image_url` (L154-181) carries it (omitted when `None`), `app/api/docs.py::get_document_content` (L219-318) returns `is_image` + `image_url` + `format`, `app/api/docs.py::get_document_image` (L319-378) serves the stored bytes with `IMAGE_MIMES.get(suffix, IMAGE_FALLBACK_MIME)`, `frontend/assets/app.js::appendSources` (L2386+) renders a `.source-image` figure per `image_url` ref, `frontend/assets/document.js` renders the `.doc-image` `<img>` block for `doc.is_image` (L453-470, L560+), `frontend/assets/sources.js` (L2328) renders the KB-tree thumbnail. `app/rag/importer.py::_index_file` already prefers the image branch over the text path (L1417-1425) and `_index_image_file` (L3532+) owns bytes-digest → `_markdown_image_context` (phase 125) → `_describe_or_skip` → copy to `image_dir/<doc-id><ext>` → `is_image`/`image_path`/`content`/`summary`. `app/rag/summarizer.py::IMAGE_MIMES` (L152-159, dotted keys) is the one map for both the vision data-URL mime and the route's `Content-Type`; `describe_image` is a CHAT-model vision call (phase 122 LOCKED A3). `app/core/security_headers.py::CSP` (L67-68) + its middleware (which PRESERVES a CSP an inner layer set — L109-110) is the injection boundary. `app/rag/md_images.py::extract_image_refs` (the phase-125 `![alt](path)` scanner) already treats an `.svg` path like any other image. Test patterns: `tests/unit/test_image_documents.py` (toggle / ingest / describe-seam), `tests/integration/test_docs_api.py` (the image route), `tests/e2e/test_image_documents.py` + `tests/e2e/test_answer_images.py` (image E2E), `tests/unit/test_markdown_blockquotes.py` (source-pin + node harness).
## Objective
An `.svg` in a source tree becomes an **image document**: its own text (`<title>`/`<desc>`/`<text>`) is what gets indexed and retrieved, its bytes are served as `image/svg+xml` under a sandboxed CSP, and every citation surface — the chat figure, the document modal, the `/document.html` viewer, the Sources thumbnail — paints the rendered vector graphic instead of XML. Plus the phase-180 renderer, an SVG referenced as `![alt](/api/documents/<id>/image)` inside an answer or a document renders as a real image.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — SVG is a *vector* image document; the vision call is bypassed (tasks 01–02, locked A1).** OpenAI-compatible vision endpoints do not accept `image/svg+xml`, and adding a rasteriser dependency is out of scope. So an `.svg` takes the phase-122 image branch but its indexed text comes from the file itself: a new pure module `app/rag/svg_text.py` (the `app/rag/md_images.py` precedent — stdlib only, no I/O, fully unit-testable) returns the SVG's own words. `content`/`summary` follow the phase-122 image convention (`summary = content`) and the normal chunk pipeline embeds them; the phase-125 `image_context` (the `![alt](path)` paragraphs around a reference to the file) still prepends, unchanged.
- **D2 — the text of a diagram is its `<title>`, `<desc>` and `<text>` (task 01).** Matched by LOCAL name (namespace-agnostic: plain `<title>`, `svg:title`, and namespaced variants all count), in DOCUMENT order, whitespace-normalised, de-duplicated case-insensitively, joined with `\n`. Parsing is `xml.etree.ElementTree` — never a regex over untrusted markup — guarded against the billion-laughs/XE class by (i) refusing any input containing a `<!DOCTYPE` or `<!ENTITY` declaration (→ extraction failure, no parse attempt) and (ii) a `MAX_SVG_BYTES = 2 MiB` input cap (over-cap → extraction failure). XML entity refs inside text (`&amp;`) are decoded by the parser, which is the correct direction (the value is re-escaped on the way out by the renderer).
- **D3 — a textless SVG still gets honest, retrievable text (task 01, locked A2).** Many real diagrams carry no words at all (pure `<path>`s). Skipping them (the phase-122 "undescribed image is unsearchable noise" posture) would make them invisible and never citable — the opposite of what the owner asked. So `extract_svg_text` falls back to a structural line built ONLY from facts in the file: `Vector diagram "<file stem>" — <N> shapes (<the three most common element local names with counts>), viewBox <w>×<h>` (each clause omitted when the file does not state it, the stem always present). No invented prose, no model call.
- **D4 — ingestion mechanics (task 02).** `image_extensions` default gains `svg` (`.env.example` documents it; a live `.env` keeps its own value — the phase does not edit `.env`). `app/rag/importer.py::_index_image_file` branches on `full_path.suffix.lower() == ".svg"`: vector files take `extract_svg_text` INSTEAD of `_describe_or_skip`; everything else (copy to `image_dir`, digest rules, `is_image`/`image_path`, title = file stem, counters, the `on_chunk` hook) is unchanged and shared. Fail-soft stays LOCKED A3-shaped: extraction returning `None` (over-cap, DOCTYPE-bearing, or unparseable XML) → `summary.images_failed += 1`, a warning, no row, no orphan copy. **One extra guard is required for existing libraries:** `_index_file`'s fast path (L1403, `doc.file_hash == file_digest`) returns BEFORE the image branch, so a row that was indexed as TEXT when `svg` lived only in `BOR_IMPORT_EXTENSIONS` would stay a text doc forever even after the switch. Add a narrow mismatch guard — take the fast path only when the file's kind matches the row's (`settings.images` ∧ the extension is in `image_extension_set` ∧ `doc.is_image is False` ⇒ fall through to the image branch). Everything else keeps the phase-147 fast path byte-for-byte.
- **D5 — serving the bytes safely (task 03).** `IMAGE_MIMES[".svg"] = "image/svg+xml"` in `app/rag/summarizer.py` (one map, one truth — the vector branch never reaches `image_data_url`, so the vision path is unaffected). `get_document_image` keeps its public, id-as-credential posture (phase 124 LOCKED A4 — a shared chat shows the figure to a guest) and its `Cache-Control: private, max-age=3600`, and adds ONE response header for `image/svg+xml` only: `Content-Security-Policy: default-src 'none'; sandbox` (the `SecurityHeadersMiddleware` preserves an inner CSP, so the route's value wins) — an `<img>` can never execute SVG script, but a user who opens the URL directly must not get a script-capable same-origin document either. `X-Content-Type-Options: nosniff` is already global.
- **D6 — the frontend (task 04).** The modal, the viewer, the citation figure and the Sources thumbnail already key off `is_image`/`image_url`, so they get the SVG for free — the task is the sizing + honesty work: `.doc-image-img--vector` (applied in `document.js` where `doc.format === "svg"`) with `width: 100%` so a `viewBox`-only SVG (no intrinsic size, the CSS default replaced-element box is 300×150) fills the `.doc-image` card instead of sitting small in it; the onerror `.doc-image-unavailable` fallback stays as-is; a legacy SVG row still stored as TEXT keeps rendering its XML in `.doc-raw` (honest — there are no bytes behind it) until the next sync converts it.
- **D7 — the chat figure is a 96px-capped thumbnail** (`.source-image-img`, `max-height: 96px; max-width: 100%; height: auto`), so a size-less SVG inside it stays inside the cap; `SourceRef` carries no format field and NONE is added (the wire surface stays byte-identical for every text doc — phase-83/122 contract). Clicking the figure/chip opens the modal, which is the full-size preview (`doc.format` known there). Documented in the module comments.
- **NOT touched:** the CSP constant in `security_headers.py`, `app/api/chat_images.py::_CHAT_IMAGE_FILENAME_RE` (question attachments are raster-only by design — a vector question image is out of scope), any rasteriser dependency, `describe_image`'s prompt, the phase-124 answer-image path, the `SourceRef` schema, `.env` (user-owned), the PDF embedded-image pass (`pdf.py` — PDFs never embed SVG for this app's extractor).
## Dependencies
- `180_markdown_inline_links_images` (todo) — its `![alt](src)` transform is what makes `![alt](/api/documents/<id>/image)` inside an answer or a document render; this phase supplies the route that answers that URL. No code-level dependency beyond that (the ingestion, the route and the viewer work without it).
## Tasks
1. `01_svg_text_extraction.md` — the new pure `app/rag/svg_text.py` module: `<title>`/`<desc>`/`<text>` extraction with the XE/size guards and the structural fallback.
2. `02_svg_index_as_image.md` — `svg` in `image_extensions`, the vector branch in `_index_image_file`, and the fast-path kind-mismatch guard that converts existing text-SVG rows.
3. `03_svg_bytes_route_csp.md` — `IMAGE_MIMES[".svg"]` and the sandboxed `image/svg+xml` response from `GET /api/documents/{id}/image`.
4. `04_svg_preview_render.md` — the vector sizing class + CSS in the viewer, and the figure/modal/thumbnail verification (the "preview instead of XML" contract).
5. `05_unit_integration_e2e.md` — `tests/unit/test_svg_text_extraction.py`, `tests/integration/test_svg_document_image.py`, the phase's dedicated Playwright suite `tests/e2e/test_svg_document_preview.py`, the gates, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_svg_text_extraction.py` (new, pure-function, no DB): title+desc+text in document order, namespace-prefixed variants, whitespace normalisation, de-duplication, `&amp;` decoded, the `<!DOCTYPE`/`<!ENTITY` refusal, the 2 MiB cap, malformed XML → `None`, the structural fallback for a textless SVG (stem + shape counts + viewBox, and each clause absent when unknown). Importer-level: the vector branch indexes an SVG with NO vision call (the `_describe_or_skip` seam patched to FAIL the test if called), `content == extracted text`, `summary == content`, the copy lands at `image_dir/<id>.svg`, `is_image=True`, the `images_failed` fail-soft path, and the fast-path conversion of a pre-existing text row (same bytes, `file_hash` unchanged).
- **Integration:** `tests/integration/test_svg_document_image.py` (new): `GET /api/documents/{id}/image` for an SVG row → `200`, `Content-Type: image/svg+xml`, the sandbox CSP header present, `Cache-Control: private, max-age=3600`; a raster row's response headers UNCHANGED (no CSP key added); the four 404 shapes (unknown id, unparseable id, text doc, missing file on disk) unchanged.
- **E2E (AGENTS rule 4):** `tests/e2e/test_svg_document_preview.py`, run **in isolation** (`uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov`, DB up).
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`) — `app/rag/svg_text.py` is new pure code and must be covered near-completely; the importer/route deltas are covered by the tests above.
- **Lint/types:** `uv run ruff check . && uv run pyright` clean.
## Completion Criteria
- [ ] an `.svg` in a source tree imports as `is_image` with `image_path` set, no vision call made, and its indexed text drawn from its own `<title>`/`<desc>`/`<text>`
- [ ] `GET /api/documents/<id>/image` serves `image/svg+xml` with `Content-Security-Policy: default-src 'none'; sandbox`
- [ ] clicking an SVG citation chip/figure in the chat opens the modal showing a rendered `<img>` — no XML text anywhere in the modal body
- [ ] the `/document.html` viewer renders the SVG full-width in its `.doc-image` card (not the 300×150 default box) with the extracted text below it
- [ ] a `![alt](/api/documents/<id>/image)` reference inside a markdown document paints the graphic (the phase-180 renderer + this route — the E2E's two-stage seeding, `test_markdown_image_ref_renders`)
- [ ] every raster/PDF image behaviour is unchanged (the phase-122/123/124/133/140 suites pass unedited)
- [ ] `uv run pytest` green; coverage TOTAL >90%; the phase's E2E green in isolation; `uv run ruff check . && uv run pyright` clean
- [ ] one `--no-gpg-sign` Conventional Commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate
## Locked decisions
- **A1 — an SVG's indexed text is extracted from the file itself (`<title>`/`<desc>`/`<text>`), the vision describe call is bypassed for `.svg`; no rasteriser dependency is added (owner-confirmed 2026-10-14: "SVG as image document").**
- **A2 — a textless SVG is indexed with a structural metadata line built only from facts in the file (stem, shape counts, viewBox) rather than skipped, so it stays retrievable and previewable (assistant decision made explicit here; it relaxes the phase-122 fail-soft skip for the vector case only — a genuinely unparseable or over-cap file still skips).**
- **A3 — SVG bytes render ONLY through `<img>`, never inlined into the DOM, and the bytes route carries `default-src 'none'; sandbox` so a direct navigation cannot execute script (owner-confirmed 2026-10-14, roadmap confirmation).**
- **A7 — `svg` joins the `BOR_IMAGE_EXTENSIONS` DEFAULT; the `BOR_IMAGES` toggle remains the single knob (phase 122 LOCKED A3). With `BOR_IMAGES=0` an `.svg` stays a text document (raw XML in the viewer) and the walk never admits it as an image — no special-casing of the off state (owner-confirmed 2026-10-14, roadmap confirmation).**
## Commit
```bash
git add app/ frontend/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "feat(kb): index SVGs as image documents and preview them instead of raw XML"
```
@@ -0,0 +1,33 @@
# Task 01 — `app/rag/svg_text.py`: the words inside a vector drawing
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` — "Need SVG render support on the front end when citing … source" (the indexed-text half: an SVG document must be retrievable without a vision call)
**Story:** n/a
## Objective
A new pure module turns SVG bytes into the text that makes the document searchable — its own `<title>`/`<desc>`/`<text>` words, or an honest structural line when it has none — with no third-party dependency, no I/O, and no network.
## Work
1. `app/rag/svg_text.py` (new — the `app/rag/md_images.py` module shape: module docstring citing this phase + the LOCKED decisions, `from __future__ import annotations`, stdlib only, full type hints, pure functions):
- `MAX_SVG_BYTES: int = 2 * 1024 * 1024` and `MAX_SVG_TEXT_CHARS: int = 4000` (module constants, documented — the cap binds on the EXTRACTED text, never silently mid-word: truncate at the last newline/period before the cap).
- `@dataclass(frozen=True) class SvgText:` — `text: str` (the searchable text) and `structural: bool` (`True` when the fallback line was used, so the importer can log/count it).
- `def extract_svg_text(data: bytes, *, stem: str = "") -> SvgText | None`.
2. `app/rag/svg_text.py` — the guards, BEFORE any parse (LOCKED A3's sibling: untrusted XML):
- empty/whitespace-only input → `None`;
- `len(data) > MAX_SVG_BYTES` → `None` (the billion-laughs/memory guard, alongside the next one);
- the lower-cased bytes containing `<!doctype` or `<!entity` → `None` (a document type declaration is refused outright, which is what makes stdlib `ElementTree` safe here — no external entities, no entity expansion);
- `ElementTree.fromstring` raising `ParseError` → `None` (never re-raise: the caller's fail-soft is a skip, not a crash).
3. `app/rag/svg_text.py` — extraction: a `local_name(tag)` helper that strips the `{namespace}` prefix, then walk `root.iter()` (document order) and collect, for local name `title` / `desc` → the element's `.text` block (`"".join(el.itertext())` so nested markup is carried), and for `text` → `" ".join(el.itertext())` (this is what covers `<tspan>` children). Normalise each item with `" ".join(item.split())`, drop empties, de-duplicate case-insensitively keeping the FIRST occurrence, cap at `MAX_SVG_TEXT_CHARS`, and return `SvgText("\n".join(items), structural=False)` when at least one item survived (parser-decoded entities like `&amp;` come out decoded — the renderer re-escapes on display).
4. `app/rag/svg_text.py` — the structural fallback (LOCKED A2), used only when the walk produced nothing: `Vector diagram "<stem>" — <N> shapes (<top-3 local names by count>), viewBox <w>×<h>` built ONLY from the file — `N` counts descendant elements excluding `svg`/`title`/`desc`; the `shapes` clause is omitted when `N == 0` and the noun pluralises normally (`1 shape`, `2 shapes`); the `viewBox` clause is omitted when the root has no parseable `viewBox` (four whitespace/comma-separated numbers; the last two are `w`/`h`) and when present reports `w×h` with integral values formatted without a trailing `.0`; the stem is `""` → the clause `Vector diagram` alone (no empty quotes). The em-dash form matches the app's own doc-comment typography and is the string the tests pin.
5. `app/rag/svg_text.py` — module docstring: why the vision call is bypassed (LOCKED A1: vision endpoints reject `image/svg+xml`; no rasteriser dependency), the guard list, the document-order/dedupe rules, and the fallback contract (LOCKED A2: facts from the file, never invented prose).
- ASSUMPTION: `<image>` elements embedding a raster inside the SVG are NOT followed (their `href` is not text; the byte-embedded raster belongs to the phase-151 PDF path, not here).
- ASSUMPTION: `aria-label`/`role` attributes are not extracted — `<title>`/`<desc>` are the SVG-spec carriers and the fallback covers the rest.
## Testing & Quality
- Unit: `tests/unit/test_svg_text_extraction.py` starts here (completed in task 05) — every branch above is a pure-function test with a literal SVG string, no DB, no LLM, no filesystem: document order across `title`/`desc`/`text`, `tspan` joining, namespace-prefixed tags (`<svg:title>`, `xlink:` attributes ignored), whitespace normalisation, case-insensitive dedupe, `&amp;` decoded, the `<!DOCTYPE`/`<!ENTITY` refusals (both a `<!DOCTYPE svg [<!ENTITY x "boom">]>` wrapper and a bare `<!ENTITY`), the over-cap refusal, malformed XML → `None`, and the fallback (shape counts + viewBox present/absent/malformed, empty stem).
- Coverage: **>90%** on this module (target: near-100% — it is pure).
## Completion Criteria
- [ ] `uv run python -c "from app.rag.svg_text import extract_svg_text; print(extract_svg_text(b'<svg viewBox=\"0 0 8 6\"><title>Top</title><text>a</text></svg>', stem=\"topo\"))"` reports the two extracted items in order with `structural=False`
- [ ] a `<svg>` with only `<path>`s yields `Vector diagram "topo" — 1 shape (path)`-shaped text with `structural=True`
- [ ] `extract_svg_text(b'<!DOCTYPE svg [<!ENTITY x "y">]><svg><title>&x;</title></svg>')` returns `None`
- [ ] `uv run pytest tests/unit/test_svg_text_extraction.py -v` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,38 @@
# Task 02 — SVG joins the image set and indexes as a vector document
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` — "Need SVG render support on the front end when citing … source. Also would be nice to click an svg and see a preview … rather than the xml"
**Story:** n/a
## Objective
`svg` is an image extension, and `_index_image_file` indexes an `.svg` from its bytes + extracted text instead of a vision call — so the row arrives with `is_image=True`, `image_path` set and searchable content. A pre-existing text-indexed SVG row is converted on the next sync.
## Work
1. `app/config.py` (L436) — `image_extensions: str = "png,jpg,jpeg,webp,gif,bmp"` → `"png,jpg,jpeg,webp,gif,bmp,svg"`; extend the field's doc-comment: SVG is the one VECTOR member (LOCKED A1 — no vision call, `app.rag.svg_text` supplies its text) and `BOR_IMAGES` remains the single knob (LOCKED A7). The existing `_image_extensions_known` validator (L659-673) needs no change (`svg` is a well-formed token); `image_extension_set` (L899-909) yields `.svg`.
2. `.env.example` (the `# BOR_IMAGE_EXTENSIONS=…` line ~L167) and `README.md` (L794 prose, the L1239 settings table row) — add `svg` to the documented default, one sentence on the vector behaviour (indexed from its own `<title>`/`<desc>`/`<text>`; no vision call) and one on the `BOR_IMAGES=0` consequence (LOCKED A7: an `.svg` stays a text doc — raw XML in the viewer). Never touch the live `.env` (user-owned).
3. `app/rag/importer.py::_index_image_file` (L3532+) — the vector branch, at the `_describe_or_skip` call site (L3632) ONLY:
```
if full_path.suffix.lower() == ".svg":
extracted = extract_svg_text(full_path.read_bytes(), stem=full_path.stem)
content = None if extracted is None else extracted.text
else:
content = await _describe_or_skip(...) # unchanged, byte-identical
```
Keep everything after it exactly as it is — the `images_failed += 1` + warning + `return` fail-soft when `content is None` (LOCKED A2: only a genuinely unusable file skips; the structural fallback means a textless SVG does NOT), the `image_context` prepend (`f"{image_context}\n\n{content}"`, phase 125 — unchanged), the digest over the raw bytes, the `image_dir/<doc-id>.svg` copy written only after success, `title = full_path.stem`, `is_image=True`, `image_path`, `summary = content`, the chunk pipeline and the `on_chunk` hook.
4. `app/rag/importer.py` — import `extract_svg_text` next to the existing `app.rag.summarizer` imports (L356-359) and update the module docstring's image paragraph: the image branch is raster (vision-described) OR vector (`svg`, text-extracted, no model call).
5. `app/rag/importer.py` — a log-only `svg_indexed: int = 0` counter on `ImportSummary` (docstring: log-only, the phase-133 stats convention — the sync `detail` API shape is untouched) incremented on a successful vector index, plus one `logger.info("import: svg indexed source=%s path=%s structural=%s chars=%d", …)` line so a run's SVG handling is auditable (AGENTS rule 10).
6. `app/rag/importer.py::_index_file` — the fast-path kind guard (design D4). At the `if doc is not None and doc.file_hash == file_digest:` fast path (L1403), take it only when the row's KIND already matches how the file would be indexed now: compute the would-be image membership once (`settings.images` ∧ `match_extension(full_path, settings.image_extension_set) is not None`) and require `doc.is_image` to agree. That converts a row imported as TEXT (before `svg` joined the image set) — same bytes, same `file_hash` — into the image branch instead of returning early, while every correctly-classified file keeps the phase-147 fast path byte-for-byte. Add the explanatory comment (this is the migration; no alembic migration is needed — `is_image` flips through the normal index write).
- The reverse direction (a raster row when `BOR_IMAGES` is off) must NOT be forced through the image path (the walk never admits it and the phase-122 prune guard protects it) — the guard only applies when `settings.images` is on.
- ASSUMPTION: no dedicated backfill script (the phase-118/138 backfill pattern) — the next scheduled sync converts every SVG through the guard in step 6, which is idempotent and needs no new entrypoint.
- ASSUMPTION: an SVG in BOTH `BOR_IMPORT_EXTENSIONS` and `BOR_IMAGE_EXTENSIONS` becomes an image document — `_index_file` already prefers the image branch (L1417-1425) over the text path, so no walk-precedence change is made.
## Testing & Quality
- Unit (`tests/unit/test_image_documents.py` extended, or a new `tests/unit/test_svg_image_ingest.py` if the file's scope reads better — the executor's call, stated in the commit body): the vector branch indexes an SVG while the `_describe_or_skip` seam is patched to raise (proving NO vision call), `content` equals the extracted text, `summary == content`, one `is_summary` chunk, `is_image=True`, `image_path` ends with `.svg` inside `image_dir`, `svg_indexed == 1`; extraction failure (over-cap bytes) → `images_failed == 1`, no row, no copy; the `image_context` enrichment prepends for an SVG exactly as for a raster; the fast-path guard — a row with `is_image=False` and a matching `file_hash` is RE-indexed as an image on the next run, while an already-correct row (raster `is_image=True`, same hash) still takes the fast path (assert `_describe_or_skip` is NOT called).
- Regression: `tests/unit/test_image_documents.py`, `tests/integration/test_importer.py`, `test_import_docs_git.py`, `test_importer_dates.py` stay green — no raster/PDF behaviour changes.
- Coverage: **>90%** on the changed lines.
## Completion Criteria
- [ ] `Settings().image_extension_set` contains `.svg` with no env override
- [ ] importing a tree containing one `.svg` creates a row with `is_image=True`, an `image_path` under `image_dir`, and content from the SVG's own text — with the vision seam never invoked
- [ ] a second sync over the unchanged file reports it `unchanged` (the guard fires once, the fast path resumes)
- [ ] a pre-existing text-SVG row flips to `is_image=True` on the next sync (fixture test)
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,31 @@
# Task 03 — Serve the bytes as sandboxed `image/svg+xml`
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` — "…see a preview … rather than the xml" (the transport half: something must answer `/api/documents/<id>/image` with SVG the browser will paint, safely)
**Story:** n/a
## Objective
`GET /api/documents/{doc_id}/image` serves an SVG row as `image/svg+xml` behind a `default-src 'none'; sandbox` CSP, while every raster response keeps its current bytes, headers and 404 semantics unchanged.
## Work
1. `app/rag/summarizer.py::IMAGE_MIMES` (L152-159) — add `".svg": "image/svg+xml"`. Extend the map's doc-comment: SVG is the one vector entry, and the vector import branch (task 02) never routes through `image_data_url`, so this entry exists for the BYTES route only (`app/api/docs.py::get_document_image` reads the same map — one map, one truth).
2. `app/api/docs.py::get_document_image` (L319-378) — keep the lookup, the public id-as-credential posture (phase 124 LOCKED A4 — a shared chat shows the figure to a guest), the four 404 shapes (unknown id, unparseable id, text doc, missing file) and the `Cache-Control: private, max-age=3600`. Add ONE conditional header next to `media_type`:
```
headers = {"Cache-Control": "private, max-age=3600"}
if media_type == "image/svg+xml":
headers["Content-Security-Policy"] = "default-src 'none'; sandbox"
```
(LOCKED A3.) Raster responses gain NO new header — their header set is byte-identical.
3. `app/api/docs.py` — docstring on the route: why SVG needs the extra header — an `<img>` can never execute SVG script, but the URL is directly navigable, and without the policy a stored SVG would be a script-capable same-origin document; `sandbox` blocks scripting AND makes the response an opaque origin (no same-origin fetch/form capability), `default-src 'none'` blocks every subresource. Note that `app/core/security_headers.py::SecurityHeadersMiddleware` PRESERVES an inner CSP (L109-110), so this route's value is the one the browser sees, and that the middleware's global `X-Content-Type-Options: nosniff` still applies.
4. `README.md` — the security-headers section (where phase 82/123's CSP is described) gains one line: SVG document bytes are served sandboxed, with the one-sentence rationale.
- ASSUMPTION: no `Content-Disposition: attachment` — the graphic must render inline; `sandbox` is the stronger and less intrusive control.
- ASSUMPTION: the CSP is scoped to `image/svg+xml` (derived from the extension map, never from a client value) rather than applied to all image types — a raster needs no policy change, and adding one would risk the phase-123/124 image surfaces.
## Testing & Quality
- Integration: `tests/integration/test_svg_document_image.py` (new, task 05 completes it): an SVG row → `200`, `content-type == "image/svg+xml"`, `content-security-policy == "default-src 'none'; sandbox"`, `x-content-type-options == nosniff`, `cache-control == "private, max-age=3600"`, body bytes byte-equal to the stored copy. A raster (`.png`) row → NO `content-security-policy` key in the response headers and the existing headers unchanged. The 404 shapes: unknown uuid, a non-uuid string (`404`, not 422), a text doc, an image row whose copy was deleted from disk.
- Unit: `IMAGE_MIMES` contains `.svg` and the existing entries are untouched (the map is asserted by identity, not membership, so a silent removal fails).
- Coverage: **>90%** on the changed lines.
## Completion Criteria
- [ ] `curl -sD- -o/dev/null http://127.0.0.1:8000/api/documents/<svg-doc-id>/image` shows `content-type: image/svg+xml` and `content-security-policy: default-src 'none'; sandbox`
- [ ] the same request for a PNG row shows no `content-security-policy` header (and `content-type: image/png`)
- [ ] `uv run pytest tests/integration/test_svg_document_image.py tests/integration/test_docs_api.py -v` green; `uv run ruff check . && uv run pyright` clean
@@ -0,0 +1,28 @@
# Task 04 — The preview instead of the XML (viewer, figure, modal, thumbnail)
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` — "Also would be nice to click an svg and see a preview in the RAG tab rather than the xml"
**Story:** n/a
## Objective
An SVG document paints as a graphic at readable size in the document modal and the `/document.html` viewer, keeps its compact figure in the chat citation row, and keeps its Sources thumbnail — no XML text anywhere on those surfaces.
## Work
1. `frontend/assets/document.js::docImageBlock` (L571-591) — one addition: when `doc.format === "svg"`, set the img's class to `doc-image-img doc-image-img--vector` (keep the existing base class so every current rule still applies). Rationale to put in the comment: an SVG carrying only a `viewBox` has NO intrinsic size, so the replaced-element default box (300×150) would sit small inside the full-width `.doc-image` card; `doc.format` is the only place in the app that knows the type (the bytes URL is extension-less and `SourceRef` carries no format field by design — design D7). Do NOT inline the SVG into the DOM (LOCKED A3): `img.src = doc.image_url` stays `createElement` + property assignment, which is already the file's XSS contract.
2. `frontend/assets/styles.css` (next to `.doc-image-img`, L5623-5630) — `.doc-image-img--vector { width: 100%; height: auto; object-fit: contain; }` with the house comment (the no-intrinsic-size explanation, why `object-fit: contain` protects `max-height: 70vh` from squashing the drawing, static content/no animation). Palette-neutral: no new colour.
3. `frontend/assets/app.js::appendSourceImageFigure` (~L2485+) — NO structural change: the figure already keys off `image_url`, already carries the summary caption and the chip's navigation (the click → `openDocumentModal` path is the "preview in the RAG tab" the owner asked for). Add the comment recording the deliberate non-change: a size-less SVG inside the 96px-capped `.source-image-img` scales inside the cap, and the full-size preview is the modal (no `format` field is added to `SourceRef` — the wire contract stays byte-identical for text docs).
4. `frontend/assets/sources.js` (the `d.is_image` thumbnail branch, ~L2328) — verify the KB-tree thumbnail renders for an SVG row through the same `image_url` and add the same one-line comment; only touch the code if the E2E proves a real gap (report the finding in the commit body either way).
5. Legacy honesty — do NOT special-case it: a row still indexed as TEXT (an `.svg` under `BOR_IMAGES=0`, or not yet re-synced) keeps rendering its XML in `.doc-raw`, which is the truth about that row (task 02's fast-path guard converts it on the next sync). Comment this in `docImageBlock` so nobody "fixes" it by sniffing the content string.
- ASSUMPTION: no lightbox/zoom affordance is added — the modal at full width IS the preview the owner asked for; zoom is a separate feature request.
- ASSUMPTION: no "download the SVG" control — the bytes URL is directly openable and the existing "Copy path" control already covers provenance.
## Testing & Quality
- Unit: `tests/unit/test_document_viewer.py` extended — `docImageBlock` for `format === "svg"` yields `class="doc-image-img doc-image-img--vector"` with `src` = `image_url` and `alt` = summary (title fallback); for `format === "png"` the class string is unchanged (the base class only); the CSS pins `.doc-image-img--vector` exists, references no colour literal, and keeps `object-fit: contain`.
- E2E (task 05): the modal + viewer assertion that an SVG row renders an `<img>` with `naturalWidth > 0` and that the modal body contains no `<` XML text and no `<pre class="doc-raw">`.
- Coverage: n/a (frontend) — the unit pins + the E2E are the gate.
## Completion Criteria
- [ ] clicking an SVG source chip in a chat answer opens the modal showing the rendered graphic (no XML, no `.doc-raw` in the modal body)
- [ ] `/document.html?source=…&path=…svg` shows the graphic spanning the `.doc-image` card (width ≥ 600px at a 1280px viewport for a `viewBox`-only SVG) with the extracted text below it
- [ ] the chat citation figure and the Sources thumbnail show the graphic; the figure's caption is the extracted text
- [ ] a raster image document's viewer DOM is unchanged (the phase-122/133 suites pass unedited)
- [ ] `uv run pytest` green; `uv run ruff check .` clean
@@ -0,0 +1,38 @@
# Task 05 — Unit + integration + the dedicated Playwright suite, the gates, the commit
**Phase:** `181_svg_document_preview` · **Source:** `TODO.md:3` (the acceptance evidence for tasks 01–04)
**Story:** n/a
## Objective
Prove the whole SVG path — extraction, image-document ingestion, the sandboxed bytes route, and the preview at every surface — with the phase's own test files, run the AGENTS gates, and land the single atomic commit.
## Work
1. `tests/unit/test_svg_text_extraction.py` — completes task 01's pure-function matrix (document order, `tspan` joining, namespaced tags, whitespace/dedupe rules, entity decoding, `<!DOCTYPE`/`<!ENTITY` refusal, the 2 MiB cap, malformed XML → `None`, the structural fallback with its plural + clause-omission rules, `MAX_*` constants pinned).
2. `tests/unit/test_svg_image_ingest.py` (or the extension of `tests/unit/test_image_documents.py` chosen in task 02) — the vector branch with the `_describe_or_skip` seam patched to RAISE (the "no vision call" proof), `content`/`summary`/chunk shape, `is_image`/`image_path` under `image_dir`, `svg_indexed`, the extraction-failure skip (`images_failed`, no row, no copy), the phase-125 `image_context` prepend for an SVG, and the `_index_file` fast-path kind guard in BOTH directions (text row converts; correctly-classified row stays `unchanged`).
3. `tests/integration/test_svg_document_image.py` — task 03's contract (SVG → `image/svg+xml` + the sandbox CSP + `nosniff` + `Cache-Control` + exact bytes; PNG → NO CSP key; the four 404 shapes) plus `GET /api/documents/content` for an SVG row: `is_image: true`, `image_url` present and of the form `/api/documents/<uuid>/image`, `format: "svg"`, `content` = the extracted text. Add the wire-contract pin that ties the phases together: that `image_url` shape is accepted by phase 180's `allowImageUrl` (assert the literal path passes the renderer's allowlist regex — the `![alt](<image_url>)` synergy).
4. `tests/fixtures/svg_docs/` (new dedicated tree — never the shared fixture set) — `diagrams/topology.svg` (a hand-written multi-kilobyte diagram: a `viewBox` and NO `width`/`height`, a `<title>Homelab network topology</title>`, a `<desc>` sentence, two `<text>` labels, and a dozen `<path>`/`<rect>` shapes so retrieval has real words) + `notes/infra.md` (prose that names the diagram so the grounded question has a text co-citation) + `diagrams/blank.svg` (pure paths — the structural-fallback subject).
5. `tests/e2e/test_svg_document_preview.py` (new; the `tests/e2e/test_image_documents.py` module-app pattern — its own app instance with `BOR_IMAGES=true` and `BOR_IMAGE_DIR` pinned to a suite tmp dir so a local `.env` cannot leak either way, the phase-138 `_reset_db` truncate-then-`import_sources` seeding, `login(...)`, mock LLM, `E2E_REAL_LLM` skipped):
- `test_viewer_renders_svg_not_xml` — open the Sources row for `diagrams/topology.svg` (admin): the modal shows `img.doc-image-img--vector` with `naturalWidth > 0` and a non-empty `alt`; the modal body contains no `<pre class="doc-raw">` and no `<svg` XML text; the extracted `<title>` text appears as the description below the graphic; the "Full page" link's `/document.html` view shows the same graphic at ≥600px width at a 1280px viewport (the vector sizing contract).
- `test_chat_figure_and_modal_preview` — the grounded question whose answer cites the SVG doc shows `.source-image` whose `<img>` loads (`naturalWidth > 0`) with the caption from the extracted text; clicking the figure opens the modal with the full graphic (the "preview in the RAG tab" contract) and Escape closes it.
- `test_blank_svg_is_indexed_and_searchable` — `diagrams/blank.svg` exists as a document with the structural `Vector diagram "blank" — …` content (the LOCKED A2 fallback), listed on the Sources page with its thumbnail.
- `test_markdown_image_ref_renders` (two-stage seeding — copy the fixture tree to the suite's tmp sources dir, import it, read the SVG doc's id from the DB, then write `notes/inline.md` containing `![topology](/api/documents/<id>/image)` and re-import): the viewer for that document shows `img.md-img` pointing at the bytes route and loading (`naturalWidth > 0`) — the phase-180 renderer + this route end to end.
- `test_raster_and_pdf_surfaces_unchanged` — the smoke guard: a PNG doc in the same tree still renders through `.doc-image-img` (no `--vector` class) and the CSP header is absent on its bytes route.
6. Run the gates in order, fixing what fails (no baselining, no skipping):
- `uv run pytest`
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL >90%
- `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` (in isolation, DB up)
- `uv run ruff check . && uv run pyright`
- `uv run python -m scripts.api_payload_audit` — the phase-167 byte-budget line for the new SVG route is INFO-sized (an SVG fixture stays well under 256 KB).
7. Move the phase directory to `.agents/phases/complete/` and commit — see the overview's Commit block.
## Testing & Quality
- This task IS the phase's gate. If a test cannot be made deterministic, fix the product, not the assertion (the AGENTS rule 9 posture) — and record anything genuinely untestable in the commit body.
- Coverage: **>90%** on `app/`; `app/rag/svg_text.py` should sit at ≥95% (pure functions, no excuses).
## Completion Criteria
- [ ] `uv run pytest` green (new unit + integration files included)
- [ ] `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%
- [ ] `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` green in isolation
- [ ] `uv run ruff check . && uv run pyright` clean
- [ ] `uv run python -m scripts.api_payload_audit` clean for the SVG route
- [ ] one atomic commit, `--no-gpg-sign`: `feat(kb): index SVGs as image documents and preview them instead of raw XML`, containing `app/`, `frontend/`, `tests/`, `.env.example` and `.agents/`
@@ -0,0 +1,79 @@
# Phase 182 — Verbatim file display: a `show_text` tool + `bor` locator fences the server resolves
**Story:** n/a (owner chat 2026-10-14 — "show me my pi.dev config" / "show me the quadlet for gufo" answers are re-typed from context and silently add/remove characters, so the answer text cannot be copy-pasted; the owner has to open the source bubble and ignore the written code block)
**Context:** `app/rag/agent.py` — the harness-aligned tool surface `AGENT_TOOLS` (L332: `ls` / `read` / `grep`) plus the 4th, DISPLAY tool `SHOW_IMAGE_TOOL` (L501) composed by `agent_tools(settings)` (L538), the shared identity resolver `_resolve_path` (L1357), the line-range validator `_read_range_args` (L1474) and the `OFFSET_PAST_END` (L671) / `BAD_RANGE_ARGS` (L628) teachings, the refusal block L571–L690 (`MISSING_SHOW_ARGS` L571, `NOT_AN_IMAGE_DOC` L581, `ALREADY_SHOWING` L585), `AgentHolder` (L1600–L1680: `shown_images`, `read_coverage`, `read_truncations`), `_execute_tool` (L1681, the `show_image` branch L2023–L2068 is the shape this phase mirrors); `app/rag/prompts.py` — `TOOLS_SECTION` (L293) and the appended-only-when-offered `SHOW_IMAGE_CLAUSE` (L379) + `build_high_prompt(show_image=…)` (L759); `app/api/chat.py` — the prompt/tool call site `show_image=settings.answer_images` (L529), the per-turn visible-content counter `content_chars` (L934), the `done`-frame builder (L1320–L1365, `answer_ref`/`answer_image` L1343–L1355), the pinned per-turn log line (L1272–L1291); `app/schemas.py` — `SourceRef` (L169), `ChatToolResultEvent` (L279), `ChatDoneEvent.answer_image` + its omission serializer (L305–L380), `ChatMessageRecord.answer_image` + `_drop_image_when_absent` (L1463, L1478); `app/config.py` — `read_max_chars` (L296) + its validator (L686), `answer_images: bool = False` (L539), `log_large_response_bytes` (L76); `app/core/settings_catalog.py` — `read_max_chars` (L267) / `answer_images` (L296) rows (the completeness pin in `tests/unit/test_settings_catalog.py` makes a new `Settings` field a build failure until it has a row); `frontend/assets/markdown.js` — `renderMarkdown(md)` (L43), fence-protection pass (L44–L48: fences → tables → escape+inline → blockquotes → paragraphs → restore, escape-first, no CDN); `frontend/assets/app.js` — the per-delta whole-bubble re-render `renderMarkdown(acc)` (L4322), the `done` handler (L4325–L4460), `appendAnswerImage` (L2598), the restore renderer (L2890–L2960), `copyShareLinkWithFallback` (L3250, the http-origin clipboard degradation), `rememberBrainTurn` save point (L4390–L4420); `frontend/assets/shared.js` — `addAnswerImage` (L573) + the record renderer (L619–L660); `frontend/index.html` L1696–L1701 and `frontend/shared.html` L197–L199 (classic `brand.js` + `markdown.js`, then the module script); `frontend/assets/styles.css` — `.answer-image` (L1028–L1059), the bounded-scroll-panel and `.cell-clip` conventions (PLAN §7.1, phases 156/167); `tests/e2e/mock_llm.py` — the scripted-trigger registry (L1162–L1520, `SHOW_IMAGE_TRIGGER` L2606, `_SHOW_CONFIRM_MARKER`, the "verified: no existing E2E question or fixture file contains the phrase" discipline); `tests/e2e/test_answer_images.py` (the two-app ON/OFF pattern), `tests/unit/test_frontend_*.py` (the JS/CSS text-pin convention).
## Objective
When the owner asks *"show me the quadlet for gufo"*, the exact bytes of the indexed file reach the chat answer — **typed by the server, never by the model** — in a panel with a path + line-range caption, a `VERBATIM` badge, line numbers, a working Copy button and a link to the source. The model only *names* the location (`show_text(path, offset?, limit?)`, or an inline ` ```bor <path> <range> ` locator fence where it wants the block to sit); the server resolves the slice from Postgres and the client places it. This kills the class of bug where the answer's code block silently differs from the file, and it removes the user-facing line-number problem entirely: the numbers shown are the server's resolved extent, not the model's claim.
## Design (shared by all tasks — the executor reads this, not the chat)
- **D1 — the two mechanisms, one authority (A1).** (a) a 5th agent tool `show_text(path, offset?, limit?)`, the display sibling of `show_image`; (b) an **inline locator fence** the model may write in its answer — ` ```bor ` + the combined `source/path` + an optional range — so a resolved block can sit anywhere in the prose. **Both are resolution requests, never content requests**: the only party that supplies bytes is the server. `show_text` is the authority for teaching (a refusal teaches the model to self-correct before the user sees anything); the fence is placement sugar and ALSO resolves standalone (a fence with no prior `show_text` call still renders, because resolution is a plain `(source, path)` lookup, not a replay of a tool call).
- **D2 — server resolves, client places (A4/A7).** The client cannot resolve (no new route, A4), so: `app/api/chat.py` accumulates the turn's **clean visible answer text** (next to the existing `content_chars` counter, L934 — the post-scaffold-filter text only, never `thinking`), and at `done`-build time passes it through `app.rag.verbatim.resolve_answer_blocks(db, answer, holder.shown_texts, settings)`; the result rides the `done` frame as the ADDITIVE `answer_text` array. `markdown.js` turns a `bor` fence into `<div class="verbatim" data-bor-locator="<canonical>">` and **drops the model's fence body**; `app.js`/`shared.js` hydrate each placeholder by matching `data-bor-locator` against the record's `answer_text[].locator` (canonical-form match → unique path-only match → unresolved). A block the tool registered that the answer never locates renders APPENDED, at the `answer_image` position (below the answer, above the source clusters).
- **D3 — the model's bytes are NEVER painted (A7, the phase's core contract).** Inside a `bor` fence the model's body is discarded at render time — while streaming the placeholder reads `Retrieving the exact text…` (`role="status"`, polite by definition), and a fence the model has not finished typing shows that SAME pending state rather than raw backticks, at `done` it becomes the resolved panel or, when it cannot be resolved, the honest state `I could not verify this against the knowledge base — open <source>` (persona: radically honest, never a silent substitution and never the model's possibly-corrupt text). The E2E proves it with a mock that deliberately types a CORRUPTED body and asserts the panel carries the DB bytes. `renderMarkdown` gains a SECOND, OPTIONAL argument (`{ verbatim: true }`); default OFF keeps every existing call site — the document viewer (`document.js` L494), the thinking blocks, the user bubbles — byte-identical, and a document that legitimately contains a ` ```bor ` fence still renders its own text.
- **D4 — line ranges are the server's (A2).** `offset` (1-based) / `limit` are OPTIONAL and validated by the existing `_read_range_args` (L1474): omitted → the WHOLE document (the taught default for "show me X"); `offset` alone → to the end; `limit` overshoot → CLAMPED to the last line; `offset` past the last line → the `OFFSET_PAST_END`-style refusal naming the document's true line count (counts in nothing, consumes one round, invisible to the user). Slicing uses `content.splitlines()` with the SAME 1-based indexing as `read` (L1836) and `grep`, so every line number in the app agrees. The char cap (`settings.show_text_max_chars`) cuts the SLICE at a line boundary, appends the shared `TRUNCATION_MARKER` (`app/rag/retriever.py`) and shrinks `end_line` to the last fully-shown line — the caption never lies.
- **D5 — what the model gets back (A3).** The tool result is a confirmation + the truth + an instruction, then the exact slice:
`Text shown to the user — these lines are displayed verbatim; describe them, do NOT retype them.` / `Document {source}/{path}:` / `date: …` / `{origin lines}` / `lines: {first}-{end} of {total_lines}` / the slice. The `Document …` + `date:` + `lines:` header contract is the one `read` established (the E2E mock's `_READ_RESULT_PREFIX` pin keeps parsing it). Returning the text is deliberate (A21: LLM calls are free) — the model's commentary can then never disagree with the panel.
- **D6 — new module, no import cycle (D2).** `app/rag/verbatim.py` owns `VerbatimBlock` (frozen dataclass: `source, path, title, start_line, end_line, total_lines, text, truncated, locator`), the pure half (`canonical_locator`, `slice_lines`, `fences_in`, `cap_slice`) and one DB function (`document_by_identity(db, combined)`, a bare `select(Document).where(source==, path==)` — the same `(source, path)` identity contract as `agent._resolve_path`, WITHOUT its did-you-mean suggestions). `agent.py` imports the pure half, so `verbatim.py` must never import `agent.py` (documented at the module top — protocol D2). Path traversal is a non-issue by construction: a traversal string is simply a non-existent row (404-shaped refusal, no filesystem access anywhere).
- **D7 — canonical locator (the matching key).** `source/path` for a whole document, `source/path#12-40` for a range. The tolerant input grammar (both the fence parser and the tool's `path` argument) accepts `source/path`, `source/path 12-40`, `source/path:12-40`, `source/path#12-40`, `source/path#L12-L40` and `source/path 12` (single line); anything unparseable is not a `bor` locator at all and renders as an ordinary code block (the body is then the model's own, exactly as today, with a Copy button and no `VERBATIM` badge). Resolution normalizes to the canonical form, so the client's match is a string compare.
- **D8 — caps, kill switch, observability (A8).** `answer_text: bool = True` (`BOR_ANSWER_TEXT`, the master switch — `false` = the tool is not offered, the prompt clause is absent, resolution does not run, no `answer_text` key: every frame and every byte-identity contract is exactly pre-phase), `show_text_max_chars: int = 12_000` (per block; `0` = inert, negative fails startup naming the field — the `agent_max_rounds`/`read_max_chars` pattern), `show_text_max_blocks: int = 5` (per answer, in answer order; `0` = inert, negative fails startup). Worst-case frame growth 5 × 12 000 = 60 KB against the 256 KB budget, which the phase-167 byte-budget line (`api … bytes=…`, `grep ' :: api '`) makes visible. The §9 per-turn line gains ONE additive trailing field `verbatim_blocks=N` (the blocks actually sent, tool-registered + fence-resolved, deduped) — `tests/unit/test_chat_gate.py`'s pinned line is updated in the same task, never silently.
- **D9 — resolution scope (the honesty posture).** Resolution runs on GROUNDED turns only (the agent ran). A `bor` fence hallucinated into a DEFLECTED answer has no registered blocks and the gate said "I haven't done anything like that" — it renders the unresolved state rather than leaking KB text through a turn that just disclaimed it. `done.sources` / `related` semantics are NOT changed (no regression to the citation surface): the panel itself carries the path, the line range, a Copy button and a link to the document viewer (the git origin link stays the viewer's job, phase 145's `origin_for_source` — the panel does not duplicate it) — the citation affordance sits at the point of use.
- **NOT touched:** `show_image` / `SHOW_IMAGE_TOOL` / `answer_image` (image display is out of scope, A6 — verified, not modified), the `ls`/`read`/`grep` contracts, the SSE event-type LIST (A15 stays at seven types — this phase rides `done`, no 8th event), `documents`/`chunks` schema (no migration: `saved_chats.messages` is raw JSONB, the phase-50/160 precedent), `document.js` (the viewer renders documents, not answers), `app/core/response_budget.py`, `scripts/*`, `frontend/assets/router.js`.
## Locked decisions (owner-confirmed 2026-10-14, from the phase-authoring interview)
- **A1 — BOTH mechanisms in this phase:** the `show_text` tool AND the inline ` ```bor ` locator fence. (The owner chose "A + B in one phase" over the tool alone.)
- **A2 — whole-file default + server clamp:** `offset`/`limit` optional; omitted = the whole document; overshoot clamps; past-the-end refuses to the model with the true total; the caption always shows the SERVER's resolved extent. No `expect`-anchor snapping (considered and deferred — the clamp plus the model's own `grep`/`read` line numbers cover it).
- **A3 — the tool returns the verbatim slice** plus the "do not retype" instruction (not a terse confirmation, not the read_coverage-conditional variant).
- **A4 — the bytes ride INLINE** in `done.answer_text` and persist in the `bor.chat.v1` brain record: no new bytes route, no new SSE event type, no new anonymous surface (a shared page renders from the share's own record, the same trust model as the phase-123 chat image).
- **A5 — Copy buttons on EVERY fenced code block in an answer** (live, restored and shared), but the `VERBATIM` badge + caption belong ONLY to a server-resolved block. The clipboard goes through the phase-55 degradation (a non-secure origin gets the inline select-to-copy fallback) — never a bare `navigator.clipboard` assumption.
- **A6 — images are out of scope:** `show_image` keeps images; `show_text` on an `is_image` document is REFUSED with a pointer to `show_image`. The phase's E2E asserts the answer figure is unchanged.
- **A7 — the model's fence body never reaches the DOM** (see D3). Unresolvable = honest, visible, non-copyable-as-truth state.
- **A8 — settings:** `BOR_ANSWER_TEXT` default `true` (master switch), `BOR_SHOW_TEXT_MAX_CHARS` default `12000`, `BOR_SHOW_TEXT_MAX_BLOCKS` default `5`; `0` = inert, negative = loud startup failure.
- **A9 — additive wire only:** `answer_text` is dropped when empty (absent, never `null`) — every frame without blocks, every pre-phase record and the whole default-off configuration stay byte-identical. No PLAN §2/§4 anchor changes beyond recording these revisions.
## Dependencies
- `124_answer_images` (complete) — the display-tool precedent this phase mirrors end to end (`SHOW_IMAGE_TOOL` → `holder.shown_images` → `done.answer_image` → `appendAnswerImage` → persistence → shared page).
- `131_read_line_ranges` / `141_read_range_dedupe_fix` (complete) — `offset`/`limit` + `_read_range_args` + the `lines: N-M of T` header contract and `splitlines()` indexing.
- `145_git_source_links` (complete) — the document viewer's `origin_for_source` link, which the panel links TO rather than duplicating.
- `167_api_payload_diet` (complete) — the byte-budget line that will show the `done`-frame growth.
- `180_markdown_inline_links_images` (todo, already authored) — independent of this phase; both touch `markdown.js` and `styles.css`, so run it FIRST (or rebase) to avoid a renderer conflict.
## Tasks
1. `01_settings_and_kill_switch.md` — the three `Settings` fields + validators + the three `CATALOG` rows + `.env.example` (+ their unit/integration pins). Inert: nothing reads them yet.
2. `02_verbatim_resolver_module.md` — `app/rag/verbatim.py`: the locator grammar, the canonical normalizer, `splitlines()` slicing, the line-boundary cap and `resolve_answer_blocks` (+ the resolver unit suite).
3. `03_show_text_tool.md` — `SHOW_TEXT_TOOL`, the `agent_tools` gate, the `_execute_tool` branch, the refusal set, `AgentHolder.shown_texts` (+ agent unit suite).
4. `04_prompt_teaching_clause.md` — `SHOW_TEXT_CLAUSE` inside `<tools>`, `build_high_prompt(show_text=…)`, the `app/api/chat.py` call site (+ byte-identity pins when off).
5. `05_done_frame_and_persistence.md` — the `VerbatimBlockRef` schema, `ChatDoneEvent.answer_text`, `ChatMessageRecord.answer_text`, the visible-text accumulator + resolution + `verbatim_blocks=N` in the per-turn line (+ integration/frame tests).
6. `06_markdown_locator_and_panel_css.md` — `renderMarkdown(md, opts)`'s opt-in `verbatim` pass → `.verbatim[data-bor-locator]` (body dropped; ordinary fences keep rendering exactly as today), and the `styles.css` panel/copy-button CSS contract. Inert until task 07 opts in and wraps `<pre>` nodes at the DOM level.
7. `07_hydration_and_copy_buttons.md` — `frontend/assets/verbatim.js` (+ `index.html`/`shared.html` script tags): hydrate-at-`done`, restore, shared page, error/settled paths, `rememberBrainTurn` persistence, copy buttons on every answer code block.
8. `08_e2e_and_docs_pass.md` — the mock triggers (`show_text` call, corrupt-body fence, fence-only, off-app), `tests/e2e/test_verbatim_show_text.py` in isolation, the regression set, the full gates, the README/AGENTS full consistency pass, the atomic commit.
## Testing & Quality
- **Unit:** `tests/unit/test_config.py` (the three defaults; env overrides; `0` accepted as inert; each negative value fails validation naming the field); `tests/unit/test_settings_catalog.py` (the completeness pin green with the three new rows); `tests/unit/test_verbatim_resolver.py` (NEW — the locator grammar incl. every tolerant form + rejects, canonical normalization, whole-file/offset/limit/clamp/past-end, the line-boundary cap + `TRUNCATION_MARKER` + shrunk `end_line`, `fences_in` ordering + dedupe, tool-then-fence dedupe, `max_blocks` cut, an `is_image` doc refused, an unknown path unresolved, a fenced `bor` line inside an ordinary fence, an unclosed fence, an empty answer); `tests/unit/test_agent.py` (the `show_text` branch: success shape + headers, `shown_texts`, refusal set — missing args, non-string, unknown path did-you-mean, image doc, bad range args, offset past end, block cap, duplicate identical show; every refusal counts in nothing and executes nothing; `agent_tools` offers 4 vs 5 tools by flag); `tests/unit/test_prompts.py` (the clause is INSIDE `<tools>` before `</tools>`, the off-build is byte-identical to pre-phase, `TOOLS_SECTION` itself unchanged); `tests/unit/test_chat_gate.py` (the per-turn line's new `verbatim_blocks=N`, and `0` on a turn with none); `tests/unit/test_frontend_verbatim_markdown.py` + `tests/unit/test_frontend_verbatim_blocks.py` + a CSS pin module (the `test_frontend_*` text-pin convention: the `verbatim` option, the dropped body, `data-bor-locator`, the `.verbatim`/`.code-block`/`.verbatim-copy` markers, the new `<script>` tags in `index.html` + `shared.html` (plain relative `src` — no `cache-busting` token in the path, the `?v=` rewrite is the middleware's job, phase 160 A7), `shared.js` parity, no inline `onclick` — CSP `default-src 'self'`).
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` (all three rows: `bool` + two `int`s, correct groups/labels, live apply without a restart, reset-to-env); the chat API suite — a grounded turn with a `show_text` call and/or a fence yields `done.answer_text` with the SERVER's text and extent; a turn with none has NO `answer_text` key (byte-identity); `answer_text=false` yields the pre-phase frame byte-identically; the saved-chat round trip preserves `answer_text` (`extra="forbid"` accepts it — the phase-113 lesson) and `GET /api/chats/{id}` returns it.
- **E2E (AGENTS rule 4, task 08):** `tests/e2e/test_verbatim_show_text.py`, run **in isolation** — the panel's text equals the fixture file's bytes (the mock's corrupted fence body never appears anywhere in the DOM), the caption reads `source/path · lines 12–40 of N` from the SERVER's extent, `VERBATIM` badge + line numbers + Copy button copy EXACTLY the panel's bytes (clipboard read with granted permissions, or the fallback field), the panel's open link targets `/document?source=…&path=…`, an unresolvable locator renders the honest state with no model bytes, restored-after-reload and the anonymous shared page render the same panel, every ordinary code block has a Copy button and no badge, and the `show_image` figure still renders (A6 regression).
- **Coverage:** **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing`).
- **Lint/types:** `uv run ruff check . && uv run pyright`.
## Documentation & Consistency
- The final pass reviews `README.md` and `AGENTS.md` in full against the project as it stands after this phase — fixing every inconsistency found, **including pre-existing ones not caused by this phase**.
- Tasks update `README.md`/`AGENTS.md` immediately when their own changes are user-visible (setup, commands, configuration, features, conventions). This phase's user-visible additions: the `show_text` capability + the `bor` locator syntax in the feature list, the three `BOR_*` settings in the env table, and (in `AGENTS.md` rule 5's UI list) the verbatim-panel contract "an answer's code block is either verbatim-and-badged or plainly the model's — never a silently-substituted hybrid".
## Completion Criteria
- [ ] `ask "show me the quadlet for <x>"` on a running app (real or mock LLM) renders a panel whose text is byte-identical to `documents.content`'s resolved slice, with a caption whose numbers came from the server, and a Copy button whose clipboard content equals that slice exactly.
- [ ] `uv run python -c "from app.config import get_settings as g; s=g(); print(s.answer_text, s.show_text_max_chars, s.show_text_max_blocks)"` prints `True 12000 5`; each negative value refuses startup naming its field.
- [ ] `GET /api/admin/settings` lists `answer_text`, `show_text_max_chars`, `show_text_max_blocks`; `PUT`/`DELETE` round-trip per the overlay contract.
- [ ] A grounded turn's `done` frame carries `answer_text[]` with `source,path,title,start_line,end_line,total_lines,text,truncated,locator`; a turn without verbatim content carries NO key; with `BOR_ANSWER_TEXT=false` the frame is byte-identical to pre-phase and only FOUR tools are offered.
- [ ] The per-turn log line ends with `verbatim_blocks=N` (`grep` it on a shown turn and on a plain turn).
- [ ] `uv run pytest` green, `app/` coverage >90%, ruff + pyright clean.
- [ ] `uv run pytest tests/e2e/test_verbatim_show_text.py -v --no-cov` green **in isolation**; `test_answer_images.py`, `test_harness_aligned_tools.py`, `test_agent_document_tools.py`, `test_markdown_tables.py` (and the phase-180 suite if it landed first) green in isolation.
- [ ] README.md and AGENTS.md are accurate and consistent with the whole project (verified in the final pass).
- [ ] no behavior change in completed phases (image answers, `read`/`grep`/`ls`, deflected byte-identity, the document viewer, saved-chat byte-identity).
- [ ] One atomic `--no-gpg-sign` Conventional Commit (e.g. `feat(agent): display file contents verbatim with a show_text tool and server-resolved locator fences`); the phase dir moves to `.agents/phases/complete/`.
@@ -0,0 +1,33 @@
# Task 01 — Three settings and the kill switch (inert)
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
Add the three `Settings` fields the phase runs on — the master `answer_text` switch and the two budgets — with their admin-panel rows and `.env.example` entries. Nothing reads them yet: after this task the app behaves byte-identically to pre-phase.
## Work
1. `app/config.py` — in the agent-tools neighborhood of `read_max_chars` (L296), add:
- `answer_text: bool = True` — the master switch for verbatim display (`BOR_ANSWER_TEXT`). Docstring: names the env var, the default `True`, and what `False` means (the `show_text` tool is not offered, the `<tools>` clause is absent, locator resolution does not run, the `done` frame carries no `answer_text` key — every byte-identity contract is exactly pre-phase). Model the docstring on `answer_images` (L539) but note the DEFAULT IS TRUE here (the owner chose the fix on by default; the switch is an escape hatch, not an opt-in).
- `show_text_max_chars: int = 12_000` (`BOR_SHOW_TEXT_MAX_CHARS`) — the per-block character budget for one displayed slice; `0` = inert (no block displays), a negative value is a typo and fails startup naming the field (the `_read_max_chars_non_negative` validator at L686 is the exact pattern).
- `show_text_max_blocks: int = 5` (`BOR_SHOW_TEXT_MAX_BLOCKS`) — the per-answer cap on displayed blocks; `0` = inert, negative fails startup (the `agent_max_rounds` validator at L678 is the pattern).
Add the two `@field_validator`s next to `_read_max_chars_non_negative`, each raising a `ValueError` that names the field (house rule: fail loud, never half-configured).
2. `app/core/settings_catalog.py` — three `SettingMeta` rows beside `read_max_chars` (L267): `answer_text` → `"Answer text"`, `show_text_max_chars` → `"Show text max chars"`, `show_text_max_blocks` → `"Show text max blocks"`, all in the `Agent tools` group (the `read_max_chars` group). The completeness pin in `tests/unit/test_settings_catalog.py` (`set(CATALOG) == set(Settings.model_fields)`) is the enforcement — without the rows the build fails.
3. `.env.example` — three lines in the agent-tools block, each with a one-line comment: `BOR_ANSWER_TEXT=true`, `BOR_SHOW_TEXT_MAX_CHARS=12000`, `BOR_SHOW_TEXT_MAX_BLOCKS=5`. State the `0`/negative semantics in the comments (the surrounding file's style).
4. No consumer changes. Do NOT touch `app/rag/agent.py`, `app/rag/prompts.py`, `app/api/chat.py`, `app/schemas.py` or any frontend file in this task.
## Testing & Quality
- **Unit:** `tests/unit/test_config.py` — defaults (`True`, `12000`, `5`); each env override applies; `BOR_SHOW_TEXT_MAX_CHARS=0` and `BOR_SHOW_TEXT_MAX_BLOCKS=0` are accepted (inert, not an error); each negative value raises a validation error whose message names the field (the `read_max_chars` test is the template). `tests/unit/test_settings_catalog.py` — the completeness pin green, the three rows carry the right type/group/label.
- **Integration:** `tests/integration/test_admin_settings_api.py` / `test_admin_settings_live.py` — the three rows appear in `GET /api/admin/settings` (`bool` + two `int`, group `Agent tools`); `PUT` stores each raw value and it resolves live without a restart; `DELETE` resets each to the env default; the `0` values are storable.
- **Coverage:** **>90%** on this task's new/modified code (the validators are the only new logic — cover both branches of each).
## Completion Criteria
- [ ] `uv run python -c "from app.config import get_settings as g; s=g(); print(s.answer_text, s.show_text_max_chars, s.show_text_max_blocks)"` → `True 12000 5`.
- [ ] `BOR_SHOW_TEXT_MAX_CHARS=-1 uv run python -c "import app.config"` refuses with a message naming `show_text_max_chars` (same for `show_text_max_blocks`).
- [ ] The three rows are visible in the admin Settings panel's `Agent tools` group (or asserted by the integration test if the browser is not running).
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the env table) — the phase's final pass re-checks both project-wide.
- [ ] no behavior change in completed work (no consumer reads the new fields yet).
@@ -0,0 +1,47 @@
# Task 02 — `app/rag/verbatim.py`: the locator grammar, the slicer, the resolver
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
The single source of truth for "which exact lines of which indexed document, and what are their bytes". Pure functions for the locator grammar and the slicing, one bare DB lookup, and `resolve_answer_blocks` which the API layer calls at `done` time. Inert: nothing calls it yet (tasks 03 and 05 wire it in).
## Work
1. Create `app/rag/verbatim.py`. Module docstring states the contract (PLAN §6's tool surface extension, phase 182 D1–D7): *the model names a location, this module supplies the bytes*; and the import rule — **this module must never import `app.rag.agent`** (agent.py imports the pure half of this module; a reverse import cycles). Allowed imports: `sqlalchemy`, `app.models.Document`, `app.rag.retriever.TRUNCATION_MARKER`, `app.config.Settings`, stdlib.
2. `LOCATOR_LANG = "bor"` and `@dataclass(frozen=True) class VerbatimBlock` — `source: str`, `path: str`, `title: str`, `start_line: int`, `end_line: int`, `total_lines: int`, `text: str`, `truncated: bool`, `locator: str` (the canonical form, built by the factory, never passed in by a caller). The dataclass docstring pins: `start_line`/`end_line` are 1-based and INCLUSIVE and describe exactly the lines `text` contains after any cap; `text` never carries a trailing newline (lines are joined with `"\n"`).
3. `canonical_locator(raw: str) -> str | None` — the tolerant parser of D7. Accepts, in order: a combined `source/path` (the first `/` splits it, the `read`/`grep` identity contract) followed by an optional separator + range — separators ` `, `:`, `#`, `#L`; ranges `12-40`, `12–40` (en dash), `L12-L40`, or a bare `12` (one line, end == start). Returns `source/path` for a whole document and `source/path#12-40` for a range (always the canonical `#` + plain digits, never the en dash, never the `L`). Returns `None` (NOT a locator) for: an empty/whitespace input, a value with no `/` in the path part, a non-numeric or non-positive bound, `start > end`, trailing junk after the range, or a leading `bor` token left in by the caller. No DB, no normalization of case (paths are case-sensitive row values).
4. `fences_in(answer: str) -> list[str]` — every `bor` fence's RAW locator string, in document order, deduped keeping the first occurrence. Rules: the opening line is a fence whose info string is exactly `` ```bor `` + whitespace + a non-empty remainder (a bare `` ```bor `` with no path is NOT a locator — it is an ordinary code block and must not be swallowed); the fence closes at the next line that is `` ``` `` (trimmed); an UNCLOSED fence contributes nothing (defensive — a half-streamed answer); the body is read and DISCARDED (this module never looks at what the model typed); nested `` ``` `` inside the body is not supported and is a documented limitation (the fence ends at the first closer, the same rule `markdown.js` applies).
5. `@dataclass(frozen=True) class SliceResult` — `text: str`, `first_line: int`, `last_line: int`, `total_lines: int`, `past_end: bool` — and `slice_lines(content: str, offset: int | None = None, limit: int | None = None) -> SliceResult`. Indexing is `content.splitlines()` with the SAME 1-based semantics as `read` (`app/rag/agent.py` L1836) and `grep`: `offset` omitted → 1; `limit` omitted → to the end; `end = min(offset + limit - 1, total_lines)` (an overshooting `limit` CLAMPS); `offset > total_lines` → `past_end=True` with empty text and the true `total_lines` (the caller builds the teaching refusal); `total_lines == 0` (an empty document) → `past_end=True`. Never raises for a range problem — the refusal is the caller's, the total always travels with the result.
6. `cap_slice(text: str, first_line: int, cap: int) -> tuple[str, int, bool]` — the character budget applied at a LINE boundary, returning `(text_to_show, last_line_shown, truncated)`. `cap <= 0` raises `ValueError` (a mis-wired caller must be loud — the `show_text_max_chars == 0` kill switch is checked by `resolve_answer_blocks`, never by reaching here). When `len(text) <= cap`: `(text, last_line_of(text), False)`. Otherwise cut to the last whole line that fits within `cap` and return `(cut + "\n" + TRUNCATION_MARKER, first_line + n_lines_in_cut - 1, True)`; a first line LONGER than `cap` cuts that one line mid-way (the only sub-line cut, documented) and still ends with the marker. The marker is the shared `app.rag.retriever.TRUNCATION_MARKER` — never a second copy of the text.
7. `document_by_identity(db: Session, combined: str) -> Document | None` — split at the FIRST `/` (source names are directory basenames and never contain one); `select(Document).where(Document.source == source, Document.path == path)` → the row or `None`. Docstring: this is the same `(source, path)` identity contract as `app.rag.agent._resolve_path` REIMPLEMENTED as a bare lookup — no did-you-mean suggestions here (the tool branch already has `_resolve_path` for teaching; the fence resolver only needs yes/no), and the duplication exists to keep the import direction one-way. A traversal string is simply a row that does not exist.
8. `block_for(db, combined: str, *, offset: int | None = None, limit: int | None = None, cap: int, doc: Document | None = None) -> VerbatimBlock | None` — resolve (or take the caller's already-resolved `doc`), refuse `None` for: no document, `doc.is_image` (image display belongs to `show_image`, locked A6), an empty/past-end slice; otherwise `slice_lines` → `cap_slice` → the block with `locator = f"{source}/{path}"` (+ `f"#{first}-{last}"` when the request named a range or the cap cut it — the canonical form task 07 matches on). `title` is the row's title.
9. `resolve_answer_blocks(db, answer: str, shown: Sequence[VerbatimBlock], settings: Settings) -> list[VerbatimBlock]`:
- returns `[]` when `not settings.answer_text`, or `settings.show_text_max_blocks == 0`, or `settings.show_text_max_chars == 0` (the kill switches, checked HERE so no caller has to remember);
- for each `fences_in(answer)` raw string in order: `canonical_locator` → skip when `None`; `block_for(...)` → skip when `None` (unknown path, image doc, past end) — the caller sees simply "not among the blocks", which is what task 07 renders as the unresolved state;
- dedupe by canonical locator, the FENCE winning over a same-locator entry from *`shown`* (the fence controls placement);
- append the `shown` blocks the answer never located, in their original call order;
- truncate the list to `settings.show_text_max_blocks`.
Docstring pins the ordering rule (answer order first, then the unlocated tool blocks) — task 07's DOM order and the E2E assertions depend on it.
10. Update the module-level `__all__` and keep every public name docstring'd in the house style (contract + phase reference).
## Testing & Quality
- **Unit:** create `tests/unit/test_verbatim_resolver.py`.
- `canonical_locator`: `homelab/a.md` → itself; `homelab/a.md 12-40` / `:12-40` / `#12-40` / `#L12-L40` / `12–40` (en dash) → `homelab/a.md#12-40`; `homelab/a.md 12` → `#12-12`; `None` for `a.md` (no source), `""`, `" "`, `homelab/a.md 0-5`, `homelab/a.md 40-12`, `homelab/a.md x-y`, `homelab/a.md 12-40 extra`, `"bor homelab/a.md"`.
- `slice_lines`: whole file, offset-only, limit-only, both, overshooting limit clamps `last_line` to the total, `offset` past the end → `past_end` + true total, empty content → `past_end`, and the `splitlines()` agreement check (a fixture whose line `N` is asserted by content, mirroring what `grep` would report).
- `cap_slice`: under cap (untouched, `truncated=False`), cut at a line boundary (text ends with the shared `TRUNCATION_MARKER`, `last` = the last whole line shown, and the marker line is not counted as content), a single over-long first line, `cap=0` raises `ValueError`.
- `document_by_identity`: hit, miss, traversal-shaped miss, first-slash splitting when the path itself contains slashes.
- `block_for`: whole-document block, ranged block, canonical locator with/without a range, image doc → `None`, unknown path → `None`, past-end → `None`, capped block's `end_line` equals the lines actually shown.
- `resolve_answer_blocks`: no fences + no shown → `[]`; fence-only (no tool call) resolves; tool-only appends; fence + identical shown block → ONE block (fence wins, appears at the fence position); two fences keep answer order; `show_text_max_blocks=1` truncates; every kill-switch combination returns `[]`; an unresolvable fence is simply absent; a duplicated fence appears once.
- Session-fixture style follows the existing agent/resolver unit suites (the same in-memory/session fixture the `tests/unit/test_agent.py` helpers use).
- **Coverage:** **>90%** on this task's new/modified code (every branch above is a test).
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_verbatim_resolver.py -v` green.
- [ ] `uv run python -c "import app.rag.verbatim as v; print(v.canonical_locator('homelab/a.md#L12-L40'))"` prints `homelab/a.md#12-40`.
- [ ] `grep -n "import" app/rag/verbatim.py` shows NO import of `app.rag.agent` (the one-way rule).
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (nothing user-visible yet — the phase's final pass re-checks both project-wide).
- [ ] no behavior change in completed work (nothing calls this module yet).
@@ -0,0 +1,57 @@
# Task 03 — The `show_text` agent tool: name a location, the server shows the bytes
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
The 5th agent tool: `show_text(path, offset?, limit?)` displays an indexed document (or a line range of one) to the user verbatim. The handler resolves the slice through task 02, records it for the `done` frame, and hands the SAME bytes back to the model with an explicit "do not retype" instruction.
## Work
1. `app/rag/agent.py` — add `SHOW_TEXT_TOOL` right after `SHOW_IMAGE_TOOL` (L501), same OpenAI function shape:
- `name: "show_text"`.
- `description` (this IS the behavior contract — the phase-124 discipline): "Display part of a document to the user EXACTLY as stored — the user sees the server's bytes and can copy them. Call it when the user wants to see, copy or run a file (a quadlet, a config, a script): they copy from what is displayed, so do NOT retype the content yourself after calling it — describe it instead. `path` is the combined `source/path` string exactly as shown in `ls`. `offset` (1-based first line) and `limit` (line count) are optional — omit them to display the whole document; use them when `grep`/`read` told you which lines matter. The displayed line numbers come from the server, not from your arguments: an overshooting `limit` is clamped and an `offset` past the end is refused with the true line count. Only text documents — for an image use `show_image`. Call one tool at a time — wait for this result before your next call."
- `parameters`: `path` (string, required, the `read` "a bare document path will not resolve" teaching), `offset` and `limit` (integers, optional, the `read` wording reused).
2. `agent_tools(settings)` (L538) — append `SHOW_TEXT_TOOL` when `settings.answer_text`, keeping `AGENT_TOOLS` byte-identical and the list fresh per call (the `show_image` precedent; the offer order is `ls`, `read`, `grep`, `show_image`, `show_text`). Update the docstring (four base + two display tools, each gated by its own flag).
3. `AgentHolder` (L1600–L1680) — add `shown_texts: list[VerbatimBlock] = field(default_factory=list)` with a docstring paragraph in the `shown_images` style: the blocks in CALL order, capped by `settings.show_text_max_blocks`, read by `app.api.chat` after the stream for the `done` frame's `answer_text` (the `shown_images`/`read_docs` read-after-stream precedent); refused shows append nothing.
4. Refusal constants beside the `show_image` ones (L571–L590), each single-braced and pinned in the house style:
- `MISSING_SHOW_TEXT_ARGS = "show_text requires a string argument 'path'."`
- `SHOW_TEXT_NOT_A_TEXT_DOC` — one `{arg}` field: the doc is a standalone image; `show_image` displays images. (LOCKED A6 — never silently converted.)
- `SHOW_TEXT_BLOCK_CAP` — one `{max}` field: that many blocks are already displayed this answer — describe them instead of showing more.
- `SHOW_TEXT_ALREADY_SHOWN` — one `{locator}` field: that exact range is already on screen — say so instead of showing it again.
Reuse `BAD_RANGE_ARGS` (L628) and `OFFSET_PAST_END` (L671) verbatim for the range problems (the same teaching, the same echoed values — one copy of each string in the codebase).
5. `_execute_tool` (L1681) — a `show_text` branch placed next to the `show_image` branch (L2023), in this order:
1. `raw_path = call.arguments.get("path")`; non-string/blank → `MISSING_SHOW_ARGS`-shaped refusal (`MISSING_SHOW_TEXT_ARGS`).
2. `settings.answer_text` false (defensive — the tool was not offered) → the same refusal copy as the round-cap-off posture: state that the tool is unavailable; counts in nothing.
3. `doc, _source, _path = _resolve_path(db, arg)`; `doc is None` → `_no_document_refusal(db, arg)` (the phase-72 did-you-mean teaching, reused unchanged).
4. `doc.is_image` → `SHOW_TEXT_NOT_A_TEXT_DOC`.
5. `ok, err = _read_range_args(offset, limit)` — reuse the existing validator unchanged; `not ok` → `err`.
6. `len(holder.shown_texts) >= settings.show_text_max_blocks` → `SHOW_TEXT_BLOCK_CAP` (checked BEFORE the work; `0` therefore disables the tool's effect while keeping it offered — pinned in the tests).
7. `block_for(db, arg, offset=offset, limit=limit, cap=settings.show_text_max_chars, doc=doc)` → `None` (empty document / `offset` past the end) → `OFFSET_PAST_END.format(total_lines=...)` (compute the total with `len(doc.content.splitlines())`, the same expression `read` uses); a returned block whose canonical `locator` already appears in `holder.shown_texts` → `SHOW_TEXT_ALREADY_SHOWN`.
8. Success: `holder.shown_texts.append(block)`; `holder.tool_calls += 1` (executed-only counting, the house rule); return the result string in this EXACT shape (the `read`/`show_image` header contract the E2E mock's `_READ_RESULT_PREFIX` parses):
```
Text shown to the user — these lines are displayed verbatim; describe them, do NOT retype them.
Document {source}/{path}:
date: {YYYY-MM-DD}
{origin lines — the same _origin_lines(origin) helper the read branch uses}
lines: {first}-{last} of {total_lines}
{block.text}
```
`block.text` already ends with `TRUNCATION_MARKER` when the char cap cut it; when `block.truncated` is true append one line, in the `READ_TRUNCATION_NOTICE` spirit, naming that the rest is NOT displayed and that the user can open the document (no `offset`/`limit` re-show loop is implied — the full text is a click away, and only the displayed slice is what the user copies).
- Every refusal path appends nothing, increments nothing, and consumes a round exactly like the existing refusals.
6. Module docstring — extend the numbered loop contract (the `show_image` paragraph's neighborhood) with the `show_text` contract and the locked rules A1–A3/A6: the model names a location, the server supplies the bytes, the displayed extent is the server's, images belong to `show_image`, the fence form is resolved at the API layer (task 05), not here.
## Testing & Quality
- **Unit (`tests/unit/test_agent.py`, the `show_image` family is the template):** `agent_tools` offers 4 tools with `answer_text=False` (byte-identical to pre-phase, `show_image` last) and 5 with it on (`show_text` present, correct order); a successful whole-document show (result headers in order, `lines: 1-N of N`, the slice verbatim, `shown_texts` length 1, `tool_calls` 1, `read_docs` unchanged); a ranged show (clamped `limit` reflected in the `lines:` header); a capped show (`TRUNCATION_MARKER` present + the extra notice line); every refusal — missing `path`, non-string `path`, unknown path (did-you-mean copy, no `shown_texts`), image doc, bad range args, `offset` past the end (the true total named), block cap (`max_blocks=1`, second call refused), duplicate identical show; each refusal asserts `tool_calls` and `shown_texts` untouched; `max_blocks=0` → the first call is refused with the cap copy.
- **Integration:** none required in this task (the frame wiring is task 05) — but `tests/integration/` must stay green, and any test that pins the exact tool-offer array is updated here, deliberately, with a comment.
- **Coverage:** **>90%** on this task's new/modified code.
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_agent.py -v -k show_text` green, including all nine refusal cases and the counting assertions.
- [ ] With `answer_text=True`, a scripted agent turn returns a result whose 2nd line starts `Document ` and whose `lines:` header matches the slice actually returned.
- [ ] `grep -n "SHOW_TEXT_" app/rag/agent.py` shows the four new constants, each used by exactly one branch.
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the tool surface list in README's agent-tools section) — the phase's final pass re-checks both project-wide.
- [ ] no behavior change in completed work (`ls`/`read`/`grep`/`show_image` results byte-identical; the pre-phase four-tool offer byte-identical when the flag is off).
@@ -0,0 +1,36 @@
# Task 04 — Teach the model to point instead of type: the `<tools>` clause
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
The HIGH prompt teaches two things it has never said: **the user copies from what is displayed, not from your prose**, and the exact ` ```bor ` locator-fence syntax for placing a displayed block. Gated by `settings.answer_text`, byte-identical when off.
## Work
1. `app/rag/prompts.py` — add `SHOW_TEXT_CLAUSE: str` right after `SHOW_IMAGE_CLAUSE` (L379), with a docstring comment in the same style (why it exists, what each sentence pins, and that `TOOLS_SECTION` stays byte-identical). The clause text must contain, in this order:
- the display contract: "`show_text` displays part of a document to the user EXACTLY as stored — the user copies from what is displayed, not from your prose.";
- when to call it: "Call it whenever the user wants to SEE, COPY or RUN a file — a quadlet, a config, a script, a unit file — instead of writing the content into your answer yourself.";
- the do-not-retype rule (the phase's reason for existing): "After calling it, do NOT repeat the displayed lines in your answer — describe them, explain them, point at them. Anything you type yourself may differ from the file by a character, and the user cannot tell.";
- the placement syntax: "To put a displayed block exactly where you want it in your answer, write a locator line on its own: three backticks followed by `bor`, a space, the combined `source/path`, and optionally the line range (` 12-40`). Leave its body EMPTY — the app replaces it with the exact stored text. Never type the file's content inside a `bor` block.";
- the honesty-about-numbers rule: "Never guess line numbers — use the numbers `grep` and `read` reported. When you are not sure which lines, OMIT the range and display the whole document: the numbers you display are the server's, never yours.";
- the failure rule: "If `show_text` refuses, say what you could not display — do not reconstruct it from memory.";
- the image pointer: "For an image use `show_image` instead."
2. `build_high_prompt(...)` (L759) — add `show_text: bool = False` after `show_image`; when true, the clause rides INSIDE `<tools>` AFTER the `SHOW_IMAGE_CLAUSE` (when that is present) and before the closing `</tools>` tag, using the existing "section text minus the final `</tools>` + clause + closing tag" mechanism (L782–L791 documents it). Both clauses compose independently: `show_image` only → exactly the phase-124 output; `show_text` only → the three-tool copy + the text clause; both → image clause then text clause; neither → byte-identical to pre-phase. Update the docstring (a *show_text* paragraph mirroring the *show_image* one, incl. the note that the deflection prompt NEVER carries it — a deflected turn has no tools and D9 forbids resolving locators on it).
3. `app/api/chat.py` (the grounded `TurnPlan` builder, L516–L529) — pass `show_text=settings.answer_text` next to `show_image=settings.answer_images`, with the same style of phase comment.
4. Do NOT change `TOOLS_SECTION`, `build_deflect_prompt`, or any mock in `tests/e2e/mock_llm.py` (the new scripted triggers belong to task 08).
## Testing & Quality
- **Unit (`tests/unit/test_prompts.py` + the `tests/unit/test_prompt_lock.py` pins):** the off-build (`show_text=False`, the default) is byte-identical to the pre-phase string for every combination of the other arguments; the on-build contains the clause INSIDE `<tools>` (index of the clause text > index of `<tools>`, `< ` index of `</tools>`); `show_image=True, show_text=True` yields the image clause before the text clause; `TOOLS_SECTION` itself unchanged (the existing lock test still passes untouched); the clause names `show_text`, `bor`, the do-not-retype rule and the "omit the range" rule (the E2E and the persona depend on those exact ideas); the deflect prompt never contains `show_text` or `bor`.
- **Integration:** the grounded-turn integration tests stay green (the prompt string is not pinned byte-for-byte there beyond the `<tools>` marker); add one assertion that a grounded plan built with `answer_text=True` carries the clause and one that `answer_text=False` does not.
- **Coverage:** **>90%** on this task's new/modified code.
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_prompts.py tests/unit/test_prompt_lock.py -v` green.
- [ ] `uv run python -c "from app.rag.prompts import build_high_prompt as b; print('show_text' in b([], show_text=True))"` prints `True`, and the same call without the flag prints `False`.
- [ ] A grounded request against a running app shows the clause in the model request (log/assert via the E2E mock's request capture in task 08 — here, the unit proof is enough).
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the prompt/tool teaching summary) — the phase's final pass re-checks both project-wide.
- [ ] no behavior change in completed phases (deflected prompt byte-identical; the phase-124 clause position untouched).
@@ -0,0 +1,34 @@
# Task 05 — The `done` frame's `answer_text`, the visible-text accumulator, the log field
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
Put the resolved bytes on the wire and in the persisted record: the turn's clean visible answer text is accumulated during streaming, resolved through `app.rag.verbatim` at `done`-build time, and sent as the ADDITIVE `answer_text` array; the per-turn log line gains `verbatim_blocks=N`.
## Work
1. `app/schemas.py` — `class VerbatimBlockRef(BaseModel)` beside `SourceRef` (L169): `source: str = Field(max_length=120)`, `path: str = Field(max_length=1000)`, `title: str = Field(max_length=500)` (the `SourceRef` caps, the same identity), `start_line: int = Field(ge=0)`, `end_line: int = Field(ge=0)`, `total_lines: int = Field(ge=0)`, `text: str = Field(max_length=600_000)` (the phase-83 answer-text cap — generous above the 12 000 default, so an operator raising `BOR_SHOW_TEXT_MAX_CHARS` does not hit the wire limit), `truncated: bool = False`, `locator: str = Field(max_length=2100)`. Docstring: the phase-182 display block; `text` is SERVER-supplied bytes from `documents.content`, never model output; 1-based inclusive lines.
2. `ChatDoneEvent` (L305) — `answer_text: list[VerbatimBlockRef] | None = None` + docstring in the `answer_image` style, and extend its `model_serializer` to POP the key when `None` **or empty** (absent, never `null`, never `[]`: every frame without blocks stays byte-identical to pre-phase — the pin that proves it lives in the integration suite). Note the two deliberate differences from `answer_image` in the docstring: it is a LIST (several blocks per answer, D1), and it carries the TEXT itself (A4 — no bytes route, no new anonymous surface, a shared page renders from its own record).
3. `ChatMessageRecord` (L1400–L1510) — `answer_text: list[VerbatimBlockRef] | None = None` on the BRAIN record, popped when `None`/empty in the existing `_drop_image_when_absent` serializer (extend its docstring). Required so `extra="forbid"` does not 422 the done-time auto-save of a turn with blocks (the phase-113 lesson, applied up front) and so the restore path + the shared page re-render the panels. No migration (`saved_chats.messages` is raw JSONB, the phase-50/160 precedent).
4. `app/api/chat.py`:
1. Next to the visible-content counter (`content_chars`, L934) add `visible_parts: list[str]` and append EVERY visible (post-scaffold-filter) content piece in the piece loop — the same pieces counted into `content_chars`, never `thinking` (D3: locators only ever come from the answer). Bounded by the model's output-token budget; note that in the comment.
2. In the `done` builder (L1320–L1365), after `answer_ref`: `blocks = resolve_answer_blocks(db, "".join(visible_parts), holder.shown_texts, settings) if not plan.deflected else []` (D9 — a deflected turn resolves NOTHING: no fence leaks KB text through a turn that just disclaimed it; the deflected path has no `holder`, so the guard is load-bearing). Convert with `VerbatimBlockRef(**dataclasses.asdict(b))` in order, pass `answer_text=blocks or None`.
3. Per-turn log line (L1272–L1291): append `verbatim_blocks=%d` after `scaffold_stripped=%d` (the count of blocks SENT, i.e. after dedupe + `show_text_max_blocks`; `0` on every turn without them, including deflected ones) and the module docstring's line-format block with the new field. This is the §9 additive revision — update `tests/unit/test_chat_gate.py`'s pinned line in the same commit, never silently.
4. `holder.shown_texts` is read AFTER the stream (the `shown_images` precedent) — do not reorder the generator's finally/cancel semantics (a cancelled turn still writes neither line nor `query_log` row).
5. Nothing in `app/schemas.py`'s existing validators changes; no new SSE event type (A15 stays at seven).
## Testing & Quality
- **Unit:** `tests/unit/test_chat_gate.py` (or the module that owns the per-turn line pins) — the line ends `… scaffold_stripped=N verbatim_blocks=0` on a plain grounded turn, `verbatim_blocks=2` on a turn whose holder has two blocks, `verbatim_blocks=0` on a deflected turn (and the deflected frame still carries no `answer_text`).
- **Integration:** the chat API/streaming suite — a scripted grounded turn that calls `show_text` yields a `done` frame whose `answer_text[0]` has the SERVER's `text`, the clamped extent, the canonical `locator`, and `truncated` false under the cap; a turn whose ANSWER contains a ` ```bor ` fence (no tool call) resolves the same block from the fence; fence + identical tool call produce exactly ONE entry; three blocks with `show_text_max_blocks=2` send two in answer order; a turn with no blocks serializes WITHOUT the `answer_text` key (byte-identity pin, exact-dict compare); `answer_text=False` yields the pre-phase frame byte-for-byte AND `verbatim_blocks=0`; a deflected frame carries no `answer_text` even when its (mocked) text contains a fence. The saved-chat suite — `POST /api/chats` with a brain record carrying `answer_text` is accepted (`extra="forbid"`), `GET /api/chats/{id}` returns it unchanged, and a record without it round-trips byte-identically to pre-phase.
- **Coverage:** **>90%** on this task's new/modified code.
## Completion Criteria
- [ ] A scripted SSE turn with a `show_text` call returns `answer_text: [{source,path,title,start_line,end_line,total_lines,text,truncated,locator}]` and the text equals `documents.content` for that range exactly.
- [ ] `grep ' :: api ' ` on that request shows the `done`-bearing response's byte line (the phase-167 audit) — record the observed size in the phase's task notes if it approaches the 256 KB budget.
- [ ] The per-turn log line shows `verbatim_blocks=N`; a plain turn shows `0`.
- [ ] `uv run pytest` green (incl. the updated line pins), coverage >90%, ruff + pyright clean.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the SSE `done` frame contract in README's API section; AGENTS rule 10's log-line list) — the phase's final pass re-checks both project-wide.
- [ ] no behavior change in completed phases (`answer_image`, `sources`, `related`, `budget_exhausted`, the deflected byte-identity, the cancelled-turn contract).
@@ -0,0 +1,43 @@
# Task 06 — `markdown.js`: the opt-in `bor` locator pass (model bytes never painted) + the panel CSS
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
Teach the shared renderer to recognize a ` ```bor ` locator fence and replace it — body and all — with a placeholder node, and write the panel's CSS. Inert: the pass is opt-in through a second argument that NO call site passes yet (task 07 opts the answer renderers in), so every existing render path stays byte-identical.
## Work
1. `frontend/assets/markdown.js` — `renderMarkdown(md)` becomes `renderMarkdown(md, opts)` (a SECOND, OPTIONAL argument; `opts && opts.verbatim === true` is the only truthy path). Header comment: the phase-182 pass, why the model's body is discarded, and the byte-identical-when-off contract.
2. Add the **verbatim pass BEFORE the existing fence pass** (order: verbatim → fences → tables → escape+inline → blockquotes → paragraphs → restore):
- it matches a fence whose info string is exactly `bor` + whitespace + a non-empty remainder: `` /^```bor[ \t]+\S[^\n]*\n(?:[\s\S]*?^```[ \t]*$|(?![\s\S]))/gm `` — the second alternative matches a fence left OPEN at the very end of the input, so a half-streamed locator shows the pending placeholder instead of raw backticks (a bare ` ```bor ` with no path still does NOT match: it is an ordinary code block and must survive to the fence pass untouched); note the deliberate asymmetry with the server's stricter `fences_in` (task 02 ignores unclosed fences — the server only ever sees a finished answer, the client sees a partial one);
- the matched locator text is run through the new local `borLocator(raw)` normalizer; if it returns `""` (unparseable), the match is put BACK byte-for-byte and left to the existing fence pass — a malformed locator degrades to whatever that pass does with this text today (no placeholder, no badge, the model's own body): this phase neither improves nor worsens it;
- a parsed one is replaced with `"\u0000VERBATIM" + n + "\u0000"` and recorded, and restored in the final restore step as:
`<div class="verbatim verbatim-pending" data-bor-locator="<escaped canonical>"><span class="verbatim-caption"><escaped source/path></span><span class="verbatim-state" role="status">Retrieving the exact text…</span></div>`
— every interpolated value goes through the existing `escapeHtml`; the model's body appears NOWHERE in the output (the phase's core contract, A7).
3. `borLocator(raw)` (module-local, docstring'd): trims; splits the path at the FIRST `/` (no `/` → invalid); accepts the separators ` `, `:`, `#` and `#L` before the range and the ranges `12-40`, `12–40` (en dash), `L12-L40`, `12`; returns `source/path` or `source/path#12-40` (canonical digits only), `""` for anything else (non-numeric, `< 1`, `start > end`, trailing junk). Documented as the CLIENT MIRROR of Python `app.rag.verbatim.canonical_locator` — the parity is pinned by the phase's E2E (both sides must agree or hydration misses), not by a unit test.
4. Do NOT wrap ordinary fences in markdown: task 07 wraps `<pre>` nodes at the DOM level (so the document viewer, the thinking blocks and every other caller keep byte-identical HTML).
5. `frontend/assets/styles.css` — a clearly-labelled phase-182 block. Palette variables ONLY (`--surface`, `--line`, `--ink`, `--ink-soft`, `--accent-*`, `--ok-*`; the phase-92 zero-literal rule — a monochrome theme must keep every state readable, WCAG "text + color, never color alone"):
- `.verbatim` — block, `margin` in the bubble's rhythm, `border: 1px solid var(--line)`, `border-radius: var(--radius-sm)`, `background: var(--surface)`, `min-width: 0`, `overflow: hidden`.
- `.verbatim-head` (row) + `.verbatim-caption` — the `source/path · lines 12–40 of 210` line, mono, `--ink-soft` (≥4.5:1), clipped with the existing `.cell-clip` ellipsis convention (phase 156) while the control's `aria-label`/`title` keeps the full text; `.verbatim-head` STICKS to the top of the scroll panel (`position: sticky; top: 0` + a matching background — PLAN §7.1's sticky-header rule for bounded panels).
- `.verbatim-badge` — a small TEXT badge reading `VERBATIM` (text, never an icon alone), `--ok` family, `text-transform: none` (no uppercase surprises for screen readers), and a `title` explaining "these bytes are copied from the knowledge base, not written by the model".
- `.verbatim-body` — the bounded scroll panel: `max-height: min(40rem, 70vh)`, `overflow: auto`, monospace, `white-space: pre`, `tab-size: 2`, `overflow-wrap: normal` (code scrolls horizontally, prose never does — phase 165's `word-wrap` split), `min-width: 0`.
- `.verbatim-lines` — the line-number gutter (`<ol>`/`<li>` or a `data-line` column, the implementer's choice, but numbers must be `user-select: none` so a Copy/select never drags line numbers into the clipboard), right-aligned, `--ink-soft`, `position: sticky; left: 0` optional.
- `.verbatim-pending` + `.verbatim-state` — the streaming/pending look (muted, no animation beyond what `prefers-reduced-motion` allows); `.verbatim-unresolved` — the honest failure state, `--err`/`--ink-soft`, no `VERBATIM` badge.
- `.code-block` (the task-07 DOM wrapper) + `.code-block-copy` / `.verbatim-copy` — a small button, `min-height: 44px` target on `@media (max-width: 640px)`, `:focus-visible` 3px ring like the rest of the app, `text-overflow: ellipsis` on its label, and a `.copied` state that changes TEXT (`Copy` → `Copied`), not color alone.
- The panel is a code surface, so it must stay readable in a light theme too: no hard-coded `#fff`/`#000`, no new hue.
## Testing & Quality
- **Unit (`tests/unit/test_frontend_verbatim_markdown.js` is not possible — use the project's text-pin convention, `tests/unit/test_frontend_verbatim_markdown.py`):** the `renderMarkdown(md, opts)` signature and the `opts.verbatim` guard exist; the verbatim pass runs BEFORE the fence pass; `\u0000VERBATIM` placeholders + the restore step exist; the output template contains `verbatim-pending`, `data-bor-locator`, `verbatim-caption`, `role="status"` and the pinned pending copy `Retrieving the exact text…`; `borLocator` exists and the separator/range tokens it handles are all present in its body; the ordinary-fence `<pre><code>` output string is unchanged in the source.
- **Unit (CSS, `tests/unit/test_frontend_verbatim_css.py`, the `test_frontend_cluster_css.py` pattern):** the phase-182 selectors exist; `.verbatim-body` has a `max-height` in the `min(…, 70vh)` form and `overflow: auto`; `.verbatim-head` is `position: sticky`; line numbers are `user-select: none`; no new hex/rgb literals in the phase-182 block (parse the block and assert every color is a `var(--…)`); the mobile block keeps a ≥44px control height; `prefers-reduced-motion` is honored by anything that animates.
- **Coverage:** **>90%** on the task's new/modified code (Python-side: the pin modules execute the assertions; there is no new `app/` code in this task, so `app/` coverage is unchanged — it must stay >90%).
## Completion Criteria
- [ ] `uv run pytest tests/unit/test_frontend_verbatim_markdown.py tests/unit/test_frontend_verbatim_css.py -v` green.
- [ ] Serving the app and calling `renderMarkdown("```bor homelab/a.md 12-40\nGARBAGE\n```", { verbatim: true })` in the browser console yields the placeholder DOM with `data-bor-locator="homelab/a.md#12-40"` and NO `GARBAGE` anywhere in the returned HTML; the same call WITHOUT the second argument returns the ordinary `<pre><code>` block.
- [ ] `renderMarkdown` output for the phase-44 table fixtures, the phase-138 blockquote fixtures and the phase-180 image fixtures is unchanged (the existing suites stay green).
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the renderer pass list in README's frontend section) — the phase's final pass re-checks both project-wide.
- [ ] no behavior change in completed work (document viewer, thinking blocks, user bubbles, tables, blockquotes, inline images).
@@ -0,0 +1,43 @@
# Task 07 — Hydration, panels, and a Copy button on every answer code block
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
Wire the client: resolve each placeholder against the frame's `answer_text`, render the panel (caption, `VERBATIM` badge, line numbers, Copy, viewer link), persist the blocks with the turn, and give every settled answer code block a Copy button. Live turn, restore-after-reload and the anonymous shared page all render identically.
## Work
1. Create `frontend/assets/verbatim.js` (classic script, like `markdown.js`) and include it AFTER `markdown.js` in `frontend/index.html` (L1696–L1697 block) and `frontend/shared.html` (L197–L198 block) — plain relative/absolute `src` exactly as the neighbors, the `?v=` token rewrite is the caching middleware's job (A19). Expose globals (the same pattern `markdown.js` uses), all DOM built with `createElement` + `textContent` (the frontend house rule — never an HTML string), all handlers via `addEventListener` (CSP `default-src 'self'`, no inline handlers):
- NO locator normalizer here — `markdown.js::borLocator` is the single implementation (task 06), and hydration compares the server's canonical `locator` against the already-normalized `data-bor-locator` attribute with a plain string compare. Re-implementing the grammar in a second file is the phase's main drift risk; do not.
- `buildVerbatimPanel(block, opts)` → the `.verbatim` node: `.verbatim-head` with `.verbatim-caption` (`source/path · lines 12–40 of 210`, or `source/path · all 210 lines` when the block is the whole document), `.verbatim-badge` (`VERBATIM`, `title` = "these bytes are copied from the knowledge base, not written by the model"), `.verbatim-copy` (`Copy`, `aria-label` = `Copy N lines from <source/path>` — the phase-156 clipped-label rule), an optional `.verbatim-open` link to the document (`opts.onOpen` hook: `app.js` passes the modal opener built from `documentUrl` + `openDocumentModal` (back → the chat, phase 153); `shared.js` passes a plain `/document?source=&path=` href — the shared page has no modal); `.verbatim-body` with the line-number gutter (`user-select: none`) and the text lines; the root gets `role="group"` + `aria-label` = `Verbatim text from <source/path><, lines N–M>`, and `data-bor-locator`. A truncated block (`block.truncated`) appends one muted line naming that the rest is in the document, plus the same open link. The git ORIGIN link stays the document viewer's job (phase 145) — the panel links to the viewer, it does not duplicate the origin.
- `hydrateVerbatim(root, blocks)` — for every `.verbatim[data-bor-locator]` in *root*: exact canonical-locator match in *blocks* → replace with the panel; else exactly ONE block sharing the same `source/path` → use it (the model's range was wrong; the panel shows the SERVER's extent — this is A2 made visible); else → the unresolved state: `.verbatim verbatim-unresolved` with the text `I could not verify this against the knowledge base — open <source/path>` (the link only when the path itself parsed) and NO badge, NO body, and none of the model's bytes. Never throws on a malformed/missing `blocks`.
- `appendUnlocatedVerbatim(wrap, blocks)` — the blocks no placeholder matched, appended in order at the `answer_image` position (below the answer text, above the source clusters, the phase-124/130 DOM order).
- `settleVerbatimPlaceholders(root, kind)` — turns leftover `.verbatim-pending` into the unresolved state (used by the error/stop paths so no panel ever sits at "Retrieving…" after a terminal frame).
- `enhanceAnswerCodeBlocks(root)` — every `<pre>` inside an answer bubble that is NOT inside a `.verbatim`: wrap in `.code-block` (once — idempotent on re-render) and append `.code-block-copy` with the same clipboard path. NO badge, NO caption — a `VERBATIM` badge belongs only to a server-resolved block (locked A5).
- `copyTextWithFallback(text, statusEl)` — the phase-55 degradation generalized from `app.js::copyShareLinkWithFallback` (L3250): try `navigator.clipboard.writeText`, on rejection render/return the inline select-to-copy fallback; returns `true` when the clipboard took it. `app.js`'s share path then DELEGATES to it (single implementation) — the owner-locked copy strings stay exactly as they are, `test_history_copy.py` / the share suites are the guard.
2. `frontend/assets/app.js`:
- the answer renderers pass the opt-in: the per-delta `renderMarkdown(acc)` (L4322), the `done`-time re-render (L4371) and the brain restore renderer (L2909) become `renderMarkdown(text, { verbatim: true })`. The USER bubble renderers (L1272, L1347, L2890) and every thinking-block render (L2841, L2939) stay as they are — user text and the scratchpad are not locators.
- the `done` handler (the `answer_image` neighborhood, L4340–L4360): `if (ev.answer_image) appendAnswerImage(...)` then `hydrateVerbatim(bubble, ev.answer_text || [])` then `appendUnlocatedVerbatim(wrap, ev.answer_text || [])` then `appendSources(...)`; `enhanceAnswerCodeBlocks(wrap)` runs once the turn settles (buttons appear with the settled answer, never mid-stream — pinned copy in the tests).
- persistence (the `rememberBrainTurn(...)` save point, L4390–L4420): `answer_text: ev.answer_text || undefined` (undefined drops the key — the `answer_image` pattern, so a pre-phase record stays byte-identical).
- the restore path re-renders + hydrates from `m.answer_text` (the `answer_image` restore precedent at L3001); a restored record with no `answer_text` renders its fences as the unresolved state (the record is what it is — never resurrect model bytes).
- the terminal failure paths (the `error` frame handler, the 300 s `TURN_TIMEOUT_MS` settle, the Stop action) call `settleVerbatimPlaceholders(bubble, "unverified")`.
- the Retry / edit-and-resend paths: nothing extra — they re-run the turn, the new turn hydrates on its own (verify the phase-120/159 suites stay green).
3. `frontend/assets/shared.js` — the record renderer (L619–L660, the `addAnswerImage` neighborhood) passes `{ verbatim: true }`, hydrates from `m.answer_text`, appends the unlocated blocks, enhances the settled code blocks, and uses the plain-href `onOpen` (no modal on the shared page). Parity is a hard requirement: the same blocks render the same panel on both surfaces.
4. `frontend/assets/styles.css` — extend task 06's block only where the real DOM needs it (states the JS adds: `.copied`, `.verbatim-unresolved`); palette variables only, no new literals.
## Testing & Quality
- **Unit (`tests/unit/test_frontend_verbatim_blocks.py`, the `test_frontend_*` text-pin convention):** `verbatim.js` is included after `markdown.js` in BOTH `index.html` and `shared.html`; `hydrateVerbatim` / `appendUnlocatedVerbatim` / `settleVerbatimPlaceholders` / `enhanceAnswerCodeBlocks` / `copyTextWithFallback` / `buildVerbatimPanel` exist; `app.js` passes `{ verbatim: true }` at the three brain renderers and does NOT pass it in the user/thinking renderers (assert by scoping the regex to the right call sites); the `done` handler hydrates BEFORE `appendSources`; `rememberBrainTurn` persists `answer_text` with `|| undefined`; `shared.js` reaches the same three globals (parity); no `onclick=`/`oncopy=` attribute strings anywhere in the new code (CSP); no `innerHTML += ` in the new code; the pinned unresolved copy `I could not verify this against the knowledge base`; the badge text `VERBATIM`; the copy button's `aria-label` template names the full path.
- **Unit (CSS):** the new state selectors exist and stay literal-free (task 06's module extends).
- **Coverage:** no new `app/` code; `app/` coverage must stay **>90%**, and the new pin module runs in the unit suite.
## Completion Criteria
- [ ] Live turn against a mock or real LLM: the panel shows, with the caption's numbers, the `VERBATIM` badge, line numbers, and a Copy button whose clipboard content equals the server text exactly; every ordinary code block in the same answer has a Copy button and NO badge.
- [ ] Reload the page: the restored bubble renders the identical panel. Open the Share link in an anonymous window: the shared page renders the identical panel.
- [ ] An answer with a ` ```bor ` fence whose path does not resolve shows the honest unresolved state with no model bytes; nothing sits at "Retrieving the exact text…" after a terminal frame (force it with the `slow_llm`/always-fail trigger).
- [ ] A `show_text` call whose block the answer never locates renders appended above the source clusters.
- [ ] `uv run pytest` green, coverage >90%, ruff + pyright clean; keyboard-only check: the copy button and the viewer link are reachable with a 3px `:focus-visible` ring, and the reduced-motion setting changes nothing jumpy.
- [ ] README.md/AGENTS.md updated for anything user-visible this task changes (the answer-panel feature list, AGENTS rule 5's UI contract line) — the phase's final pass re-checks both project-wide.
- [ ] no behavior change in completed phases (share-link copy strings, answer image, source clusters, related row, edit/resend, thinking timeline).
@@ -0,0 +1,50 @@
# Task 08 — Phase E2E suite, regression sweep, docs pass, commit
**Phase:** `182_verbatim_show_text` · **Story:** n/a (owner chat 2026-10-14)
## Objective
One dedicated Playwright file proving verbatim display end to end (including clipboard truth), the named regression suites green in isolation, the docs/README/AGENTS consistency pass, and the single atomic commit.
## Work
1. `tests/e2e/mock_llm.py` — three new scripted branches (module docstring updated; keep every existing branch byte-identical, and re-verify the phase-116 discipline: no new trigger phrase is a substring of, or contained in, any existing trigger or fixture question):
- `_SHOW_TEXT_TRIGGER = "show me the exact text of"` → request 1: a `show_text` tool call whose `path` is the parsed target (whole document, no `offset`/`limit`); the follow-up (request 2 carries the tool result starting with the new module constant `_SHOW_TEXT_CONFIRM_MARKER`, the `_READ_RESULT_PREFIX` precedent) → the final answer: prose + a ` ```bor ` fence carrying the target locator AND a DELIBERATELY CORRUPTED body (one fixture line with a character flipped and a trailing space) + a second fence for a second locator if the fixture has one. The corruption is the test: it must never reach the DOM.
- `_FENCE_ONLY_TRIGGER = "put the block inline"` → NO tool call; prose + one ` ```bor ` fence (resolved purely from the answer text — proves the standalone half of the mechanism).
- `_BAD_LOCATOR_TRIGGER = "show me a block I cannot find"` → NO tool call; prose + a ` ```bor ` fence pointing at a path that does not exist in the KB.
2. Fixture: seed (module-scoped truncate + `import_directory`-style insert, the `test_answer_images.py` idiom) TWO small text documents with distinctive, punctuation-rich content — one with a short line-count and obvious per-line identity (so a clamped range and its line numbers are checkable), one whose content includes a URL, backticks and an em dash (the escaping and the "no retype" proof). Fixture questions (asked by the happy-path spec, disjoint from every other trigger's question): *"show me the exact text of `baselab/quadlets/gufo.container`"*, *"put the block inline for `baselab/notes/verbatim-b.md 2-3`"*.
3. Create `tests/e2e/test_verbatim_show_text.py` (fixtures + style per `tests/e2e/README.md`; App A runs with the phase defaults, App B explicitly `BOR_ANSWER_TEXT=false`):
- **happy path (App A):** ask the show-me question → assert exactly FIVE tools were offered (assert on the request capture, the `test_agent_document_tools.py` idiom); a `.verbatim` panel exists whose caption shows `lines 1-N of N`; the panel's body text equals the fixture's content line-for-line; the badge `VERBATIM` is present; the answer's prose does NOT contain the corrupted line; `.verbatim-pending` is gone at settle; the panel's open link targets `/document?source=…&path=…` (the viewer, where the git origin link lives); the appended-block path renders above `.sources`.
- **clipboard truth:** grant `clipboard-read`/`clipboard-write` for the app origin (skip with a clear reason if the runner refuses), click the panel's Copy, read `navigator.clipboard.readText()`, assert byte equality with the fixture text fetched from the DB/API (no trailing newline, no line numbers); then click a plain code block's Copy in the same answer and assert IT copies exactly the model's block content (no line numbers, no badge).
- **fence only (App A):** the inline locator case resolves with no tool call in the request log, the model's body text is absent from `page.content()`, and no literal ` ```bor ` text remains in the DOM at settle (a half-typed fence must have become the pending state, then the panel).
- **unresolved (App A):** the bad-locator case renders the pinned unresolved copy, no `.verbatim-body`, no `VERBATIM`, and the model's bytes are absent.
- **wrong range corrected (App A):** drive the tool + a fence with an out-of-range `offset` (reuse the show-me trigger with a range suffix) → the panel shows the server's extent, not the model's claim; if the resolver refused the whole show, the honest unresolved state appears and the model's claimed numbers appear nowhere.
- **persistence:** after the happy path, reload `/history` and the open chat → the panel re-renders from the saved record; open the Share link in a second context (or `page.context().new_page()` in an incognito-style context) → the anonymous shared page shows the same panel; assert `/api/chat` for the same turn never emitted a second SSE event type (collect event names, assert `<= 7` distinct and `answer_text` came inside `done`).
- **line-number honesty:** assert the caption's numbers exist as a DOM range (`lines 1-… of …`) and that with a fixture of exactly N lines the `of N` matches — no invented numbers.
- **image regression (this phase's image check, per the owner):** a turn with `answer_image=True` still shows exactly one image bubble and NO `.verbatim` panel; `show_text` pointed at the image document (drive it via the tool branch if the mock can be pointed at the fixture image, else assert at unit level in task 03 and record it here) → the refusal copy naming `show_image`.
- **App B (`BOR_ANSWER_TEXT=false`):** exactly FOUR tools offered; asking the same show-me question yields no `.verbatim` element at all; a ` ```bor ` fence in the answer renders as an ordinary `<pre><code>` block; the `done` frame has no `answer_text` key; the Copy button on ordinary code blocks STILL works (A5 is flag-independent).
- **mobile:** at 390 px the panel scrolls horizontally without stretching the bubble and the Copy control is ≥44 px tall.
- **stale-UI check (owner's standing rule):** the pending → resolved transition never leaves a second panel behind (assert exactly one `.verbatim` per locator).
4. **Regression sweep (run in isolation, all green, no edits to the existing specs):** `test_answer_images.py`, `test_harness_aligned_tools.py`, `test_agent_document_tools.py`, `test_read_range_tool_lines.py`, `test_read_truncation_cap.py`, `test_tool_path_teaching.py`, `test_tool_scaffolding_guardrails.py`, `test_markdown_tables.py`, `test_markdown_blockquotes.py`, `test_long_answers.py`, `test_history_copy.py`, `test_chat_persistence.py`, `test_stale_ui_copy.py`. If `phase 180_markdown_inline_links_images` is still in `todo/`, run ITS E2E too (it shares `markdown.js`) and record the result.
5. **Docs consistency pass:**
- `README.md` — the feature (what the user sees: panel, `VERBATIM` badge, Copy on every code block), the `show_text` tool + the ` ```bor ` fence in the agent-tools/API sections, the `done.answer_text` field, the three env vars, the "the model points, the server supplies the bytes" principle stated once in plain words.
- `AGENTS.md` — rule 5's UI contract line gains the phase-182 clause (an answer code block always has a Copy button; only a server-resolved block carries `VERBATIM`; a `bor` block's model bytes are never painted); rule 10's log-line list mentions `verbatim_blocks=N`.
- `.agents/PLAN.md` — the tool surface (`show_text` as the 5th tool, gated like `show_image`), the `done` frame's additive field, the new settings; state explicitly that A1–A6 and A15 are otherwise UNCHANGED.
- `git status` must show nothing stale; every file this phase touched is in the commit.
6. **Commit** (mandatory, one atomic commit for the phase — Conventional Commits, `--no-gpg-sign` per repo default):
```bash
git add -A && git commit --no-gpg-sign -m "feat(rag): show file contents verbatim with a show_text tool and locator blocks
- add show_text(path, offset?, limit?): the server slices documents.content
and displays it, so the user copies the KB's bytes instead of a retyped
copy that may differ by a character
- resolve \`\`\`bor source/path 12-40 locator fences in answers server-side;
the model's fence body is never painted, unknown locators say so honestly
- carried in the done frame's additive answer_text array and persisted with
the saved chat, so restore and the shared page re-render identically
- Copy button on every answer code block, VERBATIM badge only on
server-resolved ones; line numbers always come from the server
- gated by BOR_ANSWER_TEXT with per-block/per-answer budgets; with the flag
off every frame, record and byte budget is exactly pre-phase"
```
Message in English (repo convention).
-2
View File
@@ -1,3 +1 @@
# TODO # TODO
- [ ] A long git clone seems to freeze the server? Investigate to find out if this is true.