phase: 182_verbatim_show_text
**Phase 182 — final verification pass (all 8 tasks already complete; this was verification + docs + one defect fix).**
- Verified: `app/rag/verbatim.py` (locator grammar, `splitlines()` slice, line-boundary cap, `resolve_answer_blocks`), `SHOW_TEXT_TOOL` + `shown_texts`, `SHOW_TEXT_CLAUSE`, `done.answer_text` (9 fields, key dropped when empty) + `ChatMessageRecord.answer_text` (≤50, 600 000-char cap counted in the 8 M budget), `verbatim.js` + 4 `{verbatim:true}` renderers, `Containerfile` minify step, `.env.example` 3 vars.
- `uv run python -c "…answer_text, show_text_max_chars, show_text_max_blocks"` → `True 12000 5`; each negative refuses startup naming its field; `0` accepted as inert ✅
- `bash .agents/validate.sh` → **validation OK**: 4320 unit+integration tests green, `TOTAL 6849 17 → 99%` (gate app/ >90% OK), `ruff check .` "All checks passed!", `pyright` 0 errors (re-run after my edits, 08:28).
- E2E, each in isolation `--no-cov`: `test_verbatim_show_text.py` 13 passed (panel bytes == `documents.content`, server caption, clipboard == server string, `/document.html?source=…&path=…`, honest unresolved state, reload+share, `show_image` regression, flag-off app) ✅; green: `test_answer_images` `test_harness_aligned_tools` `test_agent_document_tools` `test_markdown_tables` `test_markdown_inline_links_images` `test_svg_document_preview` `test_smoke` `test_document_viewer` `test_markdown_blockquotes` `test_chat_persistence` `test_thinking_timeline` `test_chat_rag` `test_settings_mobile_value_width`.
- **Defect found & fixed (pre-existing, not phase 182):** `tests/e2e/test_admin_settings_panel.py::test_admin_sees_groups_and_source_badges` failed — its whole-page `assert "reese" not in page.content()` tripped on the shell's onboarding chips, which follow the newest saved chats in the one shared E2E DB (a leftover `reeselink.json` question from a tool suite). Scoped the strict sweep to `.settings-shell` and kept a whole-page `reese:reese` credential needle → 12 passed with the leftovers still present.
- Docs: README fixed 2 inconsistencies — Quick Start omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the app fail-loud refuses boot) and the broken anchor `#chat--` → `#chat-`; full re-read of README (1816 lines) + AGENTS confirmed the phase-182 feature/settings/log/UI-contract text matches the code, and every `.env.example` var + every `Settings` field is documented.
- Notable: kept `resolve_answer_blocks` synchronous on the SSE terminal path (bounded indexed SELECTs, matching the existing `query_log` precedent and the phase's documented fence-lookup bound); no live-infra changes made.
- Next pending phase: none — `todo/` holds only 182 (its 8 task files are in `complete/`; the harness commits + moves `00_phase.md`).
This commit is contained in:
@@ -244,16 +244,16 @@ def _refuse_embedded_credentials(key: str, value: str) -> None:
|
||||
def get_catalogue(
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
) -> AdminSettingsOut:
|
||||
"""The whole catalogue (78 items, ``GROUPS`` order — the stable
|
||||
order the E2E pins against).
|
||||
"""The whole catalogue (one item per ``Settings`` field, ``GROUPS``
|
||||
order — the stable order the E2E pins against).
|
||||
|
||||
Reads the table once, resolves the effective ``Settings`` through
|
||||
``get_settings()`` (the overlay — DB-over-env, task 02), and
|
||||
emits one item per field. GETs log nothing here: the phase-167
|
||||
``ResponseBudgetMiddleware`` already writes the
|
||||
``api method=… bytes=…`` line, and the whole catalogue must stay
|
||||
under ``BOR_LOG_LARGE_RESPONSE_BYTES``' INFO bar (78 items ≈ 20 KB
|
||||
— verified in task 06 with the audit script).
|
||||
under ``BOR_LOG_LARGE_RESPONSE_BYTES``' INFO bar (the whole catalogue
|
||||
is ≈ 20 KB — verified in task 06 with the audit script).
|
||||
"""
|
||||
rows = _rows(db)
|
||||
effective = get_settings()
|
||||
@@ -371,7 +371,7 @@ def reset_value(
|
||||
@router.delete("", response_model=AdminSettingsCleared)
|
||||
def reset_all(db: Session = Depends(get_db)) -> AdminSettingsCleared: # noqa: B008
|
||||
"""The panel's "Clear all overrides": delete every row (the env
|
||||
values win again for all 78 settings), invalidate (task 03) and
|
||||
values win again for every setting), invalidate (task 03) and
|
||||
report how many rows went."""
|
||||
result = db.execute(delete(AppSetting).returning(AppSetting.key))
|
||||
db.commit()
|
||||
|
||||
+97
-2
@@ -227,6 +227,30 @@ pre-phase. A question image is turn-local: it is NEVER indexed as a
|
||||
document, and prior turns' images are never replayed into the model's
|
||||
history (``history_to_messages`` is unchanged — the text of a prior
|
||||
turn that had an image stands alone, LOCKED A7).
|
||||
|
||||
Verbatim display (phase 182, tasks 03/05; LOCKED A1/A4/A7/A9): a turn
|
||||
that showed file contents through the ``show_text`` tool (task 03, the
|
||||
bytes the SERVER resolved) — or whose ANSWER carries a `` ```bor ``
|
||||
locator fence (task 02's resolver, placement sugar that resolves
|
||||
standalone) — gains one additive ``done`` key, ``answer_text``: the
|
||||
list of :class:`~app.schemas.VerbatimBlockRef` blocks in DISPLAY order,
|
||||
each carrying the exact ``documents.content`` slice plus the SERVER's
|
||||
resolved 1-based, inclusive line extent and its canonical ``locator``.
|
||||
The model never supplies those bytes: the endpoint accumulates the
|
||||
turn's CLEAN VISIBLE answer text (the same post-scaffold-filter pieces
|
||||
that feed ``content_chars`` — never ``thinking``, so a locator can only
|
||||
ever come from the answer) and resolves it through
|
||||
:func:`app.rag.verbatim.resolve_answer_blocks` when the ``done`` frame
|
||||
is built. A deflected turn resolves NOTHING (D9 — a fence hallucinated
|
||||
into a disclaimer must not leak KB text through the turn that just said
|
||||
it had nothing); ``answer_text=False``, ``show_text_max_blocks=0`` and
|
||||
``show_text_max_chars=0`` short-circuit inside the resolver, and a turn
|
||||
with no blocks serializes WITHOUT the key (A9 — every block-free frame
|
||||
keeps its exact pre-phase bytes). The per-turn log line records
|
||||
``verbatim_blocks=N`` after ``scaffold_stripped=N`` — the blocks
|
||||
ACTUALLY SENT (after dedupe and the ``show_text_max_blocks`` cut), 0 on
|
||||
every turn without them including deflected ones (the PLAN §9 additive
|
||||
revision).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -235,7 +259,7 @@ import json
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import AsyncIterator, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
@@ -283,6 +307,9 @@ from app.rag.summarizer import (
|
||||
IMAGE_MIMES,
|
||||
image_data_url,
|
||||
) # phase 123: the phase-122 mime map + the shared data-URL helper
|
||||
from app.rag.verbatim import (
|
||||
resolve_answer_blocks, # phase 182: fences + shown blocks -> the done frame
|
||||
)
|
||||
from app.schemas import (
|
||||
ChatDoneEvent,
|
||||
ChatErrorEvent,
|
||||
@@ -292,6 +319,7 @@ from app.schemas import (
|
||||
ChatToolEvent,
|
||||
ChatToolResultEvent,
|
||||
SourceRef,
|
||||
VerbatimBlockRef,
|
||||
)
|
||||
|
||||
logger = logging.getLogger("app.chat")
|
||||
@@ -527,6 +555,12 @@ def plan_turn(
|
||||
# carries the show_image clause exactly when the
|
||||
# BOR_ANSWER_IMAGES toggle is on (off = byte-identical).
|
||||
show_image=settings.answer_images,
|
||||
# Phase 182 (task 04, LOCKED A1/A8): the same rule for the
|
||||
# show_text clause — it rides exactly when the BOR_ANSWER_TEXT
|
||||
# master switch is on (the default), and an off build is
|
||||
# byte-identical to pre-phase (no clause, and agent_tools
|
||||
# offers four tools, not five).
|
||||
show_text=settings.answer_text,
|
||||
),
|
||||
suggested,
|
||||
related_docs,
|
||||
@@ -933,6 +967,16 @@ async def chat(
|
||||
thinking_chars = 0
|
||||
content_chars = 0 # phase 71: the turn's visible (clean) content
|
||||
scaffold_stripped = 0 # phase 71: sum across the turn's requests
|
||||
# Phase 182 (task 05, D3): the turn's clean VISIBLE answer
|
||||
# text — the SAME pieces counted into ``content_chars``
|
||||
# (post-scaffold-filter, never ``thinking``, so a locator
|
||||
# can only ever come from the answer), kept so the ``done``
|
||||
# builder can resolve the `` ```bor `` fences the answer
|
||||
# wrote against the indexed rows. Bounded by the model's own
|
||||
# output-token budget (``max_output_tokens``) — the answer
|
||||
# already lives in memory for the duration of the turn, so
|
||||
# this adds no new cap.
|
||||
visible_parts: list[str] = []
|
||||
# Phase 160 (task 02): the transport half of the cap path — a
|
||||
# request made WITHOUT tools (the deflected turn, A8, and its
|
||||
# recovery) never produces a ``tool`` frame: a structured
|
||||
@@ -1082,6 +1126,9 @@ async def chat(
|
||||
yield sse_event(ChatThinkingEvent(text=piece.text).model_dump())
|
||||
else:
|
||||
content_chars += len(piece.text)
|
||||
# Phase 182 (task 05): the same clean piece kept
|
||||
# as TEXT for the done-time locator pass.
|
||||
visible_parts.append(piece.text)
|
||||
yield sse_event({"type": "delta", "text": piece.text})
|
||||
if dropped_calls:
|
||||
logger.debug(
|
||||
@@ -1269,12 +1316,46 @@ async def chat(
|
||||
except Exception: # noqa: BLE001 — the answer already went out
|
||||
logger.exception("chat: failed to write query_log question=%r", request.message)
|
||||
|
||||
# Phase 182 (task 05, D2/D9): resolve the turn's verbatim
|
||||
# display blocks — the ``show_text`` tool's blocks
|
||||
# (``holder.shown_texts``, read AFTER the stream exactly
|
||||
# like ``shown_images``) fused with the blocks the answer's
|
||||
# own `` ```bor `` locator fences name, over the CLEAN
|
||||
# visible text accumulated above (never ``thinking``).
|
||||
# A DEFLECTED turn resolves NOTHING (D9 — a fence
|
||||
# hallucinated into a disclaimer must not leak KB text
|
||||
# through the turn that just said it had nothing), which is
|
||||
# also why this is the only branch with a resolved holder.
|
||||
# A DB failure here must not break a stream whose answer
|
||||
# already went out (the query_log precedent above): the
|
||||
# turn degrades to NO blocks, the frame simply omits the
|
||||
# key, and the client renders each fence in its honest
|
||||
# unresolved state — never the model's own bytes.
|
||||
blocks: list[VerbatimBlockRef] = []
|
||||
if not plan.deflected:
|
||||
try:
|
||||
with SessionLocal() as verbatim_db:
|
||||
resolved = resolve_answer_blocks(
|
||||
verbatim_db,
|
||||
"".join(visible_parts),
|
||||
holder.shown_texts,
|
||||
settings,
|
||||
)
|
||||
blocks = [VerbatimBlockRef(**asdict(b)) for b in resolved]
|
||||
except Exception: # noqa: BLE001 — the answer went out
|
||||
logger.exception(
|
||||
"chat: verbatim block resolution failed question=%r",
|
||||
request.message,
|
||||
)
|
||||
blocks = []
|
||||
|
||||
logger.info(
|
||||
"question=%r embed_ms=%d top_score=%.3f fts_hits=%d summary_hits=%d "
|
||||
"suggested=%d tuning=%d "
|
||||
"kb_chars=%d history_msgs=%d threshold=%.2f deflected=%s sources=%r "
|
||||
"thinking=%s thinking_chars=%d tool_calls=%d rounds=%d "
|
||||
"cap_hit=%d total_ms=%d retries=%d scaffold_stripped=%d",
|
||||
"cap_hit=%d total_ms=%d retries=%d scaffold_stripped=%d "
|
||||
"verbatim_blocks=%d",
|
||||
request.message,
|
||||
embed_ms,
|
||||
plan.top_score,
|
||||
@@ -1299,6 +1380,12 @@ async def chat(
|
||||
total_ms,
|
||||
retries_used,
|
||||
scaffold_stripped,
|
||||
# phase 182 (task 05): the blocks ACTUALLY SENT (after
|
||||
# the canonical-locator dedupe and the
|
||||
# ``show_text_max_blocks`` cut) — 0 on every turn
|
||||
# without them, deflected ones included (PLAN §9
|
||||
# additive revision).
|
||||
len(blocks),
|
||||
)
|
||||
settled = True # terminal: the done frame settles the turn
|
||||
# Phase 112 (A8 revised, TODO L2): a deflected turn cites
|
||||
@@ -1360,6 +1447,14 @@ async def chat(
|
||||
# (``False`` → ABSENT on serialization) is
|
||||
# correct by construction.
|
||||
budget_exhausted=holder.cap_reached,
|
||||
# Phase 182 (task 05): the answer's verbatim display
|
||||
# blocks, resolved above in DISPLAY order (answer
|
||||
# order first, then the tool blocks the answer never
|
||||
# located). ``blocks or None`` — an empty list is
|
||||
# ``None`` here and the serializer drops the key
|
||||
# either way, so a block-free frame keeps its exact
|
||||
# pre-phase bytes (LOCKED A9).
|
||||
answer_text=blocks or None,
|
||||
).model_dump()
|
||||
)
|
||||
finally:
|
||||
|
||||
Reference in New Issue
Block a user