Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 182_verbatim_show_text
Build and Push Containers / build-and-push-app (push) Successful in 1m46s
Details
Build and Push Containers / build-and-push-db (push) Successful in 12s
Details

Browse Source 
**Phase 182 — final verification pass (all 8 tasks already complete; this was verification + docs + one defect fix).**

- Verified: `app/rag/verbatim.py` (locator grammar, `splitlines()` slice, line-boundary cap, `resolve_answer_blocks`), `SHOW_TEXT_TOOL` + `shown_texts`, `SHOW_TEXT_CLAUSE`, `done.answer_text` (9 fields, key dropped when empty) + `ChatMessageRecord.answer_text` (≤50, 600 000-char cap counted in the 8 M budget), `verbatim.js` + 4 `{verbatim:true}` renderers, `Containerfile` minify step, `.env.example` 3 vars.
- `uv run python -c "…answer_text, show_text_max_chars, show_text_max_blocks"` → `True 12000 5`; each negative refuses startup naming its field; `0` accepted as inert ✅
- `bash .agents/validate.sh` → **validation OK**: 4320 unit+integration tests green, `TOTAL 6849 17 → 99%` (gate app/ >90% OK), `ruff check .` "All checks passed!", `pyright` 0 errors (re-run after my edits, 08:28).
- E2E, each in isolation `--no-cov`: `test_verbatim_show_text.py` 13 passed (panel bytes == `documents.content`, server caption, clipboard == server string, `/document.html?source=…&path=…`, honest unresolved state, reload+share, `show_image` regression, flag-off app) ✅; green: `test_answer_images` `test_harness_aligned_tools` `test_agent_document_tools` `test_markdown_tables` `test_markdown_inline_links_images` `test_svg_document_preview` `test_smoke` `test_document_viewer` `test_markdown_blockquotes` `test_chat_persistence` `test_thinking_timeline` `test_chat_rag` `test_settings_mobile_value_width`.
- **Defect found & fixed (pre-existing, not phase 182):** `tests/e2e/test_admin_settings_panel.py::test_admin_sees_groups_and_source_badges` failed — its whole-page `assert "reese" not in page.content()` tripped on the shell's onboarding chips, which follow the newest saved chats in the one shared E2E DB (a leftover `reeselink.json` question from a tool suite). Scoped the strict sweep to `.settings-shell` and kept a whole-page `reese:reese` credential needle → 12 passed with the leftovers still present.
- Docs: README fixed 2 inconsistencies — Quick Start omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the app fail-loud refuses boot) and the broken anchor `#chat--` → `#chat-`; full re-read of README (1816 lines) + AGENTS confirmed the phase-182 feature/settings/log/UI-contract text matches the code, and every `.env.example` var + every `Settings` field is documented.
- Notable: kept `resolve_answer_blocks` synchronous on the SSE terminal path (bounded indexed SELECTs, matching the existing `query_log` precedent and the phase's documented fence-lookup bound); no live-infra changes made.
- Next pending phase: none — `todo/` holds only 182 (its 8 task files are in `complete/`; the harness commits + moves `00_phase.md`).
This commit is contained in:
ducoterra
2026-10-15 08:33:12 -04:00
parent 814a400e6d
commit b34b8ca672
104 changed files with 10852 additions and 203 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/integration/test_chat_api.py
+416 -10
View File
@@ -36,10 +36,10 @@ from app.config import Settings, get_settings
from app.main import app as fastapi_app
from app.models import Chunk, Document, GitSource, QueryLog
from app.rag import agent
from app.rag.agent import AGENT_TOOLS, READ_TRUNCATION_NOTICE
from app.rag.agent import AGENT_TOOLS, READ_TRUNCATION_NOTICE, SHOW_TEXT_TOOL
from app.rag.importer import import_sources
from app.rag.llm import EmbeddingError, LLMError, StreamPiece, ToolCallPiece, ToolResultPiece
from app.rag.prompts import build_high_prompt
from app.rag.prompts import SHOW_TEXT_CLAUSE, build_high_prompt
from app.rag.retriever import TRUNCATION_MARKER
from app.schemas import ChatDoneEvent, SourceRef
from tests.conftest import ADMIN_PASSWORD
@@ -1291,9 +1291,12 @@ def test_grounded_turn_streams_tool_frames_and_cites_read_doc(
# (not spent budgets) bounds the loop, and the model answered while
# still being offered the tools (2 rounds < default cap 10).
assert len(scripted.seen_messages) == 3
assert scripted.seen_tools[0] == AGENT_TOOLS
assert scripted.seen_tools[1] == AGENT_TOOLS
assert scripted.seen_tools[2] == AGENT_TOOLS
# Phase 182 (task 03) pin update, deliberate: ``answer_text``
# defaults ON, so the grounded offer is the base three
# (element-for-element ``AGENT_TOOLS``) plus ``show_text``.
assert scripted.seen_tools[0] == [*AGENT_TOOLS, SHOW_TEXT_TOOL]
assert scripted.seen_tools[1] == [*AGENT_TOOLS, SHOW_TEXT_TOOL]
assert scripted.seen_tools[2] == [*AGENT_TOOLS, SHOW_TEXT_TOOL]
# The query_log row carries the same combined source list.
(row,) = db.scalars(select(QueryLog)).all()
@@ -2887,7 +2890,10 @@ def test_show_image_turn_done_frame_carries_answer_image(
# The cited tier includes the same doc (the scripted read).
assert ("docs", "pic.png") in [(s["source"], s["path"]) for s in done["sources"]]
# The toggle was honoured on the wire: the first (agent) request
# offered the 4th tool.
# offered the 4th tool. Phase 182 (task 03) pin update, deliberate:
# with ``answer_text`` ON (the shipped default) ``show_text`` rides
# LAST too — the ``answer_images`` gate itself is unchanged (it
# still adds exactly ``show_image``, 3rd-position-offer order kept).
offered = scripted.seen_tools[0]
assert offered is not None
assert [t["function"]["name"] for t in offered] == [
@@ -2895,6 +2901,7 @@ def test_show_image_turn_done_frame_carries_answer_image(
"read",
"grep",
"show_image",
"show_text",
]
@@ -2903,8 +2910,9 @@ def test_toggle_off_same_flow_offers_three_tools_and_no_answer_image_key(
) -> None:
"""The SAME scripted flow with ``answer_images=False`` (the
default) → ``show_image`` is NOT in the offered tools (the mock's
recorded request carries exactly the three base tools) and the
``done`` frame has NO ``answer_image`` key — the feature's
recorded request carries the three base tools — plus ``show_text``,
phase 182's default-ON display tool, pin updated here deliberately)
and the ``done`` frame has NO ``answer_image`` key — the feature's
server-side gate is the OFFER (a model that was never offered the
tool never calls it, so no such call occurs in the flow)."""
_seed_image_doc(db) # resolvable, but the tool is never offered
@@ -2928,10 +2936,16 @@ def test_toggle_off_same_flow_offers_three_tools_and_no_answer_image_key(
done = frames[-1]
assert done["deflected"] is False
assert "answer_image" not in done # ABSENT, not null
# The offer: exactly the three base tools — no ``show_image``.
# The offer: the three base tools — no ``show_image`` (phase 182:
# ``show_text`` rides last, the default being ON).
offered = scripted.seen_tools[0]
assert offered is not None
assert [t["function"]["name"] for t in offered] == ["ls", "read", "grep"]
assert [t["function"]["name"] for t in offered] == [
"ls",
"read",
"grep",
"show_text",
]
def test_zero_tool_grounded_turn_done_frame_has_no_answer_image_key(
@@ -3202,3 +3216,395 @@ def test_grounded_turn_suggested_plain_image_doc_block_is_byte_identical(
"</document>"
)
assert "Referenced in" not in system
# ---------- phase 182 (task 04): the show_text clause reaches the model ----------
def _gate_settings(**overrides: Any) -> Settings:
"""``Settings`` for the endpoint with the live honesty gate plus
*overrides* (the thinking-kill-switch pattern above) — so only the
flag under test changes from the calibrated default."""
live = get_settings()
return Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
relevance_threshold=live.relevance_threshold,
**overrides,
)
def test_grounded_turn_carries_show_text_clause_and_tool_per_the_flag(
client, db, seeded_kb: FakeRagLLM, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Phase 182 (task 04, LOCKED A1/A8): on a GROUNDED turn the model
request carries the ``SHOW_TEXT_CLAUSE`` inside ``<tools>`` exactly
when ``BOR_ANSWER_TEXT`` is on — and the ``show_text`` tool itself is
offered on the same condition, so the model is never taught a tool it
was not given nor handed one it was not taught (``answer_images``
stays off here, so the offer is the three base tools alone off, the
three base tools + ``show_text`` on)."""
for flag in (True, False):
monkeypatch.setattr(
chat_api, "get_settings", lambda f=flag: _gate_settings(answer_text=f)
)
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: seeded_kb
first_request = len(seeded_kb.seen_tools)
try:
_, _, frames = _stream_chat(client, QUESTION)
finally:
fastapi_app.dependency_overrides.clear()
assert frames[-1]["deflected"] is False, f"flag={flag}"
system = seeded_kb.seen_messages[first_request][0]["content"]
assert (SHOW_TEXT_CLAUSE in system) is flag, f"flag={flag}"
if flag:
assert system.index("<tools>") < system.index(SHOW_TEXT_CLAUSE)
assert system.index(SHOW_TEXT_CLAUSE) < system.index("</tools>")
offered = [t["function"]["name"] for t in (seeded_kb.seen_tools[first_request] or [])]
assert ("show_text" in offered) is flag, f"flag={flag}"
assert len(offered) == (4 if flag else 3), f"flag={flag}: {offered}"
# ---------------------------------------------------------------------------
# Phase 182 (task 05): the done frame's ADDITIVE ``answer_text`` array —
# the SERVER-resolved verbatim blocks (the ``show_text`` tool's and the
# ones the answer's own ```bor locator fences name), the byte-identity of
# every frame WITHOUT them, and the per-turn line's ``verbatim_blocks=N``
# tail. The resolver itself is pinned in tests/unit/test_verbatim_resolver.py;
# here the wire is the subject.
# ---------------------------------------------------------------------------
#: A fixture doc with a stable, countable body (16 lines as imported).
_K8S = "docs/homelab/kubernetes.md"
def _verbatim_settings(**overrides: Any) -> Settings:
"""The endpoint's settings view with the live gate plus *overrides*
(the ``_gate_settings`` pattern above) — only the flag under test
changes from the calibrated default."""
return _gate_settings(**overrides)
def _doc_lines(db, combined: str) -> list[str]:
"""The indexed row's content split the way the resolver splits it
(``splitlines()``, 1-based) — the authority the frame's text is
compared against, never a copy typed into this file."""
source, _, path = combined.partition("/")
content = db.scalar(
select(Document.content).where(
Document.source == source, Document.path == path
)
)
assert content is not None, combined
return content.splitlines()
def test_show_text_turn_done_frame_carries_the_server_block(
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
) -> None:
"""A scripted grounded turn whose model calls
``show_text(docs/homelab/kubernetes.md, offset=3, limit=100)`` → the
``done`` frame carries ONE ``answer_text`` entry whose ``text`` is
byte-identical to ``documents.content`` lines 3–16 (the overshooting
``limit`` CLAMPED, LOCKED A2), whose extent + ``total_lines`` are the
SERVER's, and whose ``locator`` is the canonical
``source/path#3-16`` (D7)."""
scripted = FakeRagLLM(
tool_script=[
[
ToolCallPiece(
id="call_1",
name="show_text",
arguments={"path": _K8S, "offset": 3, "limit": 100},
)
]
]
)
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
with caplog.at_level(logging.DEBUG, logger="app.response_budget"):
_, _, frames = _stream_chat(client, QUESTION)
finally:
fastapi_app.dependency_overrides.clear()
done = frames[-1]
lines = _doc_lines(db, _K8S)
# Phase 167's byte-budget line covers the done-bearing turn (the
# stream total, ``stream=1``) — the audit surface that shows the
# blocks' growth, pinned WELL inside the 256 KB budget here.
budget = [
r.getMessage()
for r in caplog.records
if " path=/api/chat " in r.getMessage()
]
assert len(budget) == 1, budget
size = int(re.search(r"bytes=(\d+)", budget[0]).group(1)) # pyright: ignore[reportOptionalMemberAccess]
assert 0 < size < 256 * 1024, budget[0]
assert done["deflected"] is False
assert done["answer_text"] == [
{
"source": "docs",
"path": "homelab/kubernetes.md",
"title": "Kubernetes Homelab Cluster",
"start_line": 3,
"end_line": len(lines),
"total_lines": len(lines),
"text": "\n".join(lines[2:]),
"truncated": False,
"locator": f"{_K8S}#3-{len(lines)}",
}
]
def test_answer_fence_resolves_the_same_block_without_a_tool_call(
client, db, seeded_kb: FakeRagLLM
) -> None:
"""D1/D3: the ANSWER alone carries a ```bor locator fence (no tool
call at all) and the block still resolves — resolution is a plain
``(source, path)`` lookup, not a replay of a tool call. The frame's
text is the DB's bytes: the model's (deliberately corrupt) fence body
appears NOWHERE on the wire."""
scripted = FakeRagLLM(
answer=(
"Here it is:\n"
"```bor docs/homelab/kubernetes.md 3-5\n"
"Exec=/MODEL_TYPED_LIES/never-render.bin\n"
"```\n"
"That is the whole range."
)
)
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
_, _, frames = _stream_chat(client, QUESTION)
finally:
fastapi_app.dependency_overrides.clear()
lines = _doc_lines(db, _K8S)
done = frames[-1]
assert done["answer_text"] == [
{
"source": "docs",
"path": "homelab/kubernetes.md",
"title": "Kubernetes Homelab Cluster",
"start_line": 3,
"end_line": 5,
"total_lines": len(lines),
"text": "\n".join(lines[2:5]),
"truncated": False,
"locator": f"{_K8S}#3-5",
}
]
# The model's fence body never reaches the wire as a RESOLVED block
# (the delta frames still carry the answer text — task 06/07 drop the
# body at render time; here the pin is that the frame's bytes are the
# DB's, not the model's).
assert done["answer_text"][0]["text"] == "\n".join(lines[2:5])
assert "MODEL_TYPED_LIES" not in json.dumps(done["answer_text"])
# The prose frames still stream the model's own answer text (the
# render-time body drop is task 06/07's job, pinned there) — the
# frame is what carries the truth.
answer = "".join(
f["text"] for f in frames if f.get("type") == "delta"
)
assert "MODEL_TYPED_LIES" in answer
def test_fence_plus_identical_tool_call_produce_exactly_one_block(
client, db, seeded_kb: FakeRagLLM
) -> None:
"""D2 dedupe: the tool shows the range AND the answer locates the
same canonical locator → exactly ONE entry (the fence wins, the block
is not painted twice)."""
scripted = FakeRagLLM(
answer=(
"The block:\n"
"```bor docs/homelab/kubernetes.md#3-5\n"
"```"
),
tool_script=[
[
ToolCallPiece(
id="call_1",
name="show_text",
arguments={"path": _K8S, "offset": 3, "limit": 3},
)
]
],
)
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
_, _, frames = _stream_chat(client, QUESTION)
finally:
fastapi_app.dependency_overrides.clear()
done = frames[-1]
assert len(done["answer_text"]) == 1
assert done["answer_text"][0]["locator"] == f"{_K8S}#3-5"
def test_three_blocks_with_max_blocks_two_send_two_in_answer_order(
client,
db,
seeded_kb: FakeRagLLM,
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""D8: three fences in the answer, ``BOR_SHOW_TEXT_MAX_BLOCKS=2`` →
exactly TWO entries, in ANSWER order (the cut is the resolver's, the
frame keeps its order, and the log line counts what was SENT)."""
monkeypatch.setattr(
chat_api, "get_settings", lambda: _verbatim_settings(show_text_max_blocks=2)
)
scripted = FakeRagLLM(
answer=(
"A:\n```bor docs/homelab/kubernetes.md 1-2\n```\n"
"B:\n```bor docs/homelab/backups.md 1-2\n```\n"
"C:\n```bor docs/deployments/new-service.md 1-2\n```"
)
)
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
with caplog.at_level("INFO", logger="app.chat"):
_, _, frames = _stream_chat(client, QUESTION)
finally:
fastapi_app.dependency_overrides.clear()
done = frames[-1]
assert [b["locator"] for b in done["answer_text"]] == [
"docs/homelab/kubernetes.md#1-2",
"docs/homelab/backups.md#1-2",
]
lines = [
r.getMessage() for r in caplog.records if "question=" in r.getMessage()
]
assert lines and lines[-1].endswith("verbatim_blocks=2")
def test_turn_without_blocks_serializes_without_the_answer_text_key(
client, db, seeded_kb: FakeRagLLM, caplog: pytest.LogCaptureFixture
) -> None:
"""A plain grounded turn (no tool call, no fence): the ``done`` frame
has NO ``answer_text`` key — checked on the RAW wire text — and its
key set is exactly the pre-phase one (the A9 byte-identity pin), with
``verbatim_blocks=0`` on the per-turn line."""
scripted = FakeRagLLM()
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
with caplog.at_level("INFO", logger="app.chat"):
_, _, frames = _stream_chat(client, QUESTION)
finally:
fastapi_app.dependency_overrides.clear()
done = frames[-1]
assert "answer_text" not in done
assert set(done) == {
"type",
"deflected",
"sources",
"related",
"suggestions",
}
lines = [
r.getMessage() for r in caplog.records if "question=" in r.getMessage()
]
assert lines and lines[-1].endswith("verbatim_blocks=0")
def test_answer_text_off_keeps_the_pre_phase_frame_and_no_tool(
client, db, seeded_kb: FakeRagLLM, monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""The master switch OFF (LOCKED A8/A9) with an answer that DOES
carry a fence: the frame is byte-for-byte the pre-phase shape (no
key), the line reads ``verbatim_blocks=0``, and the model was never
offered ``show_text`` — every byte-identity contract is exactly
pre-phase."""
monkeypatch.setattr(
chat_api, "get_settings", lambda: _verbatim_settings(answer_text=False)
)
scripted = FakeRagLLM(
answer=(
"Here it is:\n"
"```bor docs/homelab/kubernetes.md 3-5\n"
"Exec=/MODEL_TYPED_LIES/never-render.bin\n"
"```"
)
)
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
with caplog.at_level("INFO", logger="app.chat"):
_, _, frames = _stream_chat(client, QUESTION)
finally:
fastapi_app.dependency_overrides.clear()
done = frames[-1]
assert done["deflected"] is False
assert set(done) == {"type", "deflected", "sources", "related", "suggestions"}
offered = [t["function"]["name"] for t in (scripted.seen_tools[0] or [])]
assert "show_text" not in offered
lines = [
r.getMessage() for r in caplog.records if "question=" in r.getMessage()
]
assert lines and lines[-1].endswith("verbatim_blocks=0")
def test_deflected_frame_has_no_answer_text_even_with_a_fence(
client, db, seeded_kb: FakeRagLLM
) -> None:
"""D9: a deflected answer whose mocked text contains a fence still
resolves NOTHING — the frame has no ``answer_text`` key (the
disclaimer never leaks KB bytes) and no tools were offered."""
scripted = FakeRagLLM(
answer=(
"I haven't done anything like that.\n"
"```bor docs/homelab/kubernetes.md 3-5\n"
"```\n"
"Maybe try a saved runbook?"
)
)
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
_, _, frames = _stream_chat(client, OFF_TOPIC)
finally:
fastapi_app.dependency_overrides.clear()
done = frames[-1]
assert done["deflected"] is True
assert "answer_text" not in done
def test_resolution_failure_degrades_to_no_blocks(
client,
db,
seeded_kb: FakeRagLLM,
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
) -> None:
"""The answer already went out — a DB failure inside the done-time
resolution must not break the stream: the turn settles with the plain
pre-phase ``done`` frame (no key), the line reads
``verbatim_blocks=0``, and the failure is named on the log (the
query_log-write degradation precedent, never a silent swallow)."""
def _boom(*_a: Any, **_kw: Any) -> list[Any]:
raise RuntimeError("the KB went away mid-answer")
monkeypatch.setattr(chat_api, "resolve_answer_blocks", _boom)
scripted = FakeRagLLM(answer="Here:\n```bor docs/homelab/kubernetes.md 1-2\n```")
fastapi_app.dependency_overrides[chat_api.get_llm] = lambda: scripted
try:
with caplog.at_level("INFO", logger="app.chat"):
_, _, frames = _stream_chat(client, QUESTION)
finally:
fastapi_app.dependency_overrides.clear()
assert frames[-1]["type"] == "done"
assert "answer_text" not in frames[-1]
lines = [
r.getMessage()
for r in caplog.records
if "question=" in r.getMessage() or "verbatim" in r.getMessage()
]
assert any("verbatim block resolution failed" in m for m in lines)
turn = [r.getMessage() for r in caplog.records if "question=" in r.getMessage()]
assert turn and turn[-1].endswith("verbatim_blocks=0")