phase: 182_verbatim_show_text
**Phase 182 — final verification pass (all 8 tasks already complete; this was verification + docs + one defect fix).**
- Verified: `app/rag/verbatim.py` (locator grammar, `splitlines()` slice, line-boundary cap, `resolve_answer_blocks`), `SHOW_TEXT_TOOL` + `shown_texts`, `SHOW_TEXT_CLAUSE`, `done.answer_text` (9 fields, key dropped when empty) + `ChatMessageRecord.answer_text` (≤50, 600 000-char cap counted in the 8 M budget), `verbatim.js` + 4 `{verbatim:true}` renderers, `Containerfile` minify step, `.env.example` 3 vars.
- `uv run python -c "…answer_text, show_text_max_chars, show_text_max_blocks"` → `True 12000 5`; each negative refuses startup naming its field; `0` accepted as inert ✅
- `bash .agents/validate.sh` → **validation OK**: 4320 unit+integration tests green, `TOTAL 6849 17 → 99%` (gate app/ >90% OK), `ruff check .` "All checks passed!", `pyright` 0 errors (re-run after my edits, 08:28).
- E2E, each in isolation `--no-cov`: `test_verbatim_show_text.py` 13 passed (panel bytes == `documents.content`, server caption, clipboard == server string, `/document.html?source=…&path=…`, honest unresolved state, reload+share, `show_image` regression, flag-off app) ✅; green: `test_answer_images` `test_harness_aligned_tools` `test_agent_document_tools` `test_markdown_tables` `test_markdown_inline_links_images` `test_svg_document_preview` `test_smoke` `test_document_viewer` `test_markdown_blockquotes` `test_chat_persistence` `test_thinking_timeline` `test_chat_rag` `test_settings_mobile_value_width`.
- **Defect found & fixed (pre-existing, not phase 182):** `tests/e2e/test_admin_settings_panel.py::test_admin_sees_groups_and_source_badges` failed — its whole-page `assert "reese" not in page.content()` tripped on the shell's onboarding chips, which follow the newest saved chats in the one shared E2E DB (a leftover `reeselink.json` question from a tool suite). Scoped the strict sweep to `.settings-shell` and kept a whole-page `reese:reese` credential needle → 12 passed with the leftovers still present.
- Docs: README fixed 2 inconsistencies — Quick Start omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the app fail-loud refuses boot) and the broken anchor `#chat--` → `#chat-`; full re-read of README (1816 lines) + AGENTS confirmed the phase-182 feature/settings/log/UI-contract text matches the code, and every `.env.example` var + every `Settings` field is documented.
- Notable: kept `resolve_answer_blocks` synchronous on the SSE terminal path (bounded indexed SELECTs, matching the existing `query_log` precedent and the phase's documented fence-lookup bound); no live-infra changes made.
- Next pending phase: none — `todo/` holds only 182 (its 8 task files are in `complete/`; the harness commits + moves `00_phase.md`).
This commit is contained in:
@@ -0,0 +1,636 @@
|
||||
"""Unit: the shared renderer's opt-in ``bor`` locator pass (phase 182, task 06).
|
||||
|
||||
Browser behavior lands in the phase's Playwright suite (task 08); here the
|
||||
established ``test_frontend_*`` convention pins the SOURCE contract of
|
||||
``frontend/assets/markdown.js`` without a browser — the second, optional
|
||||
argument and the single truthy guard, the pass ORDER (verbatim BEFORE
|
||||
fences), the ``\\u0000VERBATIMn\\u0000`` placeholder + restore step, the exact
|
||||
pending template (``verbatim-pending`` / ``data-bor-locator`` /
|
||||
``verbatim-caption`` / ``role="status"`` / the pinned copy
|
||||
``Retrieving the exact text…``), the local ``borLocator`` normalizer and the
|
||||
separator/range grammar it accepts, and the fact that the ordinary-fence
|
||||
``<pre><code>`` output string is UNCHANGED (task 06 item 4: task 07 wraps
|
||||
``<pre>`` nodes at the DOM level instead, so the document viewer, the
|
||||
thinking blocks and every other caller keep byte-identical HTML).
|
||||
|
||||
The node-driven behavioral half (skipped when node is unavailable, the
|
||||
phase-44/138/180 idiom) proves the story shapes: the completion criterion —
|
||||
``renderMarkdown("```bor homelab/a.md 12-40\\nGARBAGE\\n```", {verbatim:
|
||||
true})`` yields the placeholder DOM with
|
||||
``data-bor-locator="homelab/a.md#12-40"`` and NO ``GARBAGE`` anywhere in the
|
||||
returned HTML (locked A7, the phase's core contract) — while the SAME call
|
||||
without the second argument returns the pre-phase output byte-for-byte; a
|
||||
bare `` ```bor `` (no path) survives to the fence pass untouched; an
|
||||
unparseable locator is put back byte-for-byte (no placeholder, no badge, the
|
||||
model's own body — neither improved nor worsened); a fence left OPEN at the
|
||||
very end of the input shows the pending state; two fences restore in answer
|
||||
order; a locator carrying markup stays inert escaped text; and a standalone
|
||||
placeholder is not wrapped in a ``<p>``.
|
||||
|
||||
Parity half: ``borLocator`` is the documented CLIENT MIRROR of Python
|
||||
``app.rag.verbatim.canonical_locator``. The phase's E2E (task 08) pins that
|
||||
parity end to end (both sides must agree or hydration misses); this module
|
||||
pins the GRAMMAR too, by running both implementations over the same table
|
||||
and over a deterministic fuzz — the only way to catch a grammar drift
|
||||
before there is a browser in the loop.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import random
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
|
||||
MARKDOWN_JS = FRONTEND / "assets" / "markdown.js"
|
||||
|
||||
HAVE_NODE = shutil.which("node") is not None
|
||||
|
||||
#: The single truthy path of the phase-182 second argument.
|
||||
VERBATIM_GUARD = "opts && opts.verbatim === true"
|
||||
|
||||
#: The verbatim fence pass — the pinned regex of task 06 item 2.
|
||||
VERBATIM_FENCE_RE = r"/^```bor[ \t]+\S[^\n]*\n(?:[\s\S]*?^```[ \t]*$|(?![\s\S]))/gm"
|
||||
|
||||
#: The pre-existing fence pass, untouched (task 06 item 4).
|
||||
ORDINARY_FENCE_RULE = (
|
||||
'codeBlocks.push(`<pre><code>${escapeHtml(code.replace(/\\n$/, ""))}</code></pre>`);'
|
||||
)
|
||||
|
||||
#: The untouched pre-phase paragraph guard (the phase-138/180 pin).
|
||||
PARAGRAPH_GUARD = r"/^<(h\d|ul|ol|pre|li|blockquote)/"
|
||||
|
||||
#: The placeholder's block-element clause added by task 06.
|
||||
PLACEHOLDER_GUARD = r'/^\u0000VERBATIM\d+\u0000$/'
|
||||
|
||||
#: The pinned pending copy (``…`` is ONE character, not three dots).
|
||||
PENDING_COPY = "Retrieving the exact text\u2026"
|
||||
|
||||
#: The completion criterion's input.
|
||||
CRITERION_INPUT = "```bor homelab/a.md 12-40\nGARBAGE\n```"
|
||||
|
||||
#: The pre-phase (verbatim-OFF) output of :data:`CRITERION_INPUT` — the
|
||||
#: byte-pin of the "inert until opted in" contract. A `` ```bor path `` info
|
||||
#: string never matched the ordinary fence rule (``/```(\w*)\n/`` needs the
|
||||
#: language token ALONE on that line), so OFF it is escaped prose.
|
||||
CRITERION_OFF_OUTPUT = "<p>```bor homelab/a.md 12-40<br>GARBAGE<br>```</p>"
|
||||
|
||||
#: The option task 07 will pass (never passed by any call site yet).
|
||||
ON: dict[str, bool] = {"verbatim": True}
|
||||
|
||||
#: The inputs the node cases render, keyed by case name.
|
||||
CASE_INPUT: dict[str, str] = {
|
||||
"criterion": CRITERION_INPUT,
|
||||
"bare_bor": "```bor\nfoo\n```",
|
||||
"open_bare": "```bor\n",
|
||||
"malformed": "```bor not-a-locator 12-40\nSECRET_BODY\n```",
|
||||
"open_fence": "```bor homelab/a.md 12-40\n",
|
||||
"two_fences": "```bor a/b 1-2\nBODY_ONE\n```\n\n```bor c/d\nBODY_TWO\n```",
|
||||
"en_dash": "```bor a/b 12\u201340\nBODY\n```",
|
||||
"l_prefix": "```bor a/b#L12-L40\nBODY\n```",
|
||||
"xss": "```bor a\"/><script>alert(1)</script>/b.md 1-2\nBODY\n```",
|
||||
"standalone": "prose\n\n```bor homelab/a.md\nBODY\n```\n\nafter",
|
||||
"attached": "prose\n```bor homelab/a.md 1-2\nBODY\n```",
|
||||
"nested": "```\ntext\n```bor a/b 1-2\nBODY\n```\n",
|
||||
"no_bor": "# Title\n\nsome **text** and `code`\n\n| a |\n|---|\n| 1 |\n\n> quote\n",
|
||||
}
|
||||
|
||||
|
||||
def _text(path: Path) -> str:
|
||||
return path.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def _run_node(script: str) -> str:
|
||||
proc = subprocess.run(["node", "-e", script], capture_output=True, text=True, timeout=60)
|
||||
assert proc.returncode == 0, f"node failed: {proc.stderr}"
|
||||
return proc.stdout
|
||||
|
||||
|
||||
def _render(cases: dict[str, dict[str, bool] | None]) -> dict[str, str]:
|
||||
"""Run every case through the REAL renderer under node.
|
||||
|
||||
``None`` means the ONE-argument call (every pre-phase call site); a dict
|
||||
is passed as the second argument as-is.
|
||||
"""
|
||||
parts = []
|
||||
for name, opts in cases.items():
|
||||
arg = "" if opts is None else f", {json.dumps(opts)}"
|
||||
parts.append(f"out.{name} = renderMarkdown({json.dumps(CASE_INPUT[name])}{arg}); ")
|
||||
driver = _text(MARKDOWN_JS) + "\nconst out = {}; " + "".join(parts)
|
||||
driver += "console.log(JSON.stringify(out));"
|
||||
results: dict[str, str] = json.loads(_run_node(driver))
|
||||
return results
|
||||
|
||||
|
||||
def _bor_locator_source() -> str:
|
||||
js = _text(MARKDOWN_JS)
|
||||
start = js.index("function borLocator(raw) {")
|
||||
end = js.index("\nfunction renderMarkdown", start)
|
||||
return js[start:end]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# the signature + the guard (task 06 item 1)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_render_markdown_takes_the_optional_opts_argument() -> None:
|
||||
"""``renderMarkdown(md)`` became ``renderMarkdown(md, opts)`` — a SECOND,
|
||||
OPTIONAL argument, so every existing call site (the document viewer, the
|
||||
thinking blocks, the user bubbles) keeps working untouched — and the
|
||||
ONLY truthy path is ``opts.verbatim === true`` (a strict ``===``, so
|
||||
``{verbatim: 1}`` / ``{verbatim: "yes"}`` stay OFF)."""
|
||||
js = _text(MARKDOWN_JS)
|
||||
assert "function renderMarkdown(md, opts) {" in js, (
|
||||
"renderMarkdown must gain the second, optional `opts` argument"
|
||||
)
|
||||
assert "function renderMarkdown(md) {" not in js, "the one-argument signature is gone"
|
||||
body = js[js.index("function renderMarkdown(md, opts) {") :]
|
||||
assert body.count(VERBATIM_GUARD) == 1, (
|
||||
f"the pass must be guarded by exactly one `{VERBATIM_GUARD}` test"
|
||||
)
|
||||
assert "if (opts && " in js, "a missing opts object must be handled (no TypeError)"
|
||||
|
||||
|
||||
def test_header_comment_records_the_phase_182_pass() -> None:
|
||||
"""The header comment carries the three things the next reader needs: the
|
||||
pass exists and is opt-in, WHY the model's body is discarded (locked A7),
|
||||
and the byte-identical-when-off contract."""
|
||||
js = _text(MARKDOWN_JS)
|
||||
header = js[: js.find("function escapeHtml")]
|
||||
flat = " ".join(
|
||||
" ".join(line.strip().lstrip("*").strip() for line in header.splitlines()).split()
|
||||
)
|
||||
for needle, label in [
|
||||
("Phase 182", "the phase reference"),
|
||||
("opts && opts.verbatim === true", "the single truthy path"),
|
||||
("BODY IS DISCARDED", "why the model's bytes are dropped"),
|
||||
("locked A7", "the locked decision behind it"),
|
||||
("canonical_locator", "the Python function this mirrors"),
|
||||
("BYTE-IDENTICAL WHEN OFF", "the inert-when-off contract"),
|
||||
("verbatim \u2192 fences \u2192 tables", "the new pass order"),
|
||||
("STRICTER", "the deliberate asymmetry with the server's fences_in"),
|
||||
("model's own body", "the malformed-locator degradation"),
|
||||
]:
|
||||
assert needle in flat, f"the header comment must record {label}: {needle!r}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# the pass + the placeholder (task 06 item 2)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_verbatim_pass_runs_before_the_fence_pass() -> None:
|
||||
"""Order is verbatim → fences → tables → escape+inline → blockquotes →
|
||||
paragraphs → restore. The verbatim regex MUST sit ahead of the ordinary
|
||||
fence replace, otherwise a locator fence is eaten as a code block first
|
||||
and the placeholder never exists."""
|
||||
js = _text(MARKDOWN_JS)
|
||||
assert VERBATIM_FENCE_RE in js, f"the pinned fence regex is missing: {VERBATIM_FENCE_RE}"
|
||||
fence_pass = js.find(ORDINARY_FENCE_RULE)
|
||||
assert fence_pass != -1, "the ordinary fence pass must still be there"
|
||||
assert js.find(VERBATIM_FENCE_RE) < fence_pass, (
|
||||
"the verbatim pass must run BEFORE the fence pass"
|
||||
)
|
||||
# …and the rest of the pipeline keeps its order behind it.
|
||||
for marker in ("const tables = []", "text = inline(escapeHtml(text));", r"split(/\n{2,}/)"):
|
||||
assert fence_pass < js.find(marker), f"the pass order broke at {marker!r}"
|
||||
|
||||
|
||||
def test_placeholder_is_created_and_restored() -> None:
|
||||
"""A parsed locator becomes ``\\u0000VERBATIMn\\u0000`` and the FINAL restore
|
||||
step (alongside TABLE/CODE) turns it back into the panel — the
|
||||
placeholder survives escaping (NUL and digits are not escaped) and never
|
||||
reaches the DOM itself."""
|
||||
js = _text(MARKDOWN_JS)
|
||||
assert "`\\u0000VERBATIM${verbatim.length - 1}\\u0000`" in js, (
|
||||
"the placeholder must be the indexed VERBATIM marker"
|
||||
)
|
||||
restore = r".replace(/\u0000VERBATIM(\d+)\u0000/g"
|
||||
assert restore in js, "the restore step must replace the VERBATIM placeholder"
|
||||
table = js.find(r".replace(/\u0000TABLE(\d+)\u0000/g")
|
||||
code = js.find(r".replace(/\u0000CODE(\d+)\u0000/g")
|
||||
assert table < code < js.find(restore), "the restore order is TABLE → CODE → VERBATIM"
|
||||
|
||||
|
||||
def test_pending_template_is_exact() -> None:
|
||||
"""The restore emits EXACTLY the pinned pending template: the
|
||||
``verbatim verbatim-pending`` panel, the canonical locator in
|
||||
``data-bor-locator`` (task 07's hydration key), the clipped
|
||||
``verbatim-caption``, and the ``role="status"`` state line carrying the
|
||||
pinned copy — every interpolated value through ``escapeHtml``."""
|
||||
js = _text(MARKDOWN_JS)
|
||||
for marker in (
|
||||
'<div class="verbatim verbatim-pending" data-bor-locator="${escapeHtml(canonical)}">',
|
||||
'<span class="verbatim-caption">${escapeHtml(identity)}</span>',
|
||||
f'<span class="verbatim-state" role="status">{PENDING_COPY}</span>',
|
||||
):
|
||||
assert marker in js, f"the pending template must contain {marker!r}"
|
||||
assert 'data-bor-locator="${canonical}"' not in js, (
|
||||
"the locator is model-written text landing in an ATTRIBUTE — it must "
|
||||
"go through escapeHtml (the escape-first boundary)"
|
||||
)
|
||||
|
||||
|
||||
def test_malformed_locator_is_put_back_byte_for_byte() -> None:
|
||||
"""``borLocator`` returning ``""`` returns the ORIGINAL match, so a
|
||||
malformed locator degrades to whatever the fence pass does with this text
|
||||
today (no placeholder, no badge, the model's own body) — this phase
|
||||
neither improves nor worsens it."""
|
||||
js = _text(MARKDOWN_JS)
|
||||
assert 'if (canonical === "") return match;' in js, (
|
||||
"an unparseable locator must be returned to the text unchanged"
|
||||
)
|
||||
|
||||
|
||||
def test_paragraph_guard_blocks_placeholder_and_untouched_guard_stays() -> None:
|
||||
"""The step-3 guard keeps the pre-phase test verbatim (the phase-138/180
|
||||
pin) and adds the placeholder clause, so a standalone panel restores as a
|
||||
``<div>`` instead of landing inside a ``<p>``."""
|
||||
js = _text(MARKDOWN_JS)
|
||||
assert PARAGRAPH_GUARD in js, "the pre-phase paragraph guard must stay"
|
||||
assert js.count(PARAGRAPH_GUARD) == 1, "there is exactly one paragraph guard"
|
||||
assert PLACEHOLDER_GUARD in js, "the placeholder must be a block element too"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# borLocator — the client mirror (task 06 item 3)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_bor_locator_exists_and_handles_every_grammar_token() -> None:
|
||||
"""``borLocator`` exists, is documented as the CLIENT MIRROR of
|
||||
``app.rag.verbatim.canonical_locator``, and its body carries every
|
||||
separator (`` ` ``, ``:``, ``#``, the ``L`` prefix) and range shape
|
||||
(``12-40``, the en dash, ``L12-L40``, a bare ``12``) of the D7 grammar —
|
||||
plus the refusals (non-numeric, ``< 1``, ``start > end``, trailing
|
||||
junk)."""
|
||||
body = _bor_locator_source()
|
||||
for token, label in [
|
||||
(r"[ \t:#]+", "the ` `/`:`/`#` separator class"),
|
||||
(r"L?\d+", "the optional `L` line prefix"),
|
||||
(r"[-\u2013]", "the hyphen AND the en dash range separator"),
|
||||
("(?:", "the OPTIONAL range group (a bare `12` and no range at all)"),
|
||||
("$/", "the end anchor (trailing junk is not a range)"),
|
||||
('includes("/")', "the first-`/` identity requirement"),
|
||||
("start < 1", "the `< 1` refusal"),
|
||||
("start > end", "the `start > end` refusal"),
|
||||
('replace(/^[Ll]+/, "")', "the `L` prefix strip on each bound"),
|
||||
]:
|
||||
assert token in body, f"borLocator must handle {label}: {token!r}"
|
||||
|
||||
|
||||
def test_bor_locator_is_documented_as_the_python_mirror() -> None:
|
||||
"""The docstring above ``borLocator`` names the Python function it
|
||||
mirrors and states that parity is load-bearing (a divergence means
|
||||
hydration misses), so nobody 'fixes' one side alone."""
|
||||
js = _text(MARKDOWN_JS)
|
||||
comment = js[: js.index("function borLocator(raw) {")]
|
||||
flat = " ".join(
|
||||
" ".join(line.strip().lstrip("/").strip() for line in comment.splitlines()).split()
|
||||
)
|
||||
for needle, label in [
|
||||
("canonical_locator", "the Python function"),
|
||||
("CLIENT MIRROR", "that this is the mirror, not the source"),
|
||||
("PARITY IS LOAD-BEARING", "why both sides must agree"),
|
||||
("#12-40", "the canonical range form"),
|
||||
]:
|
||||
assert needle in flat, f"the docstring must record {label}: {needle!r}"
|
||||
|
||||
|
||||
def test_bor_locator_returns_canonical_digits_or_empty() -> None:
|
||||
"""Source shape: the canonical output is ``identity#start-end`` (digits
|
||||
only) and every refusal path returns ``""`` — never ``null``/``undefined``
|
||||
(the renderer compares against ``""``)."""
|
||||
body = _bor_locator_source()
|
||||
assert "return `${identity}#${start}-${end}`;" in body
|
||||
assert body.count('return "";') >= 6, "every refusal path must return the empty string"
|
||||
assert "return null" not in body and "return undefined" not in body, (
|
||||
'borLocator\'s not-a-locator value is "", so the renderer\'s '
|
||||
'`if (canonical === "")` test cannot miss'
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# ordinary fences are NOT wrapped in markdown (task 06 item 4)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_ordinary_fence_output_string_is_unchanged() -> None:
|
||||
"""Task 06 item 4: do NOT wrap ordinary fences in markdown — task 07
|
||||
wraps ``<pre>`` nodes at the DOM level, so the document viewer, the
|
||||
thinking blocks and every other caller keep byte-identical HTML. The
|
||||
fence rule and its ``<pre><code>`` output string are pinned verbatim."""
|
||||
js = _text(MARKDOWN_JS)
|
||||
assert r"/```(\w*)\n([\s\S]*?)```/g" in js, "the ordinary fence regex is unchanged"
|
||||
assert ORDINARY_FENCE_RULE in js, "the ordinary fence output string is unchanged"
|
||||
assert 'class="code-block"' not in js, (
|
||||
"markdown.js must NOT emit the task-07 wrapper — that is a DOM-level wrap"
|
||||
)
|
||||
|
||||
|
||||
#: The answer renderers task 07 opted IN (the whole client switch — the
|
||||
#: per-delta write, the ``done``-time re-render, the brain restore).
|
||||
OPTED_IN_CALLS: tuple[str, ...] = (
|
||||
"renderMarkdown(acc, { verbatim: true })",
|
||||
"renderMarkdown(finalText, { verbatim: true })",
|
||||
"renderMarkdown(m.text, { verbatim: true })",
|
||||
)
|
||||
|
||||
|
||||
def test_only_the_answer_renderers_opted_in() -> None:
|
||||
"""Task 07 opted in exactly the THREE answer renderers (the per-delta
|
||||
write, the ``done``-time re-render, the brain restore) — and the shared
|
||||
page's one brain renderer. NOTHING else: the user-bubble renders, the
|
||||
live thinking runs, the legacy thinking restore and the whole document
|
||||
viewer stay one-argument calls, because a question and a scratchpad are
|
||||
not locators (task 06's byte-identity contract for every other surface).
|
||||
The count is the pin: a fourth ``verbatim: true`` in either file is a
|
||||
surface the phase never claimed."""
|
||||
app = _text(FRONTEND / "assets" / "app.js")
|
||||
shared = _text(FRONTEND / "assets" / "shared.js")
|
||||
for call in OPTED_IN_CALLS:
|
||||
assert call in app, f"app.js must render the answer through {call}"
|
||||
assert app.count("verbatim: true") == 3, (
|
||||
"exactly the three answer renderers may opt in in app.js"
|
||||
)
|
||||
assert (
|
||||
'addSharedMessage("brain", renderMarkdown(m.text, { verbatim: true }))' in shared
|
||||
)
|
||||
assert shared.count("verbatim: true") == 1, "the shared page opts in its one renderer"
|
||||
# The neighbours stay one-argument, in both files.
|
||||
assert 'addMessage("user", renderMarkdown(m.text), true)' in app
|
||||
assert 'addSharedMessage("user", renderMarkdown(m.text))' in shared
|
||||
for name in ("app.js", "shared.js"):
|
||||
js = _text(FRONTEND / "assets" / name)
|
||||
for site in re.findall(r"renderMarkdown\([^\n]*\)", js):
|
||||
if "thinking" in site or "entry.text" in site:
|
||||
assert "verbatim" not in site, f"{name}: {site} must stay one-argument"
|
||||
# A document's own ```bor fence is the document's text — never a locator.
|
||||
assert "verbatim: true" not in _text(FRONTEND / "assets" / "document.js")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# behavior (node): the story shapes + the completion criterion
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_locator_fence_yields_the_placeholder_and_drops_the_model_bytes() -> None:
|
||||
"""THE completion criterion: the criterion input WITH ``{verbatim: true}``
|
||||
yields the placeholder DOM with
|
||||
``data-bor-locator="homelab/a.md#12-40"`` and NO ``GARBAGE`` anywhere in
|
||||
the returned HTML (locked A7). Two fences restore in answer order, the en
|
||||
dash and the ``L`` prefix canonicalize, and a standalone panel is not
|
||||
wrapped in a ``<p>``."""
|
||||
out = _render(
|
||||
{
|
||||
"criterion": ON,
|
||||
"two_fences": ON,
|
||||
"en_dash": ON,
|
||||
"l_prefix": ON,
|
||||
"standalone": ON,
|
||||
}
|
||||
)
|
||||
html = out["criterion"]
|
||||
assert 'class="verbatim verbatim-pending"' in html
|
||||
assert 'data-bor-locator="homelab/a.md#12-40"' in html
|
||||
assert ">homelab/a.md</span>" in html, "the caption carries the source/path"
|
||||
assert f">{PENDING_COPY}</span>" in html
|
||||
assert 'role="status"' in html
|
||||
assert "GARBAGE" not in html, "the model's body must appear NOWHERE (locked A7)"
|
||||
assert "\u0000" not in html, "no placeholder byte survives"
|
||||
assert "<p>" not in html, "a standalone panel is a block, not a <p>"
|
||||
|
||||
two = out["two_fences"]
|
||||
assert two.index("a/b#1-2") < two.index('data-bor-locator="c/d"'), (
|
||||
"two fences keep answer order"
|
||||
)
|
||||
assert two.count("verbatim-pending") == 2
|
||||
assert "BODY_ONE" not in two and "BODY_TWO" not in two
|
||||
assert out["en_dash"].count('data-bor-locator="a/b#12-40"') == 1
|
||||
assert "BODY" not in out["en_dash"]
|
||||
assert out["l_prefix"].count('data-bor-locator="a/b#12-40"') == 1
|
||||
assert "BODY" not in out["l_prefix"]
|
||||
assert out["standalone"] == (
|
||||
'<p>prose</p><div class="verbatim verbatim-pending" '
|
||||
'data-bor-locator="homelab/a.md"><span class="verbatim-caption">homelab/a.md</span>'
|
||||
f'<span class="verbatim-state" role="status">{PENDING_COPY}</span></div><p>after</p>'
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_off_paths_are_byte_identical_to_the_pre_phase_renderer() -> None:
|
||||
"""Every call WITHOUT ``{verbatim: true}`` — no second argument, the
|
||||
document viewer's one-argument call, and an explicit
|
||||
``{verbatim: false}`` — returns the PRE-PHASE output byte-for-byte: the
|
||||
criterion input is escaped prose (a `` ```bor path `` info string never
|
||||
matched the ``/```(\\w*)\\n/`` fence rule), and a locator-free document
|
||||
renders identically with and without the option."""
|
||||
out = _render(
|
||||
{
|
||||
"criterion": None,
|
||||
"malformed": None,
|
||||
"two_fences": None,
|
||||
"standalone": None,
|
||||
"nested": None,
|
||||
"no_bor": None,
|
||||
}
|
||||
)
|
||||
assert out["criterion"] == CRITERION_OFF_OUTPUT, "the OFF path must not change a byte"
|
||||
for name, html in out.items():
|
||||
assert "verbatim" not in html, f"{name}: no placeholder markup with the option OFF"
|
||||
# the option object exists but is false → the same bytes as no argument
|
||||
on_false = _render({"criterion": {"verbatim": False}})
|
||||
assert on_false["criterion"] == CRITERION_OFF_OUTPUT
|
||||
truthy_but_strict = _render({"criterion": {}})
|
||||
assert truthy_but_strict["criterion"] == CRITERION_OFF_OUTPUT, (
|
||||
"the guard is `=== true`: an options object without the flag stays OFF"
|
||||
)
|
||||
# a locator-free document is untouched by the whole feature
|
||||
assert _render({"no_bor": ON})["no_bor"] == out["no_bor"]
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_bare_and_malformed_fences_fall_through_to_the_ordinary_pass() -> None:
|
||||
"""A bare `` ```bor `` (no path) is an ordinary code block and survives to
|
||||
the fence pass UNTOUCHED even with the option on; a bare `` ```bor ``
|
||||
with no body at all still paints its raw marker as prose; and an
|
||||
unparseable locator degrades to exactly what the fence pass does with
|
||||
this text today — the model's own body, no placeholder, no badge."""
|
||||
out = _render({"bare_bor": ON, "open_bare": ON, "malformed": ON})
|
||||
assert out["bare_bor"] == "<p><pre><code>foo</code></pre></p>", (
|
||||
"a bare ```bor fence must render EXACTLY as it does pre-phase"
|
||||
)
|
||||
assert out["open_bare"] == "<p>```bor</p>"
|
||||
assert out["malformed"] == "<p>```bor not-a-locator 12-40<br>SECRET_BODY<br>```</p>", (
|
||||
"a malformed locator is put back byte-for-byte: no placeholder, "
|
||||
"no badge, the model's own body"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_open_fence_at_the_end_shows_the_pending_state() -> None:
|
||||
"""The second alternative: a fence left OPEN at the very end of the input
|
||||
(the info line typed, the body not yet streamed) shows the pending
|
||||
placeholder instead of raw backticks — the deliberate asymmetry with the
|
||||
server's stricter ``fences_in`` (the server only ever sees a finished
|
||||
answer, the client sees a partial one)."""
|
||||
out = _render({"open_fence": ON})
|
||||
html = out["open_fence"]
|
||||
assert 'data-bor-locator="homelab/a.md#12-40"' in html
|
||||
assert "```" not in html, "no raw backticks while a locator fence is open"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_locator_markup_stays_inert_escaped_text() -> None:
|
||||
"""Escape-first holds on the new surface: a locator carrying ``"><script>``
|
||||
is HTML-escaped in BOTH the attribute and the caption, so it can never
|
||||
become a live element (the renderer's whole reason for existing)."""
|
||||
html = _render({"xss": ON})["xss"]
|
||||
assert "<script>" not in html
|
||||
assert "<script>" in html
|
||||
assert '"/><script>' in html
|
||||
assert "#1-2" in html, "the markup still canonicalized to a locator"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_a_bor_line_inside_an_ordinary_fence_agrees_with_the_server() -> None:
|
||||
"""Documented limitation shared with ``fences_in`` (both sides scan for
|
||||
the opening line, neither supports a nested `` ``` ``): a ``bor`` locator
|
||||
line inside an ordinary fence is a LOCATOR on both sides, so hydration
|
||||
still matches — it is never silently half-code, half-panel on one side
|
||||
only."""
|
||||
from app.rag.verbatim import fences_in
|
||||
|
||||
html = _render({"nested": ON})["nested"]
|
||||
assert 'data-bor-locator="a/b#1-2"' in html
|
||||
assert fences_in(CASE_INPUT["nested"]) == ["a/b 1-2"], (
|
||||
"the server must see the SAME locator in the same text"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_fence_attached_to_prose_still_loses_its_body() -> None:
|
||||
"""A locator fence with NO blank line above it still loses the model's
|
||||
body (the browser closes the surrounding ``<p>`` when it parses the
|
||||
``<div>``, the same fix-up an ordinary fenced ``<pre>`` already relies
|
||||
on) — the body never paints either way."""
|
||||
html = _render({"attached": ON})["attached"]
|
||||
assert 'data-bor-locator="homelab/a.md#1-2"' in html
|
||||
assert "BODY" not in html
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# parity with the server's canonical_locator (task 06 item 3)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
#: The D7 grammar table: input → the canonical string BOTH sides must
|
||||
#: produce, or ``""`` for "not a locator at all".
|
||||
PARITY_TABLE: dict[str, str] = {
|
||||
"homelab/a.md": "homelab/a.md",
|
||||
"homelab/a.md 12-40": "homelab/a.md#12-40",
|
||||
"homelab/a.md:12-40": "homelab/a.md#12-40",
|
||||
"homelab/a.md#12-40": "homelab/a.md#12-40",
|
||||
"homelab/a.md#L12-L40": "homelab/a.md#12-40",
|
||||
"homelab/a.md 12\u201340": "homelab/a.md#12-40",
|
||||
"homelab/a.md 12": "homelab/a.md#12-12",
|
||||
"homelab/a.md 12-40": "homelab/a.md#12-40",
|
||||
"homelab/a.md 12 - 40": "homelab/a.md#12-40",
|
||||
" homelab/a.md ": "homelab/a.md",
|
||||
"homelab/deep/dir/a.md#L3": "homelab/deep/dir/a.md#3-3",
|
||||
"homelab/a.md#l3-l7": "homelab/a.md#3-7",
|
||||
"homelab/a.md L7": "homelab/a.md#7-7",
|
||||
# the en dash is a RANGE separator, never the identity/range separator,
|
||||
# so this row is ONE whole-document identity whose path contains one:
|
||||
"homelab/a.md\u201312-40": "homelab/a.md\u201312-40",
|
||||
# refusals — not locators at all:
|
||||
"a.md": "",
|
||||
"": "",
|
||||
" ": "",
|
||||
"homelab/a.md 0-5": "",
|
||||
"homelab/a.md 40-12": "",
|
||||
"homelab/a.md x-y": "",
|
||||
"homelab/a.md 12-40 extra": "",
|
||||
"bor homelab/a.md": "",
|
||||
"homelab/a.md#": "",
|
||||
"homelab/a.md:": "",
|
||||
"homelab/a.md-": "",
|
||||
"homelab/my note.md 12": "",
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_bor_locator_matches_the_python_canonical_locator() -> None:
|
||||
"""The mirror is exact: for every row of the grammar table the JS
|
||||
``borLocator`` and the Python ``canonical_locator`` agree — and the table
|
||||
itself is checked against the D7 spec, so a SIMULTANEOUS drift on both
|
||||
sides is caught too. A divergence here means hydration misses: the panel
|
||||
would sit unresolved forever."""
|
||||
from app.rag.verbatim import canonical_locator
|
||||
|
||||
script = (
|
||||
_bor_locator_source()
|
||||
+ "\nconst t = "
|
||||
+ json.dumps(PARITY_TABLE)
|
||||
+ ";\nconst out = {};\nfor (const k of Object.keys(t)) out[k] = borLocator(k);\n"
|
||||
"console.log(JSON.stringify(out));\n"
|
||||
)
|
||||
got: dict[str, str] = json.loads(_run_node(script))
|
||||
for raw, expected in PARITY_TABLE.items():
|
||||
assert got[raw] == expected, f"borLocator({raw!r}) != {expected!r} (got {got[raw]!r})"
|
||||
python = canonical_locator(raw) or ""
|
||||
assert python == expected, (
|
||||
f"canonical_locator({raw!r}) = {python!r} disagrees with the D7 "
|
||||
f"table value {expected!r}"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_fuzz_locator_parity() -> None:
|
||||
"""A deterministic fuzz over the grammar's alphabet: the two
|
||||
implementations must agree on every string, including the shapes nobody
|
||||
hand-wrote (dangling separators, multi-bound ranges, lone ``L``s)."""
|
||||
from app.rag.verbatim import canonical_locator
|
||||
|
||||
rng = random.Random(182)
|
||||
alphabet = [
|
||||
"a", "b", "/", ".", ":", "#", " ", "L", "l", "-", "\u2013", "1", "2", "9", "x", "\t",
|
||||
]
|
||||
samples = [
|
||||
"".join(rng.choice(alphabet) for _ in range(rng.randint(0, 9))) for _ in range(400)
|
||||
]
|
||||
script = (
|
||||
_bor_locator_source()
|
||||
+ "\nconst t = "
|
||||
+ json.dumps(samples)
|
||||
+ ";\nconst out = [];\nfor (const k of t) out.push(borLocator(k));\n"
|
||||
"console.log(JSON.stringify(out));\n"
|
||||
)
|
||||
got: list[str] = json.loads(_run_node(script))
|
||||
for raw, js_value in zip(samples, got, strict=True):
|
||||
python = canonical_locator(raw) or ""
|
||||
assert js_value == python, f"parity break on {raw!r}: js={js_value!r} py={python!r}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# no regression on the neighboring passes (the phase's byte-identity mandate)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_adjacent_renderer_fixtures_stay_pinned_where_they_live() -> None:
|
||||
"""The phase-44 table, phase-138 blockquote and phase-180 image fixtures
|
||||
are the byte-identity evidence for the passes this task sits next to —
|
||||
they live in their own suites, so this pins that those suites still pin
|
||||
them (a deleted fixture here would silently drop the guarantee)."""
|
||||
for name, needle in [
|
||||
("test_markdown_tables.py", "md-table-wrap"),
|
||||
("test_markdown_blockquotes.py", '<blockquote class="md-quote">'),
|
||||
("test_markdown_inline_links_images.py", 'class="md-img"'),
|
||||
]:
|
||||
text = _text(Path(__file__).resolve().parent / name)
|
||||
assert needle in text, f"{name} must still pin {needle!r}"
|
||||
Reference in New Issue
Block a user