**Phase 182 — final verification pass (all 8 tasks already complete; this was verification + docs + one defect fix).**
- Verified: `app/rag/verbatim.py` (locator grammar, `splitlines()` slice, line-boundary cap, `resolve_answer_blocks`), `SHOW_TEXT_TOOL` + `shown_texts`, `SHOW_TEXT_CLAUSE`, `done.answer_text` (9 fields, key dropped when empty) + `ChatMessageRecord.answer_text` (≤50, 600 000-char cap counted in the 8 M budget), `verbatim.js` + 4 `{verbatim:true}` renderers, `Containerfile` minify step, `.env.example` 3 vars.
- `uv run python -c "…answer_text, show_text_max_chars, show_text_max_blocks"` → `True 12000 5`; each negative refuses startup naming its field; `0` accepted as inert ✅
- `bash .agents/validate.sh` → **validation OK**: 4320 unit+integration tests green, `TOTAL 6849 17 → 99%` (gate app/ >90% OK), `ruff check .` "All checks passed!", `pyright` 0 errors (re-run after my edits, 08:28).
- E2E, each in isolation `--no-cov`: `test_verbatim_show_text.py` 13 passed (panel bytes == `documents.content`, server caption, clipboard == server string, `/document.html?source=…&path=…`, honest unresolved state, reload+share, `show_image` regression, flag-off app) ✅; green: `test_answer_images` `test_harness_aligned_tools` `test_agent_document_tools` `test_markdown_tables` `test_markdown_inline_links_images` `test_svg_document_preview` `test_smoke` `test_document_viewer` `test_markdown_blockquotes` `test_chat_persistence` `test_thinking_timeline` `test_chat_rag` `test_settings_mobile_value_width`.
- **Defect found & fixed (pre-existing, not phase 182):** `tests/e2e/test_admin_settings_panel.py::test_admin_sees_groups_and_source_badges` failed — its whole-page `assert "reese" not in page.content()` tripped on the shell's onboarding chips, which follow the newest saved chats in the one shared E2E DB (a leftover `reeselink.json` question from a tool suite). Scoped the strict sweep to `.settings-shell` and kept a whole-page `reese:reese` credential needle → 12 passed with the leftovers still present.
- Docs: README fixed 2 inconsistencies — Quick Start omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the app fail-loud refuses boot) and the broken anchor `#chat--` → `#chat-`; full re-read of README (1816 lines) + AGENTS confirmed the phase-182 feature/settings/log/UI-contract text matches the code, and every `.env.example` var + every `Settings` field is documented.
- Notable: kept `resolve_answer_blocks` synchronous on the SSE terminal path (bounded indexed SELECTs, matching the existing `query_log` precedent and the phase's documented fence-lookup bound); no live-infra changes made.
- Next pending phase: none — `todo/` holds only 182 (its 8 task files are in `complete/`; the harness commits + moves `00_phase.md`).
637 lines
30 KiB
Python
637 lines
30 KiB
Python
"""Unit: the shared renderer's opt-in ``bor`` locator pass (phase 182, task 06).
|
|
|
|
Browser behavior lands in the phase's Playwright suite (task 08); here the
|
|
established ``test_frontend_*`` convention pins the SOURCE contract of
|
|
``frontend/assets/markdown.js`` without a browser — the second, optional
|
|
argument and the single truthy guard, the pass ORDER (verbatim BEFORE
|
|
fences), the ``\\u0000VERBATIMn\\u0000`` placeholder + restore step, the exact
|
|
pending template (``verbatim-pending`` / ``data-bor-locator`` /
|
|
``verbatim-caption`` / ``role="status"`` / the pinned copy
|
|
``Retrieving the exact text…``), the local ``borLocator`` normalizer and the
|
|
separator/range grammar it accepts, and the fact that the ordinary-fence
|
|
``<pre><code>`` output string is UNCHANGED (task 06 item 4: task 07 wraps
|
|
``<pre>`` nodes at the DOM level instead, so the document viewer, the
|
|
thinking blocks and every other caller keep byte-identical HTML).
|
|
|
|
The node-driven behavioral half (skipped when node is unavailable, the
|
|
phase-44/138/180 idiom) proves the story shapes: the completion criterion —
|
|
``renderMarkdown("```bor homelab/a.md 12-40\\nGARBAGE\\n```", {verbatim:
|
|
true})`` yields the placeholder DOM with
|
|
``data-bor-locator="homelab/a.md#12-40"`` and NO ``GARBAGE`` anywhere in the
|
|
returned HTML (locked A7, the phase's core contract) — while the SAME call
|
|
without the second argument returns the pre-phase output byte-for-byte; a
|
|
bare `` ```bor `` (no path) survives to the fence pass untouched; an
|
|
unparseable locator is put back byte-for-byte (no placeholder, no badge, the
|
|
model's own body — neither improved nor worsened); a fence left OPEN at the
|
|
very end of the input shows the pending state; two fences restore in answer
|
|
order; a locator carrying markup stays inert escaped text; and a standalone
|
|
placeholder is not wrapped in a ``<p>``.
|
|
|
|
Parity half: ``borLocator`` is the documented CLIENT MIRROR of Python
|
|
``app.rag.verbatim.canonical_locator``. The phase's E2E (task 08) pins that
|
|
parity end to end (both sides must agree or hydration misses); this module
|
|
pins the GRAMMAR too, by running both implementations over the same table
|
|
and over a deterministic fuzz — the only way to catch a grammar drift
|
|
before there is a browser in the loop.
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import random
|
|
import re
|
|
import shutil
|
|
import subprocess
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
|
|
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
|
|
MARKDOWN_JS = FRONTEND / "assets" / "markdown.js"
|
|
|
|
HAVE_NODE = shutil.which("node") is not None
|
|
|
|
#: The single truthy path of the phase-182 second argument.
|
|
VERBATIM_GUARD = "opts && opts.verbatim === true"
|
|
|
|
#: The verbatim fence pass — the pinned regex of task 06 item 2.
|
|
VERBATIM_FENCE_RE = r"/^```bor[ \t]+\S[^\n]*\n(?:[\s\S]*?^```[ \t]*$|(?![\s\S]))/gm"
|
|
|
|
#: The pre-existing fence pass, untouched (task 06 item 4).
|
|
ORDINARY_FENCE_RULE = (
|
|
'codeBlocks.push(`<pre><code>${escapeHtml(code.replace(/\\n$/, ""))}</code></pre>`);'
|
|
)
|
|
|
|
#: The untouched pre-phase paragraph guard (the phase-138/180 pin).
|
|
PARAGRAPH_GUARD = r"/^<(h\d|ul|ol|pre|li|blockquote)/"
|
|
|
|
#: The placeholder's block-element clause added by task 06.
|
|
PLACEHOLDER_GUARD = r'/^\u0000VERBATIM\d+\u0000$/'
|
|
|
|
#: The pinned pending copy (``…`` is ONE character, not three dots).
|
|
PENDING_COPY = "Retrieving the exact text\u2026"
|
|
|
|
#: The completion criterion's input.
|
|
CRITERION_INPUT = "```bor homelab/a.md 12-40\nGARBAGE\n```"
|
|
|
|
#: The pre-phase (verbatim-OFF) output of :data:`CRITERION_INPUT` — the
|
|
#: byte-pin of the "inert until opted in" contract. A `` ```bor path `` info
|
|
#: string never matched the ordinary fence rule (``/```(\w*)\n/`` needs the
|
|
#: language token ALONE on that line), so OFF it is escaped prose.
|
|
CRITERION_OFF_OUTPUT = "<p>```bor homelab/a.md 12-40<br>GARBAGE<br>```</p>"
|
|
|
|
#: The option task 07 will pass (never passed by any call site yet).
|
|
ON: dict[str, bool] = {"verbatim": True}
|
|
|
|
#: The inputs the node cases render, keyed by case name.
|
|
CASE_INPUT: dict[str, str] = {
|
|
"criterion": CRITERION_INPUT,
|
|
"bare_bor": "```bor\nfoo\n```",
|
|
"open_bare": "```bor\n",
|
|
"malformed": "```bor not-a-locator 12-40\nSECRET_BODY\n```",
|
|
"open_fence": "```bor homelab/a.md 12-40\n",
|
|
"two_fences": "```bor a/b 1-2\nBODY_ONE\n```\n\n```bor c/d\nBODY_TWO\n```",
|
|
"en_dash": "```bor a/b 12\u201340\nBODY\n```",
|
|
"l_prefix": "```bor a/b#L12-L40\nBODY\n```",
|
|
"xss": "```bor a\"/><script>alert(1)</script>/b.md 1-2\nBODY\n```",
|
|
"standalone": "prose\n\n```bor homelab/a.md\nBODY\n```\n\nafter",
|
|
"attached": "prose\n```bor homelab/a.md 1-2\nBODY\n```",
|
|
"nested": "```\ntext\n```bor a/b 1-2\nBODY\n```\n",
|
|
"no_bor": "# Title\n\nsome **text** and `code`\n\n| a |\n|---|\n| 1 |\n\n> quote\n",
|
|
}
|
|
|
|
|
|
def _text(path: Path) -> str:
|
|
return path.read_text(encoding="utf-8")
|
|
|
|
|
|
def _run_node(script: str) -> str:
|
|
proc = subprocess.run(["node", "-e", script], capture_output=True, text=True, timeout=60)
|
|
assert proc.returncode == 0, f"node failed: {proc.stderr}"
|
|
return proc.stdout
|
|
|
|
|
|
def _render(cases: dict[str, dict[str, bool] | None]) -> dict[str, str]:
|
|
"""Run every case through the REAL renderer under node.
|
|
|
|
``None`` means the ONE-argument call (every pre-phase call site); a dict
|
|
is passed as the second argument as-is.
|
|
"""
|
|
parts = []
|
|
for name, opts in cases.items():
|
|
arg = "" if opts is None else f", {json.dumps(opts)}"
|
|
parts.append(f"out.{name} = renderMarkdown({json.dumps(CASE_INPUT[name])}{arg}); ")
|
|
driver = _text(MARKDOWN_JS) + "\nconst out = {}; " + "".join(parts)
|
|
driver += "console.log(JSON.stringify(out));"
|
|
results: dict[str, str] = json.loads(_run_node(driver))
|
|
return results
|
|
|
|
|
|
def _bor_locator_source() -> str:
|
|
js = _text(MARKDOWN_JS)
|
|
start = js.index("function borLocator(raw) {")
|
|
end = js.index("\nfunction renderMarkdown", start)
|
|
return js[start:end]
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# the signature + the guard (task 06 item 1)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_render_markdown_takes_the_optional_opts_argument() -> None:
|
|
"""``renderMarkdown(md)`` became ``renderMarkdown(md, opts)`` — a SECOND,
|
|
OPTIONAL argument, so every existing call site (the document viewer, the
|
|
thinking blocks, the user bubbles) keeps working untouched — and the
|
|
ONLY truthy path is ``opts.verbatim === true`` (a strict ``===``, so
|
|
``{verbatim: 1}`` / ``{verbatim: "yes"}`` stay OFF)."""
|
|
js = _text(MARKDOWN_JS)
|
|
assert "function renderMarkdown(md, opts) {" in js, (
|
|
"renderMarkdown must gain the second, optional `opts` argument"
|
|
)
|
|
assert "function renderMarkdown(md) {" not in js, "the one-argument signature is gone"
|
|
body = js[js.index("function renderMarkdown(md, opts) {") :]
|
|
assert body.count(VERBATIM_GUARD) == 1, (
|
|
f"the pass must be guarded by exactly one `{VERBATIM_GUARD}` test"
|
|
)
|
|
assert "if (opts && " in js, "a missing opts object must be handled (no TypeError)"
|
|
|
|
|
|
def test_header_comment_records_the_phase_182_pass() -> None:
|
|
"""The header comment carries the three things the next reader needs: the
|
|
pass exists and is opt-in, WHY the model's body is discarded (locked A7),
|
|
and the byte-identical-when-off contract."""
|
|
js = _text(MARKDOWN_JS)
|
|
header = js[: js.find("function escapeHtml")]
|
|
flat = " ".join(
|
|
" ".join(line.strip().lstrip("*").strip() for line in header.splitlines()).split()
|
|
)
|
|
for needle, label in [
|
|
("Phase 182", "the phase reference"),
|
|
("opts && opts.verbatim === true", "the single truthy path"),
|
|
("BODY IS DISCARDED", "why the model's bytes are dropped"),
|
|
("locked A7", "the locked decision behind it"),
|
|
("canonical_locator", "the Python function this mirrors"),
|
|
("BYTE-IDENTICAL WHEN OFF", "the inert-when-off contract"),
|
|
("verbatim \u2192 fences \u2192 tables", "the new pass order"),
|
|
("STRICTER", "the deliberate asymmetry with the server's fences_in"),
|
|
("model's own body", "the malformed-locator degradation"),
|
|
]:
|
|
assert needle in flat, f"the header comment must record {label}: {needle!r}"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# the pass + the placeholder (task 06 item 2)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_verbatim_pass_runs_before_the_fence_pass() -> None:
|
|
"""Order is verbatim → fences → tables → escape+inline → blockquotes →
|
|
paragraphs → restore. The verbatim regex MUST sit ahead of the ordinary
|
|
fence replace, otherwise a locator fence is eaten as a code block first
|
|
and the placeholder never exists."""
|
|
js = _text(MARKDOWN_JS)
|
|
assert VERBATIM_FENCE_RE in js, f"the pinned fence regex is missing: {VERBATIM_FENCE_RE}"
|
|
fence_pass = js.find(ORDINARY_FENCE_RULE)
|
|
assert fence_pass != -1, "the ordinary fence pass must still be there"
|
|
assert js.find(VERBATIM_FENCE_RE) < fence_pass, (
|
|
"the verbatim pass must run BEFORE the fence pass"
|
|
)
|
|
# …and the rest of the pipeline keeps its order behind it.
|
|
for marker in ("const tables = []", "text = inline(escapeHtml(text));", r"split(/\n{2,}/)"):
|
|
assert fence_pass < js.find(marker), f"the pass order broke at {marker!r}"
|
|
|
|
|
|
def test_placeholder_is_created_and_restored() -> None:
|
|
"""A parsed locator becomes ``\\u0000VERBATIMn\\u0000`` and the FINAL restore
|
|
step (alongside TABLE/CODE) turns it back into the panel — the
|
|
placeholder survives escaping (NUL and digits are not escaped) and never
|
|
reaches the DOM itself."""
|
|
js = _text(MARKDOWN_JS)
|
|
assert "`\\u0000VERBATIM${verbatim.length - 1}\\u0000`" in js, (
|
|
"the placeholder must be the indexed VERBATIM marker"
|
|
)
|
|
restore = r".replace(/\u0000VERBATIM(\d+)\u0000/g"
|
|
assert restore in js, "the restore step must replace the VERBATIM placeholder"
|
|
table = js.find(r".replace(/\u0000TABLE(\d+)\u0000/g")
|
|
code = js.find(r".replace(/\u0000CODE(\d+)\u0000/g")
|
|
assert table < code < js.find(restore), "the restore order is TABLE → CODE → VERBATIM"
|
|
|
|
|
|
def test_pending_template_is_exact() -> None:
|
|
"""The restore emits EXACTLY the pinned pending template: the
|
|
``verbatim verbatim-pending`` panel, the canonical locator in
|
|
``data-bor-locator`` (task 07's hydration key), the clipped
|
|
``verbatim-caption``, and the ``role="status"`` state line carrying the
|
|
pinned copy — every interpolated value through ``escapeHtml``."""
|
|
js = _text(MARKDOWN_JS)
|
|
for marker in (
|
|
'<div class="verbatim verbatim-pending" data-bor-locator="${escapeHtml(canonical)}">',
|
|
'<span class="verbatim-caption">${escapeHtml(identity)}</span>',
|
|
f'<span class="verbatim-state" role="status">{PENDING_COPY}</span>',
|
|
):
|
|
assert marker in js, f"the pending template must contain {marker!r}"
|
|
assert 'data-bor-locator="${canonical}"' not in js, (
|
|
"the locator is model-written text landing in an ATTRIBUTE — it must "
|
|
"go through escapeHtml (the escape-first boundary)"
|
|
)
|
|
|
|
|
|
def test_malformed_locator_is_put_back_byte_for_byte() -> None:
|
|
"""``borLocator`` returning ``""`` returns the ORIGINAL match, so a
|
|
malformed locator degrades to whatever the fence pass does with this text
|
|
today (no placeholder, no badge, the model's own body) — this phase
|
|
neither improves nor worsens it."""
|
|
js = _text(MARKDOWN_JS)
|
|
assert 'if (canonical === "") return match;' in js, (
|
|
"an unparseable locator must be returned to the text unchanged"
|
|
)
|
|
|
|
|
|
def test_paragraph_guard_blocks_placeholder_and_untouched_guard_stays() -> None:
|
|
"""The step-3 guard keeps the pre-phase test verbatim (the phase-138/180
|
|
pin) and adds the placeholder clause, so a standalone panel restores as a
|
|
``<div>`` instead of landing inside a ``<p>``."""
|
|
js = _text(MARKDOWN_JS)
|
|
assert PARAGRAPH_GUARD in js, "the pre-phase paragraph guard must stay"
|
|
assert js.count(PARAGRAPH_GUARD) == 1, "there is exactly one paragraph guard"
|
|
assert PLACEHOLDER_GUARD in js, "the placeholder must be a block element too"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# borLocator — the client mirror (task 06 item 3)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_bor_locator_exists_and_handles_every_grammar_token() -> None:
|
|
"""``borLocator`` exists, is documented as the CLIENT MIRROR of
|
|
``app.rag.verbatim.canonical_locator``, and its body carries every
|
|
separator (`` ` ``, ``:``, ``#``, the ``L`` prefix) and range shape
|
|
(``12-40``, the en dash, ``L12-L40``, a bare ``12``) of the D7 grammar —
|
|
plus the refusals (non-numeric, ``< 1``, ``start > end``, trailing
|
|
junk)."""
|
|
body = _bor_locator_source()
|
|
for token, label in [
|
|
(r"[ \t:#]+", "the ` `/`:`/`#` separator class"),
|
|
(r"L?\d+", "the optional `L` line prefix"),
|
|
(r"[-\u2013]", "the hyphen AND the en dash range separator"),
|
|
("(?:", "the OPTIONAL range group (a bare `12` and no range at all)"),
|
|
("$/", "the end anchor (trailing junk is not a range)"),
|
|
('includes("/")', "the first-`/` identity requirement"),
|
|
("start < 1", "the `< 1` refusal"),
|
|
("start > end", "the `start > end` refusal"),
|
|
('replace(/^[Ll]+/, "")', "the `L` prefix strip on each bound"),
|
|
]:
|
|
assert token in body, f"borLocator must handle {label}: {token!r}"
|
|
|
|
|
|
def test_bor_locator_is_documented_as_the_python_mirror() -> None:
|
|
"""The docstring above ``borLocator`` names the Python function it
|
|
mirrors and states that parity is load-bearing (a divergence means
|
|
hydration misses), so nobody 'fixes' one side alone."""
|
|
js = _text(MARKDOWN_JS)
|
|
comment = js[: js.index("function borLocator(raw) {")]
|
|
flat = " ".join(
|
|
" ".join(line.strip().lstrip("/").strip() for line in comment.splitlines()).split()
|
|
)
|
|
for needle, label in [
|
|
("canonical_locator", "the Python function"),
|
|
("CLIENT MIRROR", "that this is the mirror, not the source"),
|
|
("PARITY IS LOAD-BEARING", "why both sides must agree"),
|
|
("#12-40", "the canonical range form"),
|
|
]:
|
|
assert needle in flat, f"the docstring must record {label}: {needle!r}"
|
|
|
|
|
|
def test_bor_locator_returns_canonical_digits_or_empty() -> None:
|
|
"""Source shape: the canonical output is ``identity#start-end`` (digits
|
|
only) and every refusal path returns ``""`` — never ``null``/``undefined``
|
|
(the renderer compares against ``""``)."""
|
|
body = _bor_locator_source()
|
|
assert "return `${identity}#${start}-${end}`;" in body
|
|
assert body.count('return "";') >= 6, "every refusal path must return the empty string"
|
|
assert "return null" not in body and "return undefined" not in body, (
|
|
'borLocator\'s not-a-locator value is "", so the renderer\'s '
|
|
'`if (canonical === "")` test cannot miss'
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# ordinary fences are NOT wrapped in markdown (task 06 item 4)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_ordinary_fence_output_string_is_unchanged() -> None:
|
|
"""Task 06 item 4: do NOT wrap ordinary fences in markdown — task 07
|
|
wraps ``<pre>`` nodes at the DOM level, so the document viewer, the
|
|
thinking blocks and every other caller keep byte-identical HTML. The
|
|
fence rule and its ``<pre><code>`` output string are pinned verbatim."""
|
|
js = _text(MARKDOWN_JS)
|
|
assert r"/```(\w*)\n([\s\S]*?)```/g" in js, "the ordinary fence regex is unchanged"
|
|
assert ORDINARY_FENCE_RULE in js, "the ordinary fence output string is unchanged"
|
|
assert 'class="code-block"' not in js, (
|
|
"markdown.js must NOT emit the task-07 wrapper — that is a DOM-level wrap"
|
|
)
|
|
|
|
|
|
#: The answer renderers task 07 opted IN (the whole client switch — the
|
|
#: per-delta write, the ``done``-time re-render, the brain restore).
|
|
OPTED_IN_CALLS: tuple[str, ...] = (
|
|
"renderMarkdown(acc, { verbatim: true })",
|
|
"renderMarkdown(finalText, { verbatim: true })",
|
|
"renderMarkdown(m.text, { verbatim: true })",
|
|
)
|
|
|
|
|
|
def test_only_the_answer_renderers_opted_in() -> None:
|
|
"""Task 07 opted in exactly the THREE answer renderers (the per-delta
|
|
write, the ``done``-time re-render, the brain restore) — and the shared
|
|
page's one brain renderer. NOTHING else: the user-bubble renders, the
|
|
live thinking runs, the legacy thinking restore and the whole document
|
|
viewer stay one-argument calls, because a question and a scratchpad are
|
|
not locators (task 06's byte-identity contract for every other surface).
|
|
The count is the pin: a fourth ``verbatim: true`` in either file is a
|
|
surface the phase never claimed."""
|
|
app = _text(FRONTEND / "assets" / "app.js")
|
|
shared = _text(FRONTEND / "assets" / "shared.js")
|
|
for call in OPTED_IN_CALLS:
|
|
assert call in app, f"app.js must render the answer through {call}"
|
|
assert app.count("verbatim: true") == 3, (
|
|
"exactly the three answer renderers may opt in in app.js"
|
|
)
|
|
assert (
|
|
'addSharedMessage("brain", renderMarkdown(m.text, { verbatim: true }))' in shared
|
|
)
|
|
assert shared.count("verbatim: true") == 1, "the shared page opts in its one renderer"
|
|
# The neighbours stay one-argument, in both files.
|
|
assert 'addMessage("user", renderMarkdown(m.text), true)' in app
|
|
assert 'addSharedMessage("user", renderMarkdown(m.text))' in shared
|
|
for name in ("app.js", "shared.js"):
|
|
js = _text(FRONTEND / "assets" / name)
|
|
for site in re.findall(r"renderMarkdown\([^\n]*\)", js):
|
|
if "thinking" in site or "entry.text" in site:
|
|
assert "verbatim" not in site, f"{name}: {site} must stay one-argument"
|
|
# A document's own ```bor fence is the document's text — never a locator.
|
|
assert "verbatim: true" not in _text(FRONTEND / "assets" / "document.js")
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# behavior (node): the story shapes + the completion criterion
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
|
def test_locator_fence_yields_the_placeholder_and_drops_the_model_bytes() -> None:
|
|
"""THE completion criterion: the criterion input WITH ``{verbatim: true}``
|
|
yields the placeholder DOM with
|
|
``data-bor-locator="homelab/a.md#12-40"`` and NO ``GARBAGE`` anywhere in
|
|
the returned HTML (locked A7). Two fences restore in answer order, the en
|
|
dash and the ``L`` prefix canonicalize, and a standalone panel is not
|
|
wrapped in a ``<p>``."""
|
|
out = _render(
|
|
{
|
|
"criterion": ON,
|
|
"two_fences": ON,
|
|
"en_dash": ON,
|
|
"l_prefix": ON,
|
|
"standalone": ON,
|
|
}
|
|
)
|
|
html = out["criterion"]
|
|
assert 'class="verbatim verbatim-pending"' in html
|
|
assert 'data-bor-locator="homelab/a.md#12-40"' in html
|
|
assert ">homelab/a.md</span>" in html, "the caption carries the source/path"
|
|
assert f">{PENDING_COPY}</span>" in html
|
|
assert 'role="status"' in html
|
|
assert "GARBAGE" not in html, "the model's body must appear NOWHERE (locked A7)"
|
|
assert "\u0000" not in html, "no placeholder byte survives"
|
|
assert "<p>" not in html, "a standalone panel is a block, not a <p>"
|
|
|
|
two = out["two_fences"]
|
|
assert two.index("a/b#1-2") < two.index('data-bor-locator="c/d"'), (
|
|
"two fences keep answer order"
|
|
)
|
|
assert two.count("verbatim-pending") == 2
|
|
assert "BODY_ONE" not in two and "BODY_TWO" not in two
|
|
assert out["en_dash"].count('data-bor-locator="a/b#12-40"') == 1
|
|
assert "BODY" not in out["en_dash"]
|
|
assert out["l_prefix"].count('data-bor-locator="a/b#12-40"') == 1
|
|
assert "BODY" not in out["l_prefix"]
|
|
assert out["standalone"] == (
|
|
'<p>prose</p><div class="verbatim verbatim-pending" '
|
|
'data-bor-locator="homelab/a.md"><span class="verbatim-caption">homelab/a.md</span>'
|
|
f'<span class="verbatim-state" role="status">{PENDING_COPY}</span></div><p>after</p>'
|
|
)
|
|
|
|
|
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
|
def test_off_paths_are_byte_identical_to_the_pre_phase_renderer() -> None:
|
|
"""Every call WITHOUT ``{verbatim: true}`` — no second argument, the
|
|
document viewer's one-argument call, and an explicit
|
|
``{verbatim: false}`` — returns the PRE-PHASE output byte-for-byte: the
|
|
criterion input is escaped prose (a `` ```bor path `` info string never
|
|
matched the ``/```(\\w*)\\n/`` fence rule), and a locator-free document
|
|
renders identically with and without the option."""
|
|
out = _render(
|
|
{
|
|
"criterion": None,
|
|
"malformed": None,
|
|
"two_fences": None,
|
|
"standalone": None,
|
|
"nested": None,
|
|
"no_bor": None,
|
|
}
|
|
)
|
|
assert out["criterion"] == CRITERION_OFF_OUTPUT, "the OFF path must not change a byte"
|
|
for name, html in out.items():
|
|
assert "verbatim" not in html, f"{name}: no placeholder markup with the option OFF"
|
|
# the option object exists but is false → the same bytes as no argument
|
|
on_false = _render({"criterion": {"verbatim": False}})
|
|
assert on_false["criterion"] == CRITERION_OFF_OUTPUT
|
|
truthy_but_strict = _render({"criterion": {}})
|
|
assert truthy_but_strict["criterion"] == CRITERION_OFF_OUTPUT, (
|
|
"the guard is `=== true`: an options object without the flag stays OFF"
|
|
)
|
|
# a locator-free document is untouched by the whole feature
|
|
assert _render({"no_bor": ON})["no_bor"] == out["no_bor"]
|
|
|
|
|
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
|
def test_bare_and_malformed_fences_fall_through_to_the_ordinary_pass() -> None:
|
|
"""A bare `` ```bor `` (no path) is an ordinary code block and survives to
|
|
the fence pass UNTOUCHED even with the option on; a bare `` ```bor ``
|
|
with no body at all still paints its raw marker as prose; and an
|
|
unparseable locator degrades to exactly what the fence pass does with
|
|
this text today — the model's own body, no placeholder, no badge."""
|
|
out = _render({"bare_bor": ON, "open_bare": ON, "malformed": ON})
|
|
assert out["bare_bor"] == "<p><pre><code>foo</code></pre></p>", (
|
|
"a bare ```bor fence must render EXACTLY as it does pre-phase"
|
|
)
|
|
assert out["open_bare"] == "<p>```bor</p>"
|
|
assert out["malformed"] == "<p>```bor not-a-locator 12-40<br>SECRET_BODY<br>```</p>", (
|
|
"a malformed locator is put back byte-for-byte: no placeholder, "
|
|
"no badge, the model's own body"
|
|
)
|
|
|
|
|
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
|
def test_open_fence_at_the_end_shows_the_pending_state() -> None:
|
|
"""The second alternative: a fence left OPEN at the very end of the input
|
|
(the info line typed, the body not yet streamed) shows the pending
|
|
placeholder instead of raw backticks — the deliberate asymmetry with the
|
|
server's stricter ``fences_in`` (the server only ever sees a finished
|
|
answer, the client sees a partial one)."""
|
|
out = _render({"open_fence": ON})
|
|
html = out["open_fence"]
|
|
assert 'data-bor-locator="homelab/a.md#12-40"' in html
|
|
assert "```" not in html, "no raw backticks while a locator fence is open"
|
|
|
|
|
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
|
def test_locator_markup_stays_inert_escaped_text() -> None:
|
|
"""Escape-first holds on the new surface: a locator carrying ``"><script>``
|
|
is HTML-escaped in BOTH the attribute and the caption, so it can never
|
|
become a live element (the renderer's whole reason for existing)."""
|
|
html = _render({"xss": ON})["xss"]
|
|
assert "<script>" not in html
|
|
assert "<script>" in html
|
|
assert '"/><script>' in html
|
|
assert "#1-2" in html, "the markup still canonicalized to a locator"
|
|
|
|
|
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
|
def test_a_bor_line_inside_an_ordinary_fence_agrees_with_the_server() -> None:
|
|
"""Documented limitation shared with ``fences_in`` (both sides scan for
|
|
the opening line, neither supports a nested `` ``` ``): a ``bor`` locator
|
|
line inside an ordinary fence is a LOCATOR on both sides, so hydration
|
|
still matches — it is never silently half-code, half-panel on one side
|
|
only."""
|
|
from app.rag.verbatim import fences_in
|
|
|
|
html = _render({"nested": ON})["nested"]
|
|
assert 'data-bor-locator="a/b#1-2"' in html
|
|
assert fences_in(CASE_INPUT["nested"]) == ["a/b 1-2"], (
|
|
"the server must see the SAME locator in the same text"
|
|
)
|
|
|
|
|
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
|
def test_fence_attached_to_prose_still_loses_its_body() -> None:
|
|
"""A locator fence with NO blank line above it still loses the model's
|
|
body (the browser closes the surrounding ``<p>`` when it parses the
|
|
``<div>``, the same fix-up an ordinary fenced ``<pre>`` already relies
|
|
on) — the body never paints either way."""
|
|
html = _render({"attached": ON})["attached"]
|
|
assert 'data-bor-locator="homelab/a.md#1-2"' in html
|
|
assert "BODY" not in html
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# parity with the server's canonical_locator (task 06 item 3)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
#: The D7 grammar table: input → the canonical string BOTH sides must
|
|
#: produce, or ``""`` for "not a locator at all".
|
|
PARITY_TABLE: dict[str, str] = {
|
|
"homelab/a.md": "homelab/a.md",
|
|
"homelab/a.md 12-40": "homelab/a.md#12-40",
|
|
"homelab/a.md:12-40": "homelab/a.md#12-40",
|
|
"homelab/a.md#12-40": "homelab/a.md#12-40",
|
|
"homelab/a.md#L12-L40": "homelab/a.md#12-40",
|
|
"homelab/a.md 12\u201340": "homelab/a.md#12-40",
|
|
"homelab/a.md 12": "homelab/a.md#12-12",
|
|
"homelab/a.md 12-40": "homelab/a.md#12-40",
|
|
"homelab/a.md 12 - 40": "homelab/a.md#12-40",
|
|
" homelab/a.md ": "homelab/a.md",
|
|
"homelab/deep/dir/a.md#L3": "homelab/deep/dir/a.md#3-3",
|
|
"homelab/a.md#l3-l7": "homelab/a.md#3-7",
|
|
"homelab/a.md L7": "homelab/a.md#7-7",
|
|
# the en dash is a RANGE separator, never the identity/range separator,
|
|
# so this row is ONE whole-document identity whose path contains one:
|
|
"homelab/a.md\u201312-40": "homelab/a.md\u201312-40",
|
|
# refusals — not locators at all:
|
|
"a.md": "",
|
|
"": "",
|
|
" ": "",
|
|
"homelab/a.md 0-5": "",
|
|
"homelab/a.md 40-12": "",
|
|
"homelab/a.md x-y": "",
|
|
"homelab/a.md 12-40 extra": "",
|
|
"bor homelab/a.md": "",
|
|
"homelab/a.md#": "",
|
|
"homelab/a.md:": "",
|
|
"homelab/a.md-": "",
|
|
"homelab/my note.md 12": "",
|
|
}
|
|
|
|
|
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
|
def test_bor_locator_matches_the_python_canonical_locator() -> None:
|
|
"""The mirror is exact: for every row of the grammar table the JS
|
|
``borLocator`` and the Python ``canonical_locator`` agree — and the table
|
|
itself is checked against the D7 spec, so a SIMULTANEOUS drift on both
|
|
sides is caught too. A divergence here means hydration misses: the panel
|
|
would sit unresolved forever."""
|
|
from app.rag.verbatim import canonical_locator
|
|
|
|
script = (
|
|
_bor_locator_source()
|
|
+ "\nconst t = "
|
|
+ json.dumps(PARITY_TABLE)
|
|
+ ";\nconst out = {};\nfor (const k of Object.keys(t)) out[k] = borLocator(k);\n"
|
|
"console.log(JSON.stringify(out));\n"
|
|
)
|
|
got: dict[str, str] = json.loads(_run_node(script))
|
|
for raw, expected in PARITY_TABLE.items():
|
|
assert got[raw] == expected, f"borLocator({raw!r}) != {expected!r} (got {got[raw]!r})"
|
|
python = canonical_locator(raw) or ""
|
|
assert python == expected, (
|
|
f"canonical_locator({raw!r}) = {python!r} disagrees with the D7 "
|
|
f"table value {expected!r}"
|
|
)
|
|
|
|
|
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
|
def test_fuzz_locator_parity() -> None:
|
|
"""A deterministic fuzz over the grammar's alphabet: the two
|
|
implementations must agree on every string, including the shapes nobody
|
|
hand-wrote (dangling separators, multi-bound ranges, lone ``L``s)."""
|
|
from app.rag.verbatim import canonical_locator
|
|
|
|
rng = random.Random(182)
|
|
alphabet = [
|
|
"a", "b", "/", ".", ":", "#", " ", "L", "l", "-", "\u2013", "1", "2", "9", "x", "\t",
|
|
]
|
|
samples = [
|
|
"".join(rng.choice(alphabet) for _ in range(rng.randint(0, 9))) for _ in range(400)
|
|
]
|
|
script = (
|
|
_bor_locator_source()
|
|
+ "\nconst t = "
|
|
+ json.dumps(samples)
|
|
+ ";\nconst out = [];\nfor (const k of t) out.push(borLocator(k));\n"
|
|
"console.log(JSON.stringify(out));\n"
|
|
)
|
|
got: list[str] = json.loads(_run_node(script))
|
|
for raw, js_value in zip(samples, got, strict=True):
|
|
python = canonical_locator(raw) or ""
|
|
assert js_value == python, f"parity break on {raw!r}: js={js_value!r} py={python!r}"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# no regression on the neighboring passes (the phase's byte-identity mandate)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def test_adjacent_renderer_fixtures_stay_pinned_where_they_live() -> None:
|
|
"""The phase-44 table, phase-138 blockquote and phase-180 image fixtures
|
|
are the byte-identity evidence for the passes this task sits next to —
|
|
they live in their own suites, so this pins that those suites still pin
|
|
them (a deleted fixture here would silently drop the guarantee)."""
|
|
for name, needle in [
|
|
("test_markdown_tables.py", "md-table-wrap"),
|
|
("test_markdown_blockquotes.py", '<blockquote class="md-quote">'),
|
|
("test_markdown_inline_links_images.py", 'class="md-img"'),
|
|
]:
|
|
text = _text(Path(__file__).resolve().parent / name)
|
|
assert needle in text, f"{name} must still pin {needle!r}"
|