phase: 181_svg_document_preview
**Phase 181 — final verification pass (all tasks already complete; verified + docs pass)**
- Verified: `app/rag/svg_text.py` (100 % cov), vector branch in `_index_image_file`, `_row_kind_mismatch` text→image migration, `IMAGE_MIMES[".svg"]`, sandboxed CSP on `GET /api/documents/{id}/image`, `.doc-image-img--vector` in `document.js`/`styles.css`, figure/thumbnail deliberate non-changes.
- Tests/lint (exact commands): `uv run pytest --cov=app --cov-report=term-missing` → **4265 passed, TOTAL 99 %** (exit 0) · `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` → **5 passed in isolation** · `uv run ruff check .` → clean · `uv run pyright` → **0 errors**.
- Regression (each run in isolation, `--no-cov`, all rc=0): `test_image_documents`, `test_chat_image_questions`, `test_answer_images`, `test_chat_answer_clusters`, `test_image_anchor_context`, `test_image_upload_feedback`, `test_markdown_inline_links_images`, `test_prose_word_wrap`, `test_pdf_support`, `test_pictorial_pdf`, `test_garbage_pdf`, `test_pdf_image_extraction`, `test_pdf_image_context`, `test_pdf_pages_in_sources`, `test_pdf_strip_additive_selfheal`, `test_sync_image_progress`.
- Completion criteria: ①SVG→`is_image`+`image_path`, no vision call, own `<title>/<desc>/<text>` — **met**; ②`image/svg+xml` + `default-src 'none'; sandbox` + `private, max-age=3600`, raster headers byte-unchanged — **met**; ③chat figure → modal `<img>`, no XML — **met**; ④`/document.html` full-width vector in `.doc-image` + text below — **met**; ⑤`` paints — **met**; ⑥raster/PDF unchanged — **met**; ⑦all gates green — **met**; ⑧commit + phase-dir move — harness-owned.
- Documentation: found and fixed README inconsistencies — stale forward reference "left to phase 181 / is phase 181's job" → completed-tense sizing note; "Updating the documents" standalone-image list missing `svg`; new **Raster only** bullet for question uploads (`.svg` → 422) documenting the `chat_images` guard; missing `svg_indexed=N` / `import: svg indexed` log contract; env-table heading `122/123/124` → `122/123/124/181`; new troubleshooting entry for an `.svg` showing raw XML. Full audit: every `Settings` field and every `.env.example` var is documented; pages/commands/scripts all exist. AGENTS.md verified accurate (phase-181 §5 clause present) — no changes needed beyond what the phase wrote.
- Notable deviation (kept from task 02): `app/api/chat_images.py` subtracts `_VECTOR_IMAGE_EXTENSIONS` from the question-image accept set — a required consequence of LOCKED A7 (`svg` in the default image set) since `_CHAT_IMAGE_FILENAME_RE` refuses to *serve* `.svg`; the regex itself is untouched, as the phase specified.
- Note for the commit: changes include `README.md`, `AGENTS.md` and `.env.example` — the phase file's `git add app/ frontend/ tests/ .env.example .agents/phases/` line omits the two markdown files; stage them too.
- Next pending phase: `182_verbatim_show_text`.
This commit is contained in:
@@ -68,6 +68,7 @@ from pydantic import ValidationError
|
||||
|
||||
from app.api import chat as chat_api
|
||||
from app.api import chat_images
|
||||
from app.api.chat_images import _VECTOR_IMAGE_EXTENSIONS
|
||||
from app.config import Settings
|
||||
from app.main import app as fastapi_app
|
||||
from app.models import Document
|
||||
@@ -354,6 +355,32 @@ def test_upload_rejects_a_non_image_extension_with_the_fixed_detail(
|
||||
assert not image_dir.exists() # the dir is created on the accepted path only
|
||||
|
||||
|
||||
def test_upload_rejects_a_vector_svg_even_though_it_indexes(
|
||||
client: TestClient, image_dir: Path
|
||||
) -> None:
|
||||
"""Phase 181: ``svg`` joins the DEFAULT image set for the DOCUMENT
|
||||
walk, and question attachments stay RASTER-only — the upload 422s
|
||||
with the six-extension detail and writes nothing.
|
||||
|
||||
Without the subtraction this would be a silent trap: the stored
|
||||
``<uuid>.svg`` is refused by the serve route's filename regex (a
|
||||
broken figure in the user's own bubble) and an OpenAI-compatible
|
||||
vision endpoint rejects ``image/svg+xml``, so the turn could never
|
||||
be answered. The vector family belongs to INDEXING, not to
|
||||
attachments."""
|
||||
assert ".svg" in Settings(_env_file=None).image_extension_set # pyright: ignore[reportCallIssue]
|
||||
assert frozenset({".svg"}) == _VECTOR_IMAGE_EXTENSIONS
|
||||
r = _upload(
|
||||
client, "diagram.svg", b'<svg xmlns="http://www.w3.org/2000/svg"></svg>',
|
||||
mime="image/svg+xml",
|
||||
)
|
||||
assert r.status_code == 422
|
||||
assert r.json()["detail"] == (
|
||||
"only .bmp, .gif, .jpeg, .jpg, .png, .webp images are accepted"
|
||||
)
|
||||
assert not image_dir.exists()
|
||||
|
||||
|
||||
def test_upload_rejects_a_name_without_an_extension(
|
||||
client: TestClient, image_dir: Path
|
||||
) -> None:
|
||||
@@ -916,7 +943,10 @@ def test_the_attach_reveal_is_gated_on_the_config_flag() -> None:
|
||||
def test_the_client_precheck_matches_the_servers_six_extensions() -> None:
|
||||
"""ONE list of the six, on both sides: the client pre-check (the
|
||||
file NAME's extension — the accept attribute is advisory) and the
|
||||
server's phase-122 frozenset (the upload endpoint's authority).
|
||||
server's phase-122 frozenset MINUS the vector members (the upload
|
||||
endpoint's authority — phase 181: question images stay RASTER-only,
|
||||
a vision endpoint rejects ``image/svg+xml`` and the serve route's
|
||||
filename regex would never serve one).
|
||||
The client list must stay the code default's six — a drift here
|
||||
would either block a valid pick or waste a round-trip on a 422."""
|
||||
js = _app_js()
|
||||
@@ -931,12 +961,14 @@ def test_the_client_precheck_matches_the_servers_six_extensions() -> None:
|
||||
"png",
|
||||
"webp",
|
||||
}
|
||||
# and it equals the server's default set (dots stripped — the
|
||||
# client list is dotless, the server's is dotted).
|
||||
server_set = {
|
||||
e.lstrip(".")
|
||||
for e in Settings(_env_file=None).image_extension_set # pyright: ignore[reportCallIssue]
|
||||
}
|
||||
# and it equals the server's default raster set (dots stripped — the
|
||||
# client list is dotless, the server's is dotted). ``svg`` is in the
|
||||
# DEFAULT image set for the document walk (phase 181) and
|
||||
# deliberately NOT attachable here (see
|
||||
# ``chat_images._VECTOR_IMAGE_EXTENSIONS``).
|
||||
raw_default = Settings(_env_file=None).image_extension_set # pyright: ignore[reportCallIssue]
|
||||
assert ".svg" in raw_default # the vector member (phase 181)
|
||||
server_set = {e.lstrip(".") for e in raw_default - _VECTOR_IMAGE_EXTENSIONS}
|
||||
assert client_set == server_set
|
||||
# the file input's accept attribute names the same six (mime form).
|
||||
m = re.search(r'<input type="file" id="attach-file"\s+accept="([^"]+)"', _index_html())
|
||||
|
||||
@@ -94,6 +94,20 @@ format table as node behavioral runs over the real module, and the
|
||||
``.doc-prose`` CSS (the 72rem reading column + ``pre-wrap`` +
|
||||
``overflow-wrap: anywhere``, tokens only) beside the ``.doc-raw``
|
||||
contract that stays locked (phase 116 A1 / phase 165 A3).
|
||||
|
||||
Phase 181 (task 04) — the SVG vector preview: ``docImageBlock`` adds
|
||||
the ``doc-image-img--vector`` modifier ONLY when
|
||||
``doc.format === "svg"`` (the base class is always kept) — a
|
||||
``viewBox``-only SVG has NO intrinsic size, so the default 300×150
|
||||
replaced-element box would sit small inside the full-width
|
||||
``.doc-image`` card — pinned as a node behavioral run of the real
|
||||
``docImageBlock`` (svg → the compound class, ``src`` from
|
||||
``image_url`` never inlined, ``alt`` = summary with the title
|
||||
fallback; png → the class string byte-identical to the base class)
|
||||
plus the ``.doc-image-img--vector`` CSS pins (``width: 100%`` +
|
||||
``height: auto`` + ``object-fit: contain`` under the base rule's
|
||||
70vh clamp, zero colour literals, the base ``.doc-image-img`` rule
|
||||
untouched).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -2102,3 +2116,180 @@ def test_doc_prose_css_wraps_in_the_reading_column() -> None:
|
||||
raw_rule = css[raw_i : css.find("}", raw_i)]
|
||||
assert "white-space: pre" in raw_rule
|
||||
assert "overflow-x: auto" in raw_rule
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Phase 181 (task 04) — the SVG vector preview: docImageBlock's format
|
||||
# gate (base class + the doc-image-img--vector modifier, LOCKED A3:
|
||||
# bytes only ever through <img>) and the .doc-image-img--vector CSS
|
||||
# (a viewBox-only SVG has no intrinsic size — width: 100% spans the
|
||||
# .doc-image card, object-fit: contain survives the 70vh clamp,
|
||||
# palette-neutral). The chat figure (app.js) and the Sources
|
||||
# thumbnail (sources.js) are VERIFIED non-changes (design D7): both
|
||||
# key off image_url alone and their boxes pin the img
|
||||
# (.source-image-img 96px cap, .kb-doc-thumb-img fixed 48px box).
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_doc_image_block_gates_the_vector_modifier_on_format() -> None:
|
||||
"""Source pins (node-free): docImageBlock selects the compound
|
||||
class on ``doc.format === "svg"`` and the bare base class
|
||||
otherwise (every current .doc-image-img rule keeps applying);
|
||||
the bytes stay an <img> src property assignment — never inlined
|
||||
(LOCKED A3), never innerHTML. The comment records the rationale
|
||||
(no-intrinsic-size / 300×150) and the legacy-TEXT honesty
|
||||
(XML-in-.doc-raw is the truth; no content sniffing)."""
|
||||
js = _read(DOCUMENT_JS)
|
||||
fn = _fn(js, "docImageBlock")
|
||||
assert 'doc.format === "svg"' in fn
|
||||
assert '"doc-image-img doc-image-img--vector"' in fn
|
||||
assert ': "doc-image-img";' in fn, "the raster path keeps the bare base class"
|
||||
assert "img.src = doc.image_url;" in fn, "the bytes URL stays a property assignment"
|
||||
assert "innerHTML" not in fn
|
||||
# The rationale + honesty comments (house style: why, not what).
|
||||
for citation in ("Phase 181", "viewBox", "300\u00d7150", "LOCKED A3", ".doc-raw"):
|
||||
assert citation in fn, f"the comment must cite {citation!r}"
|
||||
|
||||
|
||||
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||
def test_doc_image_block_svg_yields_the_vector_class_png_unchanged() -> None:
|
||||
"""Behavioral (node, the house fake DOM): the REAL docImageBlock —
|
||||
an SVG doc renders class="doc-image-img doc-image-img--vector"
|
||||
with src = image_url and alt = the summary (title fallback when
|
||||
the summary is null/blank); a PNG doc's class string is the base
|
||||
class ONLY (byte-identical to pre-phase — the phase-122/133
|
||||
raster surfaces cannot drift); the wrapper stays .doc-image."""
|
||||
js = _read(DOCUMENT_JS)
|
||||
# The docs ride in via json.dumps (the summary carries quotes and
|
||||
# an em-dash — the alt contract must survive verbatim).
|
||||
cases = {
|
||||
"svg": {
|
||||
"format": "svg",
|
||||
"image_url": "/api/documents/7/image",
|
||||
"summary": 'Vector diagram "topo" \u2014 3 shapes',
|
||||
"title": "topo",
|
||||
},
|
||||
"svgNoSummary": {
|
||||
"format": "svg",
|
||||
"image_url": "/api/documents/9/image",
|
||||
"summary": None,
|
||||
"title": "rack",
|
||||
},
|
||||
"svgBlankSummary": {
|
||||
"format": "svg",
|
||||
"image_url": "/api/documents/10/image",
|
||||
"summary": " ",
|
||||
"title": "blank",
|
||||
},
|
||||
"png": {
|
||||
"format": "png",
|
||||
"image_url": "/api/documents/8/image",
|
||||
"summary": "A photo",
|
||||
"title": "t",
|
||||
},
|
||||
}
|
||||
script = (
|
||||
_CONTENT_BRANCH_FAKE_DOM
|
||||
+ _fn(js, "docImageBlock")
|
||||
+ "\nconst cases = "
|
||||
+ json.dumps(cases)
|
||||
+ """
|
||||
;
|
||||
function imageFor(doc) {
|
||||
const wrap = docImageBlock(doc);
|
||||
const img = wrap.children[0];
|
||||
return {
|
||||
wrapClass: wrap.className, imgClass: img.className, src: img.src, alt: img.alt,
|
||||
};
|
||||
}
|
||||
const out = {};
|
||||
for (const [k, doc] of Object.entries(cases)) out[k] = imageFor(doc);
|
||||
console.log(JSON.stringify(out));
|
||||
"""
|
||||
)
|
||||
out = json.loads(_run_node(script))
|
||||
assert out["svg"] == {
|
||||
"wrapClass": "doc-image",
|
||||
"imgClass": "doc-image-img doc-image-img--vector",
|
||||
"src": "/api/documents/7/image",
|
||||
"alt": 'Vector diagram "topo" \u2014 3 shapes',
|
||||
}
|
||||
# alt falls back to the title (the WCAG alt contract, unchanged).
|
||||
assert out["svgNoSummary"]["alt"] == "rack"
|
||||
assert out["svgBlankSummary"]["alt"] == "blank"
|
||||
assert out["svgBlankSummary"]["imgClass"] == "doc-image-img doc-image-img--vector"
|
||||
# The raster class string is byte-identical to pre-phase (base only).
|
||||
assert out["png"]["imgClass"] == "doc-image-img"
|
||||
assert out["png"]["src"] == "/api/documents/8/image"
|
||||
|
||||
|
||||
def test_doc_image_vector_css_spans_the_card_without_squashing() -> None:
|
||||
"""styles.css: ``.doc-image-img--vector`` (immediately after
|
||||
``.doc-image-img``) is ``width: 100%`` + ``height: auto`` +
|
||||
``object-fit: contain`` — a viewBox-only SVG (no intrinsic size:
|
||||
the default replaced-element box is 300\u00d7150) spans the
|
||||
full-width .doc-image card, and contain letterboxes instead of
|
||||
squashing when the base rule's max-height: 70vh clamps a tall
|
||||
drawing. Zero colour literals (palette-neutral — the monochrome
|
||||
invariant); the base .doc-image-img rule is UNTOUCHED (the
|
||||
phase-122 raster contract)."""
|
||||
css = _read(STYLES_CSS)
|
||||
base_i = css.find(".doc-image-img {")
|
||||
vec_i = css.find(".doc-image-img--vector {")
|
||||
assert -1 < base_i < vec_i, ".doc-image-img--vector must sit next to the base rule"
|
||||
# Provenance comment directly above the rule (house style).
|
||||
comment_start = css.rfind("/*", 0, vec_i)
|
||||
provenance = css[comment_start:vec_i]
|
||||
assert -1 < comment_start < vec_i
|
||||
for citation in ("phase 181", "viewBox", "300\u00d7150", "object-fit", "doc.format"):
|
||||
assert citation in provenance, f"the comment must cite {citation!r}"
|
||||
assert "no animation" in provenance
|
||||
rule = css[vec_i : css.find("}", vec_i)]
|
||||
for decl in ("width: 100%", "height: auto", "object-fit: contain"):
|
||||
assert decl in rule, f".doc-image-img--vector must carry {decl!r}"
|
||||
# Palette-neutral — zero colour literals (no new hue).
|
||||
assert not re.search(r"#[0-9a-fA-F]{3,8}\b", rule)
|
||||
assert "rgb(" not in rule and "hsl(" not in rule and "color-mix(" not in rule
|
||||
# The untouched base rule (the raster + onerror contract).
|
||||
base_rule = css[base_i : css.find("}", base_i)]
|
||||
for decl in ("max-width: 100%", "max-height: 70vh", "height: auto"):
|
||||
assert decl in base_rule, f"the base rule must keep {decl!r}"
|
||||
|
||||
|
||||
def test_figure_and_thumbnail_key_off_image_url_alone() -> None:
|
||||
"""Design D7's verified non-changes: the chat figure
|
||||
(appendSourceImageFigure) and the Sources KB-tree thumbnail
|
||||
(docThumb) resolve their bytes from ``image_url`` alone — no
|
||||
format field on the wire, none added — and their CSS boxes pin
|
||||
the img (.source-image-img: 96px cap + object-fit: contain; the
|
||||
.kb-doc-thumb-img: the fixed 48px box), so a size-less SVG
|
||||
scales inside each cap exactly like a raster. Pins: neither
|
||||
function sniffs a format at all (the deliberate non-change,
|
||||
design D7), the phase-181 comments sit where they were made,
|
||||
and the two rules keep their box declarations."""
|
||||
app_js = _read(APP_JS)
|
||||
assert app_js.count("function appendSourceImageFigure(") == 1
|
||||
fig = _fn(app_js, "appendSourceImageFigure")
|
||||
assert "img.src = s.image_url;" in fig
|
||||
assert "format" not in fig, "no format sniffing on the chat figure (D7)"
|
||||
fn_start = app_js.index("function appendSourceImageFigure(")
|
||||
assert "phase 181" in app_js[max(0, fn_start - 2400) : fn_start].lower(), (
|
||||
"the deliberate-non-change comment sits above the figure"
|
||||
)
|
||||
sources_js = _read(SOURCES_JS)
|
||||
thumb_start = sources_js.index("function docThumb(d)")
|
||||
thumb_end = sources_js.index("\n }", thumb_start)
|
||||
thumb = sources_js[thumb_start:thumb_end]
|
||||
assert "img.src = d.image_url;" in thumb
|
||||
assert "format" not in thumb, "no format sniffing on the thumbnail (D7)"
|
||||
branch_i = sources_js.index("if (d.is_image) {")
|
||||
assert "phase 181" in sources_js[branch_i - 900 : branch_i].lower(), (
|
||||
"the verified-non-change comment sits above the is_image branch"
|
||||
)
|
||||
css = _read(STYLES_CSS)
|
||||
cap_i = css.find(".source-image-img {")
|
||||
cap = css[cap_i : css.find("}", cap_i)]
|
||||
assert "max-height: 96px" in cap and "object-fit: contain" in cap
|
||||
box_i = css.find(".kb-doc-thumb-img {")
|
||||
box = css[box_i : css.find("}", box_i)]
|
||||
assert "width: 100%" in box and "height: 100%" in box
|
||||
|
||||
@@ -106,19 +106,25 @@ def _settings(**kwargs: Any) -> Settings:
|
||||
|
||||
|
||||
def test_image_defaults_match_locked_a3() -> None:
|
||||
"""LOCKED A3: image support is OFF by default (enable only when the
|
||||
chat model supports vision), the built-in extension set is the six
|
||||
common formats, and the image bytes' persistent home is the
|
||||
"""LOCKED A3 + phase 181 LOCKED A7: image support is OFF by default
|
||||
(enable only when the chat model supports vision), the built-in
|
||||
extension set is the six common raster formats PLUS ``svg`` (the one
|
||||
VECTOR member — A1: it indexes from its own extracted text, never a
|
||||
vision call), and the image bytes' persistent home is the
|
||||
``~/bor-sources/images`` raw string (``expanduser`` is the
|
||||
importer's job — the ``sources_dir``/``upload_dir`` convention)."""
|
||||
importer's job — the ``sources_dir``/``upload_dir`` convention).
|
||||
``images`` stays the SINGLE knob (A7): with it off an ``.svg`` is a
|
||||
plain text document, no special-casing."""
|
||||
s = _settings()
|
||||
assert s.images is False
|
||||
# Dotted, lowercased form — the shape the walk's matcher consumes
|
||||
# (the ``import_extension_set`` precedent).
|
||||
assert s.image_extension_set == {".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp"}
|
||||
assert s.image_extension_set == {
|
||||
".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp", ".svg",
|
||||
}
|
||||
assert s.image_dir == "~/bor-sources/images"
|
||||
# The raw string is stored untouched (no expanduser here).
|
||||
assert s.image_extensions == "png,jpg,jpeg,webp,gif,bmp"
|
||||
assert s.image_extensions == "png,jpg,jpeg,webp,gif,bmp,svg"
|
||||
|
||||
|
||||
def test_image_set_is_separate_from_the_import_set() -> None:
|
||||
@@ -127,7 +133,9 @@ def test_image_set_is_separate_from_the_import_set() -> None:
|
||||
the import set leaves the image set untouched, and vice versa."""
|
||||
s = _settings(import_extensions="md")
|
||||
assert s.import_extension_set == {".md"}
|
||||
assert s.image_extension_set == {".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp"}
|
||||
assert s.image_extension_set == {
|
||||
".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp", ".svg",
|
||||
}
|
||||
s2 = _settings(image_extensions="md")
|
||||
assert s2.image_extension_set == {".md"}
|
||||
assert ".md" in s2.import_extension_set # the default import set is intact
|
||||
@@ -762,7 +770,16 @@ def test_image_mimes_cover_the_default_family() -> None:
|
||||
default image family — dotted lowercase keys, the
|
||||
``image_extension_set`` shape — and the unlisted-token fallback is
|
||||
the generic mime (a custom ``BOR_IMAGE_EXTENSIONS`` format fails
|
||||
soft at the vision endpoint, never a guessed type)."""
|
||||
soft at the vision endpoint, never a guessed type).
|
||||
|
||||
The map is pinned by IDENTITY (not membership), so a silent removal
|
||||
or re-pointing of any entry fails here. Phase 181 (task 03) adds
|
||||
``.svg`` as the ONE vector entry; the six raster entries are
|
||||
untouched by it, and the vector import branch never reaches
|
||||
:func:`image_data_url` (LOCKED A1: no vision call for an SVG), so
|
||||
the addition cannot change a describe call — the value exists for
|
||||
the BYTES route (``app.api.docs.get_document_image`` reads the same
|
||||
map for its ``Content-Type`` and its sandboxed-CSP guard)."""
|
||||
assert IMAGE_MIMES == {
|
||||
".png": "image/png",
|
||||
".jpg": "image/jpeg",
|
||||
@@ -770,8 +787,12 @@ def test_image_mimes_cover_the_default_family() -> None:
|
||||
".webp": "image/webp",
|
||||
".gif": "image/gif",
|
||||
".bmp": "image/bmp",
|
||||
".svg": "image/svg+xml",
|
||||
}
|
||||
assert IMAGE_FALLBACK_MIME == "application/octet-stream"
|
||||
# Every member of the default image set now has a mime — raster and
|
||||
# vector alike (the vector one is serve-side only, per above).
|
||||
assert ".svg" in _settings().image_extension_set
|
||||
assert all(ext in IMAGE_MIMES for ext in _settings().image_extension_set)
|
||||
|
||||
|
||||
|
||||
@@ -1008,6 +1008,7 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
s.summary_backfilled = 1
|
||||
s.dates_updated = 0
|
||||
s.images_failed = 0
|
||||
s.svg_indexed = 4
|
||||
s.image_context_backfilled = 2
|
||||
s.pdfs_screenshotted = 1
|
||||
s.pdfs_pictorial = 2
|
||||
@@ -1021,7 +1022,7 @@ def test_import_summary_log_line_includes_summary_counters(
|
||||
assert line == (
|
||||
"import: summary files=3 added=3 updated=0 unchanged=0 pruned=0 errors=0 "
|
||||
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
|
||||
"dates_updated=0 images_failed=0 image_context_backfilled=2 "
|
||||
"dates_updated=0 images_failed=0 svg_indexed=4 image_context_backfilled=2 "
|
||||
"pdfs_screenshotted=1 pdfs_pictorial=2 pdf_pictorial_pages=5 "
|
||||
"pdf_images_deduped=3 pdf_images_blank_skipped=1 "
|
||||
"formats=yaml:2,md:1"
|
||||
|
||||
@@ -0,0 +1,482 @@
|
||||
"""Unit: phase 181 task 02 — an ``.svg`` indexes as a VECTOR image document.
|
||||
|
||||
The pure extractor (:mod:`app.rag.svg_text`) is pinned in
|
||||
``tests/unit/test_svg_text_extraction.py``; this module pins what the
|
||||
IMPORTER does with it (LOCKED A1/A2/A7):
|
||||
|
||||
* the vector branch of :func:`app.rag.importer._index_image_file` — an
|
||||
``.svg`` lands ``is_image=True`` with ``image_path`` under
|
||||
``image_dir``, its ``content`` is the file's OWN extracted text, its
|
||||
``summary`` mirrors it (the phase-122 convention), and the vision seam
|
||||
:func:`app.rag.importer._describe_or_skip` is NEVER called (the seam is
|
||||
patched with a recording stand-in that would return a sentinel — the
|
||||
empty recording list + the content assertion are the proof, and a
|
||||
call-counting seam cannot silently succeed);
|
||||
* the log-only ``svg_indexed`` counter and the ``import: svg indexed``
|
||||
auditability line (AGENTS rule 10), including ``structural=True`` for
|
||||
a drawing whose only words are the LOCKED A2 fallback line;
|
||||
* the fail-soft (LOCKED A2's boundary): a file the extractor REFUSES
|
||||
(over the byte cap, malformed XML) skips exactly like an undescribed
|
||||
raster — ``images_failed``, a warning, no row, no orphan copy — while
|
||||
a merely textless SVG does NOT skip;
|
||||
* the phase-125 ``image_context`` enrichment prepends for an SVG exactly
|
||||
as for a raster;
|
||||
* the phase-181 fast-path kind guard (:func:`_row_kind_mismatch`): a row
|
||||
imported as TEXT before ``svg`` joined ``BOR_IMAGE_EXTENSIONS`` is
|
||||
converted on the NEXT sync (same bytes, same ``file_hash`` — and for an
|
||||
ASCII file the text digest equals the byte digest too, which is why the
|
||||
image branch's own content-digest skip now requires ``is_image``), and
|
||||
a row already of the right kind — raster or vector — keeps the
|
||||
phase-147 fast path byte-for-byte (no second seam call, no second
|
||||
``svg_indexed``).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from sqlalchemy import select
|
||||
|
||||
import app.rag.importer as importer
|
||||
from app.config import Settings
|
||||
from app.models import Chunk, Document
|
||||
from app.rag.importer import import_sources
|
||||
from app.rag.svg_text import MAX_SVG_BYTES
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
#: A diagram that carries words: ``<title>`` + ``<desc>`` + a ``<text>``
|
||||
#: inside a group (document order is the extractor's contract).
|
||||
SVG_TOPO = (
|
||||
b'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 120 80">'
|
||||
b"<title>Server rack topology</title>"
|
||||
b"<desc>Two switches uplink to the router.</desc>"
|
||||
b'<g><path d="M0 0h10"/><text>rack-1</text></g>'
|
||||
b"</svg>"
|
||||
)
|
||||
|
||||
#: The text :func:`app.rag.svg_text.extract_svg_text` yields for
|
||||
#: :data:`SVG_TOPO` (pinned literally here so a change to the extraction
|
||||
#: contract breaks BOTH modules, not silently).
|
||||
SVG_TOPO_TEXT = "Server rack topology\nTwo switches uplink to the router.\nrack-1"
|
||||
|
||||
#: A pure-shape drawing: no words at all (the LOCKED A2 class).
|
||||
SVG_SHAPES_ONLY = (
|
||||
b'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64">'
|
||||
b'<path d="M0 0h10"/><path d="M0 20h10"/><circle cx="1" cy="1" r="2"/>'
|
||||
b"</svg>"
|
||||
)
|
||||
|
||||
#: The LOCKED A2 structural line for :data:`SVG_SHAPES_ONLY` — built only
|
||||
#: from facts the file states (stem, shape census, viewBox).
|
||||
SVG_SHAPES_ONLY_TEXT = (
|
||||
'Vector diagram "gizmo" — 3 shapes (path×2, circle), viewBox 64×64'
|
||||
)
|
||||
|
||||
#: Over :data:`MAX_SVG_BYTES`: the size guard refuses it BEFORE any parse.
|
||||
SVG_OVER_CAP = b"<svg>" + b"\n" * (MAX_SVG_BYTES + 1) + b"</svg>"
|
||||
|
||||
#: Well-formed bytes, unparseable markup (the second refusal class).
|
||||
SVG_MALFORMED = b'<svg xmlns="http://www.w3.org/2000/svg"><title>oops'
|
||||
|
||||
#: A real 1×1 PNG for the raster fast-path regression pins.
|
||||
PNG_1X1 = b"\x89PNG\r\n\x1a\n" + b"\x00" * 24
|
||||
|
||||
#: What a describe seam would hand back if the vector branch ever reached
|
||||
#: it — never a plausible diagram text, so it cannot pass unnoticed.
|
||||
VISION_SENTINEL = "SHOULD NEVER BE DESCRIBED: a vision call ran for an SVG."
|
||||
|
||||
|
||||
def _settings(**kwargs: Any) -> Settings:
|
||||
"""Build Settings without reading a .env file (deterministic tests)."""
|
||||
kwargs.setdefault("_env_file", None)
|
||||
return Settings(**kwargs) # pyright: ignore[reportCallIssue] (kwarg exists at runtime)
|
||||
|
||||
|
||||
def _image_llm(tmp_path: Path, **kwargs: Any) -> FakeEmbedder:
|
||||
"""A fake LLM whose settings index images into *tmp_path*."""
|
||||
kwargs.setdefault("_env_file", None)
|
||||
kwargs.setdefault("images", True)
|
||||
kwargs.setdefault("image_dir", str(tmp_path / "images"))
|
||||
llm = FakeEmbedder()
|
||||
llm.settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||
return llm
|
||||
|
||||
|
||||
def _record_description(monkeypatch: pytest.MonkeyPatch) -> list[str]:
|
||||
"""Patch the vision seam with a recorder (never a real call): the
|
||||
paths it is asked to describe, always answered with
|
||||
:data:`VISION_SENTINEL`. An empty list + a non-sentinel ``content``
|
||||
is the proof that the vector branch bypassed vision entirely."""
|
||||
handed: list[str] = []
|
||||
|
||||
async def _fake(
|
||||
llm: importer.Embedder,
|
||||
*,
|
||||
data: bytes,
|
||||
source: str,
|
||||
rel: str,
|
||||
full_path: Path,
|
||||
context: str | None = None,
|
||||
) -> str | None:
|
||||
handed.append(rel)
|
||||
return VISION_SENTINEL
|
||||
|
||||
monkeypatch.setattr(importer, "_describe_or_skip", _fake)
|
||||
return handed
|
||||
|
||||
|
||||
def _doc(db, source: str, rel: str) -> Document:
|
||||
doc = db.scalar(
|
||||
select(Document).where(Document.source == source, Document.path == rel)
|
||||
)
|
||||
assert doc is not None, f"no documents row for ({source!r}, {rel!r})"
|
||||
return doc
|
||||
|
||||
|
||||
def _cleanup_source(db, source: str) -> None:
|
||||
for doc in db.scalars(select(Document).where(Document.source == source)).all():
|
||||
db.delete(doc)
|
||||
db.commit()
|
||||
|
||||
|
||||
# --- the vector branch ------------------------------------------------------
|
||||
|
||||
|
||||
def test_svg_indexes_as_an_image_document_with_no_vision_call(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""LOCKED A1: the ``.svg`` takes the image branch (``is_image``,
|
||||
``image_path``, the ``<doc-id>.svg`` copy, the ``summary = content``
|
||||
convention, one embedded content chunk + the ``is_summary`` chunk)
|
||||
and its searchable text is the file's own ``<title>``/``<desc>``/
|
||||
``<text>`` — with the vision seam never invoked. ``svg_indexed`` and
|
||||
the per-file ``import: svg indexed`` line make the run auditable."""
|
||||
root = tmp_path / "svgsrc"
|
||||
root.mkdir()
|
||||
(root / "topo.svg").write_bytes(SVG_TOPO)
|
||||
llm = _image_llm(tmp_path)
|
||||
handed = _record_description(monkeypatch)
|
||||
|
||||
try:
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
|
||||
assert summary.errors == 0
|
||||
assert summary.formats == {"svg": 1} # counted under its image token
|
||||
assert summary.svg_indexed == 1
|
||||
assert handed == [], "an SVG must never reach the vision seam"
|
||||
|
||||
doc = _doc(db, root.name, "topo.svg")
|
||||
assert doc.is_image is True
|
||||
assert doc.title == "topo" # the non-markdown stem rule (shared with raster)
|
||||
assert doc.content == SVG_TOPO_TEXT
|
||||
assert doc.summary == doc.content # the phase-122 image convention
|
||||
assert doc.image_context is None # standalone: the pre-phase row shape
|
||||
assert doc.image_path is not None
|
||||
copy = Path(doc.image_path)
|
||||
assert copy.parent == Path(llm.settings.image_dir).expanduser()
|
||||
assert copy.name == f"{doc.id}.svg" # row and copy agree on the id
|
||||
assert copy.read_bytes() == SVG_TOPO # the exact bytes, unserved here
|
||||
|
||||
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc.id)).all()
|
||||
content_chunks = [c for c in chunks if not c.is_summary]
|
||||
assert [c.content for c in content_chunks] == [doc.content]
|
||||
summary_chunks = [c for c in chunks if c.is_summary]
|
||||
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
|
||||
assert summary_chunks[0].content == doc.content
|
||||
for c in chunks:
|
||||
assert c.embedding is not None and len(c.embedding) == 768
|
||||
|
||||
indexed = [
|
||||
r.getMessage() for r in caplog.records if "import: svg indexed" in r.getMessage()
|
||||
]
|
||||
assert indexed == [
|
||||
f"import: svg indexed source={root.name} path=topo.svg "
|
||||
f"structural=False chars={len(SVG_TOPO_TEXT)}"
|
||||
]
|
||||
line = next(
|
||||
r.getMessage()
|
||||
for r in caplog.records
|
||||
if r.getMessage().startswith("import: summary files=")
|
||||
)
|
||||
assert "svg_indexed=1" in line and "images_failed=0" in line
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_textless_svg_indexes_with_the_structural_line(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||
) -> None:
|
||||
"""LOCKED A2: a diagram of pure ``<path>``s is NOT skipped — it is
|
||||
indexed with the structural line built from the file (stem, shape
|
||||
census, viewBox), counted like any other vector doc and logged with
|
||||
``structural=True`` so the owner can tell it from a worded diagram."""
|
||||
root = tmp_path / "shapeonly"
|
||||
root.mkdir()
|
||||
(root / "gizmo.svg").write_bytes(SVG_SHAPES_ONLY)
|
||||
llm = _image_llm(tmp_path)
|
||||
handed = _record_description(monkeypatch)
|
||||
try:
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
|
||||
assert summary.svg_indexed == 1
|
||||
assert handed == []
|
||||
doc = _doc(db, root.name, "gizmo.svg")
|
||||
assert doc.is_image is True
|
||||
assert doc.content == SVG_SHAPES_ONLY_TEXT
|
||||
assert doc.summary == doc.content
|
||||
line = next(
|
||||
r.getMessage() for r in caplog.records if "import: svg indexed" in r.getMessage()
|
||||
)
|
||||
assert "structural=True" in line
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("name", "payload"),
|
||||
[
|
||||
pytest.param("huge.svg", SVG_OVER_CAP, id="over-the-byte-cap"),
|
||||
pytest.param("broken.svg", SVG_MALFORMED, id="malformed-xml"),
|
||||
],
|
||||
)
|
||||
def test_unusable_svg_skips_the_doc_counts_and_logs(
|
||||
db,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
name: str,
|
||||
payload: bytes,
|
||||
) -> None:
|
||||
"""The fail-soft stays LOCKED A3-shaped for the vector case: a file
|
||||
the extractor refuses skips the doc ENTIRELY — no row, no copy (not
|
||||
even the image dir), ``images_failed`` + a warning naming the vector
|
||||
extraction, the sync continues. This is the ONLY skip class the phase
|
||||
keeps for an SVG (:func:`app.rag.svg_text.extract_svg_text`)."""
|
||||
root = tmp_path / "badsvg"
|
||||
root.mkdir()
|
||||
(root / name).write_bytes(payload)
|
||||
llm = _image_llm(tmp_path)
|
||||
handed = _record_description(monkeypatch)
|
||||
try:
|
||||
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.images_failed) == (1, 0, 1)
|
||||
assert summary.svg_indexed == 0
|
||||
assert handed == []
|
||||
assert db.scalar(select(Document).where(Document.source == root.name)) is None
|
||||
assert not Path(llm.settings.image_dir).expanduser().exists(), (
|
||||
"no copy — and not even the image dir — may be left behind"
|
||||
)
|
||||
warnings = [
|
||||
r
|
||||
for r in caplog.records
|
||||
if "svg text extraction failed" in r.getMessage()
|
||||
]
|
||||
assert len(warnings) == 1 and warnings[0].levelno == logging.WARNING
|
||||
assert "image description failed" not in caplog.text
|
||||
line = next(
|
||||
r.getMessage()
|
||||
for r in caplog.records
|
||||
if r.getMessage().startswith("import: summary files=")
|
||||
)
|
||||
assert "images_failed=1" in line and "svg_indexed=0" in line
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_markdown_reference_context_prepends_for_an_svg(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Phase 125's enrichment is kind-agnostic: a ````
|
||||
reference in the same source still builds ``image_context`` (from the
|
||||
file on disk, no vision call) and the doc's content becomes
|
||||
``<context>\\n\\n<extracted text>`` — context first for retrieval, the
|
||||
vector file's own words second."""
|
||||
root = tmp_path / "svganchor"
|
||||
root.mkdir()
|
||||
(root / "guide.md").write_text(
|
||||
"Why this diagram exists.\n\n\n\nWhat follows it.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(root / "topo.svg").write_bytes(SVG_TOPO)
|
||||
llm = _image_llm(tmp_path)
|
||||
handed = _record_description(monkeypatch)
|
||||
try:
|
||||
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (summary.files, summary.added, summary.images_failed) == (2, 2, 0)
|
||||
assert summary.svg_indexed == 1
|
||||
assert handed == []
|
||||
doc = _doc(db, root.name, "topo.svg")
|
||||
assert doc.is_image is True
|
||||
assert doc.image_context is not None
|
||||
assert doc.image_context.startswith(f"Referenced in {root.name}/guide.md:")
|
||||
assert 'Alt text: "The rack plan"' in doc.image_context
|
||||
assert doc.content == f"{doc.image_context}\n\n{SVG_TOPO_TEXT}"
|
||||
assert doc.summary == doc.content
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
# --- the fast-path kind guard (the text → image migration) ------------------
|
||||
|
||||
|
||||
def test_text_indexed_svg_row_is_converted_on_the_next_sync(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Design D4: the row an earlier run indexed as TEXT (``svg`` not yet
|
||||
in the image set) is converted by the NEXT sync — same bytes, same
|
||||
``file_hash`` — and the sync AFTER that is fast again.
|
||||
|
||||
The conversion needs BOTH halves of the guard: the fast path in
|
||||
:func:`_index_file` must fall through, and the image branch's own
|
||||
content-digest skip must require ``is_image`` — an ASCII file's text
|
||||
digest equals its byte digest, so without it the row would report
|
||||
``unchanged`` forever and never gain a copy (asserted below, because
|
||||
it is the whole reason the guard exists)."""
|
||||
root = tmp_path / "migrate"
|
||||
root.mkdir()
|
||||
(root / "topo.svg").write_bytes(SVG_TOPO)
|
||||
handed = _record_description(monkeypatch)
|
||||
|
||||
# Stage A — the pre-phase state: images OFF, ``svg`` imported as text.
|
||||
text_llm = _image_llm(
|
||||
tmp_path,
|
||||
images=False,
|
||||
import_extensions=f"{_settings().import_extensions},svg",
|
||||
)
|
||||
try:
|
||||
summary_a = asyncio.run(import_sources([root], text_llm, session=db))
|
||||
assert (summary_a.files, summary_a.added, summary_a.svg_indexed) == (1, 1, 0)
|
||||
before = _doc(db, root.name, "topo.svg")
|
||||
assert before.is_image is False
|
||||
assert before.image_path is None
|
||||
assert before.content == SVG_TOPO.decode("utf-8") # the raw XML the owner saw
|
||||
byte_digest = hashlib.sha256(SVG_TOPO).hexdigest()
|
||||
assert before.file_hash == byte_digest
|
||||
# The trap D4 closes: the text digest of a byte-clean file IS the
|
||||
# byte digest, so no digest comparison can spot the mismatch.
|
||||
assert before.content_hash == byte_digest
|
||||
doc_id = before.id
|
||||
|
||||
# Stage B — the toggle on: the guard fires, the row converts.
|
||||
image_llm = _image_llm(tmp_path)
|
||||
summary_b = asyncio.run(import_sources([root], image_llm, session=db))
|
||||
assert (summary_b.added, summary_b.updated, summary_b.unchanged) == (0, 1, 0)
|
||||
assert (summary_b.images_failed, summary_b.svg_indexed) == (0, 1)
|
||||
assert handed == [], "the conversion must not call vision either"
|
||||
after = _doc(db, root.name, "topo.svg")
|
||||
assert after.id == doc_id # converted in place — the id (and copy name) is stable
|
||||
assert after.is_image is True
|
||||
assert after.content == SVG_TOPO_TEXT
|
||||
assert after.summary == after.content
|
||||
assert after.file_hash == byte_digest # the bytes never changed
|
||||
# The image branch stamps the BYTES digest as ``content_hash`` —
|
||||
# the SAME value the text path had already stored, which is why
|
||||
# that skip needed its own ``is_image`` guard.
|
||||
assert after.content_hash == byte_digest
|
||||
copy = Path(str(after.image_path))
|
||||
assert copy == Path(image_llm.settings.image_dir).expanduser() / f"{doc_id}.svg"
|
||||
assert copy.read_bytes() == SVG_TOPO
|
||||
# The text chunks are replaced, not appended: one content chunk
|
||||
# (the extracted text) + the is_summary mirror.
|
||||
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc_id)).all()
|
||||
assert [c.content for c in chunks if not c.is_summary] == [SVG_TOPO_TEXT]
|
||||
assert [c.content for c in chunks if c.is_summary] == [SVG_TOPO_TEXT]
|
||||
|
||||
# Stage C — the guard fired once: the fast path resumes.
|
||||
summary_c = asyncio.run(import_sources([root], image_llm, session=db))
|
||||
assert (summary_c.added, summary_c.updated, summary_c.unchanged) == (0, 0, 1)
|
||||
assert (summary_c.svg_indexed, summary_c.images_failed) == (0, 0)
|
||||
assert handed == []
|
||||
assert _doc(db, root.name, "topo.svg").is_image is True
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_correctly_classified_raster_row_still_takes_the_fast_path(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""The phase-147 fast path survives the guard for every row whose
|
||||
KIND already agrees: a raster image with a matching ``file_hash`` is
|
||||
UNCHANGED on the next sync — the seam is not called a second time and
|
||||
no vector counter moves."""
|
||||
root = tmp_path / "rasterfast"
|
||||
root.mkdir()
|
||||
(root / "pic.png").write_bytes(PNG_1X1)
|
||||
llm = _image_llm(tmp_path)
|
||||
handed = _record_description(monkeypatch)
|
||||
try:
|
||||
first = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (first.added, first.unchanged, first.svg_indexed) == (1, 0, 0)
|
||||
assert handed == ["pic.png"] # the raster branch DID use the seam
|
||||
doc = _doc(db, root.name, "pic.png")
|
||||
assert doc.is_image is True
|
||||
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
assert second.svg_indexed == 0
|
||||
assert handed == ["pic.png"], "the fast path must not re-describe the image"
|
||||
assert _doc(db, root.name, "pic.png").content == VISION_SENTINEL
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_correctly_classified_vector_row_still_takes_the_fast_path(
|
||||
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""Same pin for the vector row: once converted (or freshly indexed)
|
||||
it is UNCHANGED — no second extraction, no second ``svg_indexed``,
|
||||
no second copy write."""
|
||||
root = tmp_path / "vectorfast"
|
||||
root.mkdir()
|
||||
(root / "topo.svg").write_bytes(SVG_TOPO)
|
||||
llm = _image_llm(tmp_path)
|
||||
handed = _record_description(monkeypatch)
|
||||
try:
|
||||
first = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (first.added, first.svg_indexed) == (1, 1)
|
||||
copy = Path(str(_doc(db, root.name, "topo.svg").image_path))
|
||||
copy.unlink() # the fast path must not re-create it either
|
||||
|
||||
second = asyncio.run(import_sources([root], llm, session=db))
|
||||
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||
assert (second.svg_indexed, second.images_failed) == (0, 0)
|
||||
assert handed == []
|
||||
assert not copy.exists()
|
||||
finally:
|
||||
_cleanup_source(db, root.name)
|
||||
|
||||
|
||||
def test_row_kind_mismatch_table() -> None:
|
||||
"""The guard's whole truth table (pure — no DB, no filesystem):
|
||||
it fires ONLY for a TEXT row whose file the CURRENT settings would
|
||||
index as an image. The reverse direction (an ``is_image`` row while
|
||||
the toggle is off, or a plain text file) stays on the fast path —
|
||||
the phase-122 prune guard owns the off state (LOCKED A7)."""
|
||||
svg = Path("/src/diagrams/topo.svg")
|
||||
md = Path("/src/notes.md")
|
||||
text_row = Document(source="s", path="topo.svg", is_image=False)
|
||||
image_row = Document(source="s", path="topo.svg", is_image=True)
|
||||
|
||||
def _settings_(*, images: bool) -> Settings:
|
||||
return _settings(images=images)
|
||||
|
||||
# The migration: the toggle is on, the file is an image, the row is not.
|
||||
assert importer._row_kind_mismatch(svg, text_row, _settings_(images=True)) is True
|
||||
# Already an image doc → nothing to migrate.
|
||||
assert importer._row_kind_mismatch(svg, image_row, _settings_(images=True)) is False
|
||||
# Toggle off → the walk never admits the file, the prune guard protects
|
||||
# the row: never force the image path (LOCKED A7).
|
||||
assert importer._row_kind_mismatch(svg, text_row, _settings_(images=False)) is False
|
||||
assert importer._row_kind_mismatch(svg, image_row, _settings_(images=False)) is False
|
||||
# A file outside the image set is a text file either way.
|
||||
assert importer._row_kind_mismatch(md, text_row, _settings_(images=True)) is False
|
||||
# An owner-narrowed image set (no ``svg``) is not a mismatch either.
|
||||
narrow = _settings(images=True, image_extensions="png,jpg")
|
||||
assert importer._row_kind_mismatch(svg, text_row, narrow) is False
|
||||
@@ -0,0 +1,426 @@
|
||||
"""Unit tests: SVG text extraction (phase 181, task 01).
|
||||
|
||||
:func:`app.rag.svg_text.extract_svg_text` is a PURE function — the text
|
||||
that makes an SVG document retrievable WITHOUT a vision call (LOCKED
|
||||
A1). No DB, no filesystem, no LLM: every case is a literal SVG string,
|
||||
so the whole contract is pinned here before the importer branch (task
|
||||
02) relies on it.
|
||||
|
||||
The matrix:
|
||||
|
||||
* the guards that run BEFORE any parse — empty / whitespace-only input,
|
||||
the :data:`~app.rag.svg_text.MAX_SVG_BYTES` cap, the
|
||||
``<!DOCTYPE``/``<!ENTITY`` refusal (case-insensitive, both a doctype
|
||||
wrapper and a bare entity declaration), malformed XML → ``None``
|
||||
(never a raise);
|
||||
* extraction — ``<title>``/``<desc>``/``<text>`` in DOCUMENT order,
|
||||
``<tspan>`` children joined with a space, namespace-prefixed and
|
||||
namespaced tags treated alike, ``xlink:`` attributes ignored,
|
||||
whitespace normalised, repeats dropped case-insensitively keeping the
|
||||
first, ``&``/``—`` decoded by the parser;
|
||||
* the :data:`~app.rag.svg_text.MAX_SVG_TEXT_CHARS` cap — cut at the
|
||||
last newline/period before the cap, hard-cut only when the window
|
||||
holds no boundary at all;
|
||||
* the LOCKED A2 structural fallback — stem, shape census (counts,
|
||||
plural, top-3 selection with a document-order tie-break) and the
|
||||
``viewBox`` clause, each omitted when the file does not state it.
|
||||
|
||||
Importer-level tests (the vector branch, the no-vision-call seam) live
|
||||
in the phase's ingest module (tasks 02/05).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from app.rag.svg_text import (
|
||||
MAX_SVG_BYTES,
|
||||
MAX_SVG_TEXT_CHARS,
|
||||
SvgText,
|
||||
extract_svg_text,
|
||||
local_name,
|
||||
)
|
||||
|
||||
SVG_NS = "http://www.w3.org/2000/svg"
|
||||
XLINK_NS = "http://www.w3.org/1999/xlink"
|
||||
|
||||
|
||||
# --- constants -------------------------------------------------------------
|
||||
|
||||
|
||||
def test_module_constants_are_pinned() -> None:
|
||||
"""The caps are part of the contract the importer's tests assume."""
|
||||
assert MAX_SVG_BYTES == 2 * 1024 * 1024
|
||||
assert MAX_SVG_TEXT_CHARS == 4000
|
||||
|
||||
|
||||
# --- the local_name helper -------------------------------------------------
|
||||
|
||||
|
||||
def test_local_name_strips_the_clark_notation_namespace() -> None:
|
||||
assert local_name(f"{{{SVG_NS}}}title") == "title"
|
||||
assert local_name("title") == "title"
|
||||
|
||||
|
||||
def test_local_name_of_a_non_string_tag_is_empty() -> None:
|
||||
"""Comment / PI markers are callables — they carry no name."""
|
||||
assert local_name(object()) == ""
|
||||
|
||||
|
||||
# --- guards (no parse attempted) -------------------------------------------
|
||||
|
||||
|
||||
def test_empty_input_is_a_refusal_not_a_crash() -> None:
|
||||
assert extract_svg_text(b"") is None
|
||||
assert extract_svg_text(b"", stem="diagram") is None
|
||||
|
||||
|
||||
def test_whitespace_only_input_is_a_refusal() -> None:
|
||||
assert extract_svg_text(b" \n\t \r\n") is None
|
||||
|
||||
|
||||
def test_bytes_over_the_size_cap_are_refused() -> None:
|
||||
"""The memory guard: an over-cap file never reaches the parser."""
|
||||
padded = b"<svg><title>" + b"x" * (MAX_SVG_BYTES + 1) + b"</title></svg>"
|
||||
assert len(padded) > MAX_SVG_BYTES
|
||||
assert extract_svg_text(padded, stem="huge") is None
|
||||
|
||||
|
||||
def test_bytes_exactly_at_the_size_cap_are_parsed() -> None:
|
||||
"""The cap is ``>``-based: exactly ``MAX_SVG_BYTES`` is in range."""
|
||||
opener, closer = b"<svg><title>", b"</title></svg>"
|
||||
filler = b"x" * (MAX_SVG_BYTES - len(opener) - len(closer))
|
||||
result = extract_svg_text(opener + filler + closer)
|
||||
assert result is not None
|
||||
assert result.structural is False
|
||||
|
||||
|
||||
def test_doctype_with_an_internal_entity_is_refused() -> None:
|
||||
"""Stdlib ElementTree happily EXPANDS an internal entity — proving
|
||||
why the doctype-bearing input is refused outright (billion-laughs
|
||||
class): the reference must never be resolved."""
|
||||
svg = b'<!DOCTYPE svg [<!ENTITY x "y">]><svg><title>&x;</title></svg>'
|
||||
assert extract_svg_text(svg, stem="bomb") is None
|
||||
|
||||
|
||||
def test_bare_entity_declaration_is_refused() -> None:
|
||||
assert extract_svg_text(b'<!ENTITY x "y"><svg><title>x</title></svg>') is None
|
||||
|
||||
|
||||
def test_entity_declaration_hidden_inside_content_is_refused() -> None:
|
||||
"""The marker scan is over the raw bytes — a declaration anywhere
|
||||
(even in a position the parser would reject) refuses the file."""
|
||||
assert extract_svg_text(b"<svg>\n<!entity x 'y'>\n</svg>") is None
|
||||
|
||||
|
||||
def test_guards_are_case_insensitive() -> None:
|
||||
assert extract_svg_text(b'<!doctype svg SYSTEM "http://x/y.dtd"><svg/>') is None
|
||||
assert extract_svg_text(b'<!entity x "y"><svg/>') is None
|
||||
|
||||
|
||||
def test_malformed_xml_is_a_refusal_not_an_exception() -> None:
|
||||
"""The caller's fail-soft is a skip (``images_failed``), so a parse
|
||||
error must never escape."""
|
||||
assert extract_svg_text(b"<svg><title>unclosed") is None
|
||||
assert extract_svg_text(b"<svg><rect></svg>") is None
|
||||
assert extract_svg_text(b"\x00\x01\x02") is None
|
||||
assert extract_svg_text(b"not xml at all") is None
|
||||
|
||||
|
||||
# --- extraction: title / desc / text ---------------------------------------
|
||||
|
||||
|
||||
def test_title_desc_and_text_are_collected_in_document_order() -> None:
|
||||
svg = (
|
||||
b"<svg>"
|
||||
b"<title>Top Title</title>"
|
||||
b"<desc>What the diagram shows</desc>"
|
||||
b"<text>first label</text>"
|
||||
b"<text>second label</text>"
|
||||
b"</svg>"
|
||||
)
|
||||
assert extract_svg_text(svg, stem="topo") == SvgText(
|
||||
text="Top Title\nWhat the diagram shows\nfirst label\nsecond label",
|
||||
structural=False,
|
||||
)
|
||||
|
||||
|
||||
def test_a_title_after_its_shape_still_follows_document_order() -> None:
|
||||
svg = b"<svg><text>later</text><g><title>group title</title></g><desc>d</desc></svg>"
|
||||
assert extract_svg_text(svg) == SvgText(
|
||||
text="later\ngroup title\nd", structural=False
|
||||
)
|
||||
|
||||
|
||||
def test_title_and_desc_concatenate_nested_markup() -> None:
|
||||
svg = b"<title>Deploy <tspan>flow</tspan> map</title>"
|
||||
full = b"<svg>" + svg + b"</svg>"
|
||||
assert extract_svg_text(full) == SvgText(text="Deploy flow map", structural=False)
|
||||
|
||||
|
||||
def test_text_children_are_joined_with_a_space() -> None:
|
||||
"""``<text>CPU<tspan>usage</tspan></text>`` reads "CPU usage" —
|
||||
without the space the labels weld into one unsearchable token."""
|
||||
svg = b'<svg><text>CPU<tspan>usage</tspan>7d</text></svg>'
|
||||
assert extract_svg_text(svg) == SvgText(text="CPU usage 7d", structural=False)
|
||||
|
||||
|
||||
def test_whitespace_is_normalised_in_every_item() -> None:
|
||||
svg = b"<svg><title> rack\n\t layout map </title><text> a\n b </text></svg>"
|
||||
assert extract_svg_text(svg) == SvgText(
|
||||
text="rack layout map\na b", structural=False
|
||||
)
|
||||
|
||||
|
||||
def test_blank_title_is_dropped_and_never_stops_the_walk() -> None:
|
||||
svg = b"<svg><title> </title><desc> </desc><text>real label</text></svg>"
|
||||
assert extract_svg_text(svg) == SvgText(text="real label", structural=False)
|
||||
|
||||
|
||||
def test_only_blank_text_falls_back_to_the_structural_line() -> None:
|
||||
"""A blank ``<title>``/``<text>`` contributes no words, so the file
|
||||
is still described by what it is (the empty ``<text>`` counts as a
|
||||
drawn element)."""
|
||||
svg = b'<svg viewBox="0 0 4 4"><title></title><text> </text></svg>'
|
||||
assert extract_svg_text(svg, stem="blank") == SvgText(
|
||||
text='Vector diagram "blank" — 1 shape (text), viewBox 4×4', structural=True
|
||||
)
|
||||
|
||||
|
||||
def test_repeats_are_dropped_case_insensitively_keeping_the_first() -> None:
|
||||
svg = (
|
||||
b"<svg>"
|
||||
b"<desc>Power Draw</desc>"
|
||||
b"<title>power draw</title>"
|
||||
b"<text>Other</text>"
|
||||
b"</svg>"
|
||||
)
|
||||
assert extract_svg_text(svg) == SvgText(text="Power Draw\nOther", structural=False)
|
||||
|
||||
|
||||
def test_parser_decodes_entity_references_in_the_text() -> None:
|
||||
"""``&`` comes out decoded — the renderer re-escapes on display
|
||||
(phase 180), so storing the decoded value is the correct direction."""
|
||||
svg = b"<svg><title>Red & Blue — map</title></svg>"
|
||||
assert extract_svg_text(svg) == SvgText(text="Red & Blue — map", structural=False)
|
||||
|
||||
|
||||
def test_attributes_are_never_text() -> None:
|
||||
svg = (
|
||||
b'<svg xmlns:xlink="'
|
||||
+ XLINK_NS.encode()
|
||||
+ b'"><title lang="en">Visible</title>'
|
||||
b'<image xlink:href="secret-internal-name.png"/></svg>'
|
||||
)
|
||||
assert extract_svg_text(svg, stem="diagram") == SvgText(
|
||||
text="Visible", structural=False
|
||||
)
|
||||
|
||||
|
||||
def test_an_image_only_svg_yields_no_words_from_its_href() -> None:
|
||||
"""ASSUMPTION (task 01): an embedded ``<image>`` is not followed —
|
||||
its ``href`` is not text, so the structural line describes the file."""
|
||||
svg = (
|
||||
b'<svg xmlns:xlink="'
|
||||
+ XLINK_NS.encode()
|
||||
+ b'" viewBox="0 0 10 10">'
|
||||
b'<image xlink:href="inner.png"/>'
|
||||
b"</svg>"
|
||||
)
|
||||
result = extract_svg_text(svg, stem="poster")
|
||||
assert result is not None
|
||||
assert result.structural is True
|
||||
assert result.text == 'Vector diagram "poster" — 1 shape (image), viewBox 10×10'
|
||||
|
||||
|
||||
# --- extraction: namespaces ------------------------------------------------
|
||||
|
||||
|
||||
def test_the_default_svg_namespace_is_ignored() -> None:
|
||||
svg = (
|
||||
b'<svg xmlns="'
|
||||
+ SVG_NS.encode()
|
||||
+ b'"><title>Namespace title</title><desc>ns desc</desc>'
|
||||
b'<text>ns label</text></svg>'
|
||||
)
|
||||
assert extract_svg_text(svg) == SvgText(
|
||||
text="Namespace title\nns desc\nns label", structural=False
|
||||
)
|
||||
|
||||
|
||||
def test_a_declared_prefix_form_counts_like_the_plain_tag() -> None:
|
||||
"""``ElementTree`` expands ``<svg:title>`` to Clark notation, so the
|
||||
local-name match covers it for free."""
|
||||
svg = (
|
||||
b'<svg xmlns:svg="'
|
||||
+ SVG_NS.encode()
|
||||
+ b'"><svg:title>Prefixed</svg:title>'
|
||||
b"<svg:desc>Prefixed desc</svg:desc></svg>"
|
||||
)
|
||||
assert extract_svg_text(svg) == SvgText(
|
||||
text="Prefixed\nPrefixed desc", structural=False
|
||||
)
|
||||
|
||||
|
||||
def test_a_foreign_namespace_element_is_not_worded_text() -> None:
|
||||
"""A ``<title>`` in a NON-SVG vocabulary is still matched by local
|
||||
name (the walk is namespace-agnostic) while the foreign element
|
||||
counts as a shape when it carries no words."""
|
||||
svg = (
|
||||
b'<svg xmlns:h="http://www.w3.org/1999/xhtml">'
|
||||
b'<foreignObject><h:p>paragraph</h:p></foreignObject>'
|
||||
b"</svg>"
|
||||
)
|
||||
result = extract_svg_text(svg, stem="mixed")
|
||||
assert result is not None
|
||||
assert result.structural is True
|
||||
assert result.text == 'Vector diagram "mixed" — 2 shapes (foreignObject, p)'
|
||||
|
||||
|
||||
def test_the_xml_declaration_encoding_is_honoured() -> None:
|
||||
svg = b"<?xml version='1.0' encoding='UTF-8'?><svg><title>Caf\xc3\xa9</title></svg>"
|
||||
assert extract_svg_text(svg) == SvgText(text="Café", structural=False)
|
||||
|
||||
|
||||
def test_comments_are_not_text_and_not_shapes() -> None:
|
||||
svg = b'<svg viewBox="0 0 2 2"><!-- a comment --><path d="M0 0"/></svg>'
|
||||
result = extract_svg_text(svg, stem="clean")
|
||||
assert result is not None
|
||||
assert result.text == 'Vector diagram "clean" — 1 shape (path), viewBox 2×2'
|
||||
|
||||
|
||||
# --- the MAX_SVG_TEXT_CHARS cap --------------------------------------------
|
||||
|
||||
|
||||
def test_extraction_under_the_cap_is_untouched() -> None:
|
||||
svg = b"<svg><title>" + b"x" * (MAX_SVG_TEXT_CHARS - 20) + b"</title></svg>"
|
||||
result = extract_svg_text(svg)
|
||||
assert result is not None
|
||||
assert len(result.text) == MAX_SVG_TEXT_CHARS - 20
|
||||
|
||||
|
||||
def test_over_cap_text_ends_on_a_sentence_boundary() -> None:
|
||||
svg = b"<svg><title>" + b"x. " * 2000 + b"</title></svg>"
|
||||
result = extract_svg_text(svg)
|
||||
assert result is not None
|
||||
assert result.structural is False
|
||||
assert len(result.text) <= MAX_SVG_TEXT_CHARS
|
||||
assert len(result.text) > MAX_SVG_TEXT_CHARS - 10
|
||||
assert result.text.endswith(".")
|
||||
assert result.text.count("x.") < 2000 # the cut is mid-item
|
||||
|
||||
|
||||
def test_over_cap_text_prefers_the_last_newline_boundary() -> None:
|
||||
"""Two items joined with ``\\n``: the newline is the LAST boundary
|
||||
inside the window, so the second item is dropped whole (never a
|
||||
partial word) and the trailing newline is stripped."""
|
||||
first = b"a" * (MAX_SVG_TEXT_CHARS - 10)
|
||||
svg = b"<svg><title>" + first + b"</title><text>" + b"b" * 200 + b"</text></svg>"
|
||||
result = extract_svg_text(svg)
|
||||
assert result is not None
|
||||
assert result.text == first.decode()
|
||||
|
||||
|
||||
def test_over_cap_text_without_any_boundary_is_hard_cut() -> None:
|
||||
svg = b"<svg><title>" + b"x" * (MAX_SVG_TEXT_CHARS + 1000) + b"</title></svg>"
|
||||
result = extract_svg_text(svg)
|
||||
assert result is not None
|
||||
assert result.text == "x" * MAX_SVG_TEXT_CHARS
|
||||
|
||||
|
||||
# --- the structural fallback (LOCKED A2) -----------------------------------
|
||||
|
||||
|
||||
def test_single_shape_line_matches_the_pinned_wording() -> None:
|
||||
svg = b'<svg><path d="M0 0 L1 1"/></svg>'
|
||||
assert extract_svg_text(svg, stem="topo") == SvgText(
|
||||
text='Vector diagram "topo" — 1 shape (path)', structural=True
|
||||
)
|
||||
|
||||
|
||||
def test_shape_noun_pluralises_and_counts_are_listed() -> None:
|
||||
svg = b"<svg><path/><path/><path/></svg>"
|
||||
assert extract_svg_text(svg, stem="topo") == SvgText(
|
||||
text='Vector diagram "topo" — 3 shapes (path×3)', structural=True
|
||||
)
|
||||
|
||||
|
||||
def test_top_three_kinds_with_document_order_tie_break() -> None:
|
||||
svg = b"<svg><g/><g/><g/><path/><path/><circle/><rect/></svg>"
|
||||
result = extract_svg_text(svg, stem="census")
|
||||
assert result is not None
|
||||
assert result.text == 'Vector diagram "census" — 7 shapes (g×3, path×2, circle)'
|
||||
assert result.structural is True
|
||||
|
||||
|
||||
def test_structure_elements_are_never_counted_as_shapes() -> None:
|
||||
svg = b"<svg><title> </title><desc/><g><rect/></g></svg>"
|
||||
result = extract_svg_text(svg, stem="nested")
|
||||
assert result is not None
|
||||
assert result.text == 'Vector diagram "nested" — 2 shapes (g, rect)'
|
||||
|
||||
|
||||
def test_a_nested_svg_element_is_structure_too() -> None:
|
||||
svg = b"<svg><g><svg><rect/></svg></g></svg>"
|
||||
result = extract_svg_text(svg, stem="inset")
|
||||
assert result is not None
|
||||
assert result.text == 'Vector diagram "inset" — 2 shapes (g, rect)'
|
||||
|
||||
|
||||
def test_viewbox_reports_width_and_height() -> None:
|
||||
svg = b'<svg viewBox="0 0 128 64"><path/></svg>'
|
||||
result = extract_svg_text(svg, stem="wide")
|
||||
assert result is not None
|
||||
assert result.text == 'Vector diagram "wide" — 1 shape (path), viewBox 128×64'
|
||||
|
||||
|
||||
def test_viewbox_accepts_commas_and_integral_floats() -> None:
|
||||
svg = b'<svg viewBox=" 0 , 0 , 8.0 , 6.0 "><path/></svg>'
|
||||
result = extract_svg_text(svg, stem="comma")
|
||||
assert result is not None
|
||||
assert result.text == 'Vector diagram "comma" — 1 shape (path), viewBox 8×6'
|
||||
|
||||
|
||||
def test_viewbox_keeps_fractional_sizes_without_padding() -> None:
|
||||
svg = b'<svg viewBox="0 0 8.5 6.25"><g><circle/></g></svg>'
|
||||
result = extract_svg_text(svg)
|
||||
assert result is not None
|
||||
assert result.text == "Vector diagram — 2 shapes (g, circle), viewBox 8.5×6.25"
|
||||
|
||||
|
||||
def test_unparseable_viewbox_omits_the_clause() -> None:
|
||||
for raw in (b"", b"0 0 8", b"0 0 8 6 4", b"0 0 wide tall", b"0 0 NaN NaN"):
|
||||
svg = b'<svg viewBox="' + raw + b'"><path/></svg>'
|
||||
result = extract_svg_text(svg, stem="odd")
|
||||
assert result is not None, raw
|
||||
assert result.text == 'Vector diagram "odd" — 1 shape (path)', raw
|
||||
|
||||
|
||||
def test_missing_viewbox_omits_the_clause() -> None:
|
||||
result = extract_svg_text(b"<svg><path/></svg>", stem="nobox")
|
||||
assert result is not None
|
||||
assert "viewBox" not in result.text
|
||||
|
||||
|
||||
def test_viewbox_without_shapes_is_the_only_fact() -> None:
|
||||
svg = b'<svg viewBox="0 0 16 9"></svg>'
|
||||
assert extract_svg_text(svg, stem="canvas") == SvgText(
|
||||
text='Vector diagram "canvas" — viewBox 16×9', structural=True
|
||||
)
|
||||
|
||||
|
||||
def test_empty_stem_never_renders_empty_quotes() -> None:
|
||||
assert extract_svg_text(b"<svg></svg>") == SvgText(
|
||||
text="Vector diagram", structural=True
|
||||
)
|
||||
assert extract_svg_text(b'<svg viewBox="0 0 1 1"><path/></svg>') == SvgText(
|
||||
text="Vector diagram — 1 shape (path), viewBox 1×1", structural=True
|
||||
)
|
||||
|
||||
|
||||
def test_a_root_that_is_not_svg_still_yields_honest_text() -> None:
|
||||
"""The walk is tag-based — a mis-named root is not rejected, and the
|
||||
root is never counted among the shapes."""
|
||||
result = extract_svg_text(b"<html><title>Legacy</title></html>", stem="legacy")
|
||||
assert result == SvgText(text="Legacy", structural=False)
|
||||
structural = extract_svg_text(b"<html><body/></html>", stem="legacy")
|
||||
assert structural == SvgText(
|
||||
text='Vector diagram "legacy" — 1 shape (body)', structural=True
|
||||
)
|
||||
Reference in New Issue
Block a user