phase: 181_svg_document_preview
**Phase 181 — final verification pass (all tasks already complete; verified + docs pass)**
- Verified: `app/rag/svg_text.py` (100 % cov), vector branch in `_index_image_file`, `_row_kind_mismatch` text→image migration, `IMAGE_MIMES[".svg"]`, sandboxed CSP on `GET /api/documents/{id}/image`, `.doc-image-img--vector` in `document.js`/`styles.css`, figure/thumbnail deliberate non-changes.
- Tests/lint (exact commands): `uv run pytest --cov=app --cov-report=term-missing` → **4265 passed, TOTAL 99 %** (exit 0) · `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` → **5 passed in isolation** · `uv run ruff check .` → clean · `uv run pyright` → **0 errors**.
- Regression (each run in isolation, `--no-cov`, all rc=0): `test_image_documents`, `test_chat_image_questions`, `test_answer_images`, `test_chat_answer_clusters`, `test_image_anchor_context`, `test_image_upload_feedback`, `test_markdown_inline_links_images`, `test_prose_word_wrap`, `test_pdf_support`, `test_pictorial_pdf`, `test_garbage_pdf`, `test_pdf_image_extraction`, `test_pdf_image_context`, `test_pdf_pages_in_sources`, `test_pdf_strip_additive_selfheal`, `test_sync_image_progress`.
- Completion criteria: ①SVG→`is_image`+`image_path`, no vision call, own `<title>/<desc>/<text>` — **met**; ②`image/svg+xml` + `default-src 'none'; sandbox` + `private, max-age=3600`, raster headers byte-unchanged — **met**; ③chat figure → modal `<img>`, no XML — **met**; ④`/document.html` full-width vector in `.doc-image` + text below — **met**; ⑤`` paints — **met**; ⑥raster/PDF unchanged — **met**; ⑦all gates green — **met**; ⑧commit + phase-dir move — harness-owned.
- Documentation: found and fixed README inconsistencies — stale forward reference "left to phase 181 / is phase 181's job" → completed-tense sizing note; "Updating the documents" standalone-image list missing `svg`; new **Raster only** bullet for question uploads (`.svg` → 422) documenting the `chat_images` guard; missing `svg_indexed=N` / `import: svg indexed` log contract; env-table heading `122/123/124` → `122/123/124/181`; new troubleshooting entry for an `.svg` showing raw XML. Full audit: every `Settings` field and every `.env.example` var is documented; pages/commands/scripts all exist. AGENTS.md verified accurate (phase-181 §5 clause present) — no changes needed beyond what the phase wrote.
- Notable deviation (kept from task 02): `app/api/chat_images.py` subtracts `_VECTOR_IMAGE_EXTENSIONS` from the question-image accept set — a required consequence of LOCKED A7 (`svg` in the default image set) since `_CHAT_IMAGE_FILENAME_RE` refuses to *serve* `.svg`; the regex itself is untouched, as the phase specified.
- Note for the commit: changes include `README.md`, `AGENTS.md` and `.env.example` — the phase file's `git add app/ frontend/ tests/ .env.example .agents/phases/` line omits the two markdown files; stage them too.
- Next pending phase: `182_verbatim_show_text`.
This commit is contained in:
+125
-11
@@ -111,6 +111,20 @@ summary (stored verbatim, no ``lite`` call, no pointer line), so the
|
||||
``is_summary`` position −1 chunk mirrors ``Document.summary``, which
|
||||
equals ``Document.content``.
|
||||
|
||||
The image branch is RASTER (vision-described, the paragraph above) OR
|
||||
VECTOR (phase 181, LOCKED A1: an ``.svg``). A vector file takes the
|
||||
SAME branch — bytes digest, ``image_dir`` copy, ``is_image``/``image_path``,
|
||||
the ``summary = content`` convention, the phase-125 ``image_context``
|
||||
enrichment — but NEVER the model call: its searchable text is extracted
|
||||
from the file itself (:func:`app.rag.svg_text.extract_svg_text` — the
|
||||
``<title>``/``<desc>``/``<text>`` words, or LOCKED A2's structural line
|
||||
built from the stem, the shape census and the ``viewBox`` when the
|
||||
drawing carries no words at all). A file the extractor refuses (over the
|
||||
size cap, DOCTYPE/entity-bearing, malformed XML) fails soft exactly like
|
||||
an undescribed raster: ``images_failed`` + a warning, no row, no copy.
|
||||
Successful vector indexes count ``svg_indexed`` (log-only) and log one
|
||||
``import: svg indexed`` line.
|
||||
|
||||
PDF files (phase 126, LOCKED A1/A2/A3/A4): a ``.pdf`` file (the
|
||||
``pdf`` token of ``BOR_IMPORT_EXTENSIONS``) takes the PDF index path
|
||||
(:func:`_index_pdf_file` — the branch sits in :func:`_index_file`
|
||||
@@ -359,6 +373,7 @@ from app.rag.summarizer import (
|
||||
describe_pdf_page,
|
||||
generate_summary,
|
||||
)
|
||||
from app.rag.svg_text import SvgText, extract_svg_text
|
||||
|
||||
logger = logging.getLogger("app.importer")
|
||||
|
||||
@@ -433,6 +448,14 @@ class ImportSummary:
|
||||
#: Fail-soft: the sync continues, this counter + the warning line
|
||||
#: are the signal.
|
||||
images_failed: int = 0
|
||||
#: Vector image docs (phase 181, task 02 — LOCKED A1) indexed from
|
||||
#: their OWN extracted text (:func:`app.rag.svg_text.extract_svg_text`)
|
||||
#: with NO vision call. A subset of the image branch: it does NOT
|
||||
#: move ``added``/``updated`` (those count as for any other image).
|
||||
#: Log-only (the sync ``detail`` API shape is untouched — the stats
|
||||
#: log-only convention, phase 133): the owner reads it in the sync
|
||||
#: log beside the per-file ``import: svg indexed`` line.
|
||||
svg_indexed: int = 0
|
||||
#: PDFs (phase 133, task 03) whose text layer failed the garbage
|
||||
#: gate (:func:`app.rag.pdf.is_usable_pdf_text` — empty layer or
|
||||
#: XML/binary soup) and took the page-screenshot vision fallback
|
||||
@@ -498,7 +521,7 @@ class ImportSummary:
|
||||
logger.info(
|
||||
"import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
|
||||
"errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
|
||||
"summary_backfilled=%d dates_updated=%d images_failed=%d "
|
||||
"summary_backfilled=%d dates_updated=%d images_failed=%d svg_indexed=%d "
|
||||
"image_context_backfilled=%d pdfs_screenshotted=%d "
|
||||
"pdfs_pictorial=%d pdf_pictorial_pages=%d pdf_images_deduped=%d "
|
||||
"pdf_images_blank_skipped=%d formats=%s",
|
||||
@@ -515,6 +538,7 @@ class ImportSummary:
|
||||
self.summary_backfilled,
|
||||
self.dates_updated,
|
||||
self.images_failed,
|
||||
self.svg_indexed,
|
||||
self.image_context_backfilled,
|
||||
self.pdfs_screenshotted,
|
||||
self.pdfs_pictorial,
|
||||
@@ -1325,6 +1349,38 @@ def _regate_pdf_strip(
|
||||
)
|
||||
|
||||
|
||||
def _row_kind_mismatch(path: Path, doc: Document, settings: Settings) -> bool:
|
||||
"""Phase 181 (task 02, design D4): does this row's KIND disagree with
|
||||
how *path* would be indexed NOW — the ONE case that must fall
|
||||
THROUGH the phase-147 unchanged fast path?
|
||||
|
||||
``True`` only while the ``images`` toggle is ON and the file matches
|
||||
the image extension set while the row is still a TEXT row
|
||||
(``is_image`` False). That is exactly the migration this phase needs:
|
||||
an ``.svg`` imported before ``svg`` joined
|
||||
:py:attr:`Settings.image_extension_set` is a text document whose
|
||||
``content`` is the raw XML — and because the text digest of an ASCII
|
||||
file equals the digest of its bytes, its ``file_hash`` (and even its
|
||||
``content_hash``) matches perfectly, so a hash-only fast path would
|
||||
freeze it as XML forever: no bytes route, no preview, no
|
||||
``image_url`` citation figure — the exact complaint this phase
|
||||
answers. Falling through runs the image branch, which rewrites the
|
||||
row (``is_image``/``image_path``/extracted ``content`` + the
|
||||
``image_dir`` copy) through the normal index write: no alembic
|
||||
migration, no backfill script, and the next sync is fast again.
|
||||
|
||||
The REVERSE direction is deliberately NOT forced: with the toggle
|
||||
OFF the walk never admits an image file (so an ``is_image`` row never
|
||||
reaches :func:`_index_file` at all) and the phase-122 prune guard
|
||||
keeps the row; requiring ``settings.images`` here keeps that
|
||||
contract byte-identical (``svg`` in ``BOR_IMPORT_EXTENSIONS`` with
|
||||
the toggle off still means "text document", LOCKED A7).
|
||||
"""
|
||||
if not settings.images or doc.is_image:
|
||||
return False
|
||||
return match_extension(path, settings.image_extension_set) is not None
|
||||
|
||||
|
||||
async def _index_file(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -1400,10 +1456,17 @@ async def _index_file(
|
||||
# fast path (and the branches' unchanged path) refresh the
|
||||
# stored date from the same source.
|
||||
raw_date = file_mtime_datetime(full_path)
|
||||
if doc is not None and doc.file_hash == file_digest:
|
||||
if (
|
||||
doc is not None
|
||||
and doc.file_hash == file_digest
|
||||
and not _row_kind_mismatch(full_path, doc, settings)
|
||||
):
|
||||
# The fast path: the raw bytes are what was indexed last time
|
||||
# — the bookkeeping (backfill / D1 / D4) runs in the shared
|
||||
# helper, nothing else does.
|
||||
# helper, nothing else does. Phase 181 (task 02, D4): "the same
|
||||
# bytes" is only "the same document" when the row's KIND also
|
||||
# matches how the file would be indexed NOW — see
|
||||
# :func:`_row_kind_mismatch` (the SVG text→image migration).
|
||||
await _unchanged_doc(
|
||||
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
||||
raw_date=raw_date, file_digest=file_digest, full_path=full_path,
|
||||
@@ -3542,7 +3605,17 @@ async def _index_image_file(
|
||||
on_chunk: Callable[[int, int], None] | None = None,
|
||||
) -> None:
|
||||
"""The phase-122 image branch of :func:`_index_file` — a standalone
|
||||
image is indexed from its BYTES, never its text:
|
||||
image is indexed from its BYTES, never its text.
|
||||
|
||||
Phase 181 (task 02, LOCKED A1) makes the branch RASTER **or**
|
||||
VECTOR: an ``.svg`` takes this same path (digest, copy, ``is_image``,
|
||||
``image_path``, chunking, ``summary = content``, the phase-125
|
||||
``image_context`` enrichment) but its ``content`` comes from
|
||||
:func:`app.rag.svg_text.extract_svg_text` instead of the vision
|
||||
seam, and a success counts the log-only ``svg_indexed``. The bullets
|
||||
and the fail-soft below are shared by both kinds — the only
|
||||
difference is where the text comes from.
|
||||
|
||||
|
||||
* the sha256 digest is over the raw bytes (the digest rule is
|
||||
content identity — the same bytes are the same document);
|
||||
@@ -3600,13 +3673,20 @@ async def _index_image_file(
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||||
|
||||
if doc is not None and doc.content_hash == digest:
|
||||
if doc is not None and doc.is_image and doc.content_hash == digest:
|
||||
# Phase 147: the shared unchanged path — a pre-phase NULL-hash
|
||||
# row, or a bytes-changed/content-same image (the byte digest
|
||||
# still matches). The helper stamps the new file hash (the
|
||||
# next sync takes the fast path); for an image the summary is
|
||||
# the stored description (``doc.content``), so the image-aware
|
||||
# ``_store_summary`` backfill re-stores it verbatim.
|
||||
# Phase 181 (task 02, D4): ``doc.is_image`` joins the guard — a
|
||||
# NON-image row routed here by :func:`_row_kind_mismatch` is the
|
||||
# migration case, not an unchanged one. An ASCII SVG indexed as
|
||||
# TEXT digests to the SAME value here (the text path hashes
|
||||
# ``content.encode()``, which for a byte-clean file equals the
|
||||
# raw-bytes digest), so without it the text→image conversion
|
||||
# would report ``unchanged`` forever and never produce the copy.
|
||||
await _unchanged_doc(
|
||||
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
||||
raw_date=raw_date, file_digest=file_digest, full_path=full_path,
|
||||
@@ -3629,16 +3709,50 @@ async def _index_image_file(
|
||||
# it is generated BEFORE anything is written, so a failure skips the
|
||||
# doc with no row and no copy (the copy is only made after a
|
||||
# successful description — a failure never leaves an orphan).
|
||||
content = await _describe_or_skip(
|
||||
llm, data=data, source=source, rel=rel, full_path=full_path,
|
||||
context=image_context,
|
||||
)
|
||||
#
|
||||
# Phase 181 (task 02, LOCKED A1): the VECTOR branch. An ``.svg`` is
|
||||
# described by NOBODY — OpenAI-compatible vision endpoints reject
|
||||
# ``image/svg+xml`` and no rasteriser dependency is in scope — so
|
||||
# its text comes out of the file itself (its ``<title>``/``<desc>``/
|
||||
# ``<text>`` words, or LOCKED A2's structural line). Everything else
|
||||
# in this function is shared: the byte digest, the ``image_dir``
|
||||
# copy, ``is_image``/``image_path``, the title, the counters, the
|
||||
# phase-125 ``image_context`` prepend below, the chunk pipeline and
|
||||
# ``summary = content``. The seam below is not reached, so no vision
|
||||
# call happens for a vector file (pinned by the unit tests).
|
||||
is_vector = full_path.suffix.lower() == ".svg"
|
||||
svg: SvgText | None = None
|
||||
if is_vector:
|
||||
svg = extract_svg_text(data, stem=full_path.stem)
|
||||
content = None if svg is None else svg.text
|
||||
else:
|
||||
content = await _describe_or_skip(
|
||||
llm, data=data, source=source, rel=rel, full_path=full_path,
|
||||
context=image_context,
|
||||
)
|
||||
if content is None:
|
||||
# LOCKED A3 fail-soft: an undescribed image is unsearchable
|
||||
# noise — skip the doc entirely (no row, no copy).
|
||||
# noise — skip the doc entirely (no row, no copy). Phase 181
|
||||
# (LOCKED A2): the vector case skips ONLY when the extractor
|
||||
# refused the file (over the size cap, DOCTYPE/entity-bearing,
|
||||
# malformed XML) — a textless SVG is NOT a failure, it gets the
|
||||
# structural line.
|
||||
summary.images_failed += 1
|
||||
logger.warning("import: image description failed source=%s path=%s", source, rel)
|
||||
logger.warning(
|
||||
"import: %s source=%s path=%s",
|
||||
"svg text extraction failed" if is_vector else "image description failed",
|
||||
source, rel,
|
||||
)
|
||||
return
|
||||
if svg is not None:
|
||||
# Phase 181 (task 02): the log-only vector counter + the
|
||||
# auditability line (AGENTS rule 10) — ``structural=True`` marks
|
||||
# the docs whose text is the LOCKED A2 fallback line.
|
||||
summary.svg_indexed += 1
|
||||
logger.info(
|
||||
"import: svg indexed source=%s path=%s structural=%s chars=%d",
|
||||
source, rel, svg.structural, len(svg.text),
|
||||
)
|
||||
if image_context is not None:
|
||||
# Phase 125 (task 03): the enriched content — the reference
|
||||
# context FIRST (for retrieval), then the description. The
|
||||
|
||||
@@ -149,6 +149,17 @@ PICTORIAL_PROMPT = f"{PICTORIAL_PAGE_MODE}: {PICTORIAL_INSTRUCTION}"
|
||||
#: takes the :data:`IMAGE_FALLBACK_MIME` data-URL mime in the describe
|
||||
#: call — the vision endpoint may reject it, and the fail-soft skip
|
||||
#: (``images_failed``) is the honest outcome.
|
||||
#:
|
||||
#: Phase 181 adds ``.svg`` — the ONE **vector** entry in the map. It
|
||||
#: exists for the BYTES route only: the vector import branch
|
||||
#: (``app/rag/importer.py::_index_image_file``) takes
|
||||
#: :func:`app.rag.svg_text.extract_svg_text` INSTEAD of the vision
|
||||
#: describe call (LOCKED A1), so an ``.svg`` never reaches
|
||||
#: :func:`image_data_url` and this value is never put in a multimodal
|
||||
#: message. ``app/api/docs.py::get_document_image`` reads the same map
|
||||
#: for the ``Content-Type`` (one map, one truth), and that route is
|
||||
#: where the vector value earns its place — plus the sandboxed CSP the
|
||||
#: vector bytes need and the rasters do not (LOCKED A3, phase 181).
|
||||
IMAGE_MIMES: dict[str, str] = {
|
||||
".png": "image/png",
|
||||
".jpg": "image/jpeg",
|
||||
@@ -156,6 +167,7 @@ IMAGE_MIMES: dict[str, str] = {
|
||||
".webp": "image/webp",
|
||||
".gif": "image/gif",
|
||||
".bmp": "image/bmp",
|
||||
".svg": "image/svg+xml",
|
||||
}
|
||||
|
||||
#: The data-URL mime for an image extension :data:`IMAGE_MIMES` does not
|
||||
|
||||
@@ -0,0 +1,293 @@
|
||||
"""SVG text extraction — the words a vector drawing carries (phase 181).
|
||||
|
||||
Pure functions, no I/O — fully unit-testable. Stdlib only (the
|
||||
:mod:`app.rag.md_images` precedent, phase 125).
|
||||
|
||||
An ``.svg`` joins the image family (LOCKED A1, phase 181): its BYTES are
|
||||
served to the browser so every citation surface paints the graphic
|
||||
instead of XML. But OpenAI-compatible vision endpoints reject
|
||||
``image/svg+xml``, and adding a rasteriser dependency is out of scope —
|
||||
so the vector branch never calls the describer. The text that makes the
|
||||
document **retrievable** comes from the file itself:
|
||||
|
||||
* ``<title>``, ``<desc>`` and ``<text>`` (LOCKED D2) — collected in
|
||||
DOCUMENT order by LOCAL name (namespace-agnostic: ``<title>`` and
|
||||
``{http://www.w3.org/2000/svg}title`` both count, declared ``svg:``
|
||||
prefixes included), whitespace-normalised, de-duplicated
|
||||
case-insensitively keeping the FIRST occurrence, joined with ``\\n``.
|
||||
``<tspan>`` children are joined WITH a space so ``a<tspan>b</tspan>``
|
||||
reads "a b", not "ab". XML entity references inside the text
|
||||
(``&``) are decoded by the parser — the correct direction, the
|
||||
renderer re-escapes on display.
|
||||
* a textless diagram (pure ``<path>``s — extremely common) still gets
|
||||
honest, searchable text (LOCKED A2): one structural line built ONLY
|
||||
from facts the file states, never invented prose.
|
||||
|
||||
Guards, applied BEFORE any parse (untrusted markup, the sibling of
|
||||
LOCKED A3):
|
||||
|
||||
* empty / whitespace-only input → ``None``;
|
||||
* more than :data:`MAX_SVG_BYTES` → ``None`` (memory guard);
|
||||
* a ``<!DOCTYPE`` or ``<!ENTITY`` declaration anywhere in the bytes →
|
||||
``None``. Refusing the doctype outright is what makes stdlib
|
||||
:mod:`xml.etree.ElementTree` safe here: no external entities, no
|
||||
entity expansion (the billion-laughs class);
|
||||
* malformed XML (:class:`xml.etree.ElementTree.ParseError`) → ``None``
|
||||
— never re-raised, the caller's fail-soft is a skip, not a crash.
|
||||
|
||||
What is deliberately NOT followed: ``<image>`` elements embedding a
|
||||
raster inside the SVG (their ``href`` is not text), and
|
||||
``aria-label``/``role`` attributes — ``<title>``/``<desc>`` are the
|
||||
SVG-spec carriers of words and the fallback line covers the rest.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from xml.etree import ElementTree
|
||||
|
||||
#: Hard input cap for one SVG (phase 181, LOCKED A3's sibling) — bytes
|
||||
#: over this are refused without a parse attempt, so a hostile or
|
||||
#: accidentally gigantic file can never be the memory event of a sync.
|
||||
MAX_SVG_BYTES: int = 2 * 1024 * 1024
|
||||
|
||||
#: Cap on the EXTRACTED text (phase 181). It binds on the result, never
|
||||
#: silently mid-word: an over-cap extraction is cut at the last newline
|
||||
#: or period before the cap (a hard cut only when the window holds no
|
||||
#: boundary at all).
|
||||
MAX_SVG_TEXT_CHARS: int = 4000
|
||||
|
||||
#: A document type declaration in the raw bytes (lower-cased marker).
|
||||
#: Its presence — anywhere — refuses the input (billion-laughs /
|
||||
#: external-entity class).
|
||||
_DOCTYPE_MARKER = b"<!doctype"
|
||||
|
||||
#: An entity declaration in the raw bytes (lower-cased marker) — same
|
||||
#: refusal as :data:`_DOCTYPE_MARKER`, checked separately so a stray
|
||||
#: ``<!ENTITY`` inside a stray doctype-less prolog is still caught.
|
||||
_ENTITY_MARKER = b"<!entity"
|
||||
|
||||
#: Elements whose whole text block is one item, nested markup
|
||||
#: concatenated (:meth:`ElementTree.Element.itertext`).
|
||||
_BLOCK_TEXT_NAMES = frozenset({"title", "desc"})
|
||||
|
||||
#: Elements whose text is joined with a space so inline children
|
||||
#: (``<tspan>``) do not weld their neighbours together.
|
||||
_SPACED_TEXT_NAMES = frozenset({"text"})
|
||||
|
||||
#: Local names that are structure, not drawing: never counted as shapes
|
||||
#: in the fallback line (:func:`_shape_clause`).
|
||||
_NON_SHAPE_NAMES = frozenset({"svg", "title", "desc"})
|
||||
|
||||
#: How many element kinds the fallback line names (LOCKED D3: "the
|
||||
#: three most common element local names with counts").
|
||||
_TOP_SHAPE_NAMES = 3
|
||||
|
||||
#: ``viewBox`` components: four numbers separated by whitespace and/or
|
||||
#: commas (the SVG-spec grammar, which allows either).
|
||||
_VIEWBOX_SPLIT_RE = re.compile(r"[\s,]+")
|
||||
|
||||
#: The multiplication sign used between a shape name and its count, and
|
||||
#: between the viewBox width and height — the app's own typography.
|
||||
_TIMES = "×"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SvgText:
|
||||
"""The searchable text of one SVG document (phase 181, LOCKED A1/A2).
|
||||
|
||||
* ``text`` — what gets stored as the document's ``content`` (and
|
||||
``summary``, per the phase-122 image convention);
|
||||
* ``structural`` — ``True`` when the document carried no words and
|
||||
``text`` is the structural fallback line, so the importer can log
|
||||
and count it separately from a genuinely worded diagram.
|
||||
"""
|
||||
|
||||
text: str
|
||||
structural: bool
|
||||
|
||||
|
||||
def local_name(tag: object) -> str:
|
||||
"""The namespace-agnostic name of an element tag.
|
||||
|
||||
``{http://www.w3.org/2000/svg}title`` and ``title`` both give
|
||||
``"title"`` — ElementTree expands every declared prefix (including
|
||||
``<svg:title>``) into Clark notation, so stripping the ``{uri}`` is
|
||||
all "namespace-agnostic" needs. A non-string tag (comment /
|
||||
processing-instruction marker) has no name → ``""``.
|
||||
"""
|
||||
if not isinstance(tag, str):
|
||||
return ""
|
||||
return tag.rsplit("}", 1)[-1]
|
||||
|
||||
|
||||
def extract_svg_text(data: bytes, *, stem: str = "") -> SvgText | None:
|
||||
"""Turn raw SVG bytes into the document's searchable text.
|
||||
|
||||
Returns ``None`` when the input is unusable or unsafe: empty /
|
||||
whitespace-only, larger than :data:`MAX_SVG_BYTES`, carrying a
|
||||
``<!DOCTYPE``/``<!ENTITY`` declaration, or not well-formed XML. The
|
||||
caller's fail-soft (skip the document, count it) is the honest
|
||||
outcome — this function never raises for bad input.
|
||||
|
||||
*stem* is the file name without extension: it names the document in
|
||||
the structural fallback line and is never invented.
|
||||
|
||||
Otherwise a :class:`SvgText` whose ``text`` is the document-order
|
||||
``<title>``/``<desc>``/``<text>`` words (``structural=False``), or —
|
||||
when the file carries no words at all — the structural line built
|
||||
from the stem, the shape census and the ``viewBox``
|
||||
(``structural=True``, LOCKED A2).
|
||||
"""
|
||||
if not data or not data.strip():
|
||||
return None
|
||||
if len(data) > MAX_SVG_BYTES:
|
||||
return None
|
||||
lowered = data.lower()
|
||||
if _DOCTYPE_MARKER in lowered or _ENTITY_MARKER in lowered:
|
||||
return None
|
||||
try:
|
||||
root = ElementTree.fromstring(data)
|
||||
except ElementTree.ParseError:
|
||||
return None
|
||||
items = _dedupe(_walk_text(root))
|
||||
if items:
|
||||
return SvgText(text=_cap_text("\n".join(items)), structural=False)
|
||||
return SvgText(text=_structural_line(root, stem), structural=True)
|
||||
|
||||
|
||||
def _walk_text(root: ElementTree.Element) -> list[str]:
|
||||
"""Every worded ``title``/``desc``/``text`` block, in document order.
|
||||
|
||||
Items are whitespace-normalised; a block that normalises to nothing
|
||||
(an empty ``<title></title>``) is dropped so the fallback can still
|
||||
describe the file.
|
||||
"""
|
||||
items: list[str] = []
|
||||
for el in root.iter():
|
||||
name = local_name(el.tag)
|
||||
if name in _BLOCK_TEXT_NAMES:
|
||||
raw = "".join(el.itertext())
|
||||
elif name in _SPACED_TEXT_NAMES:
|
||||
raw = " ".join(el.itertext())
|
||||
else:
|
||||
continue
|
||||
normalised = " ".join(raw.split())
|
||||
if normalised:
|
||||
items.append(normalised)
|
||||
return items
|
||||
|
||||
|
||||
def _dedupe(items: list[str]) -> list[str]:
|
||||
"""Drop repeats case-insensitively, keeping the FIRST occurrence.
|
||||
|
||||
A ``<title>`` mirrored in a ``<desc>`` (exporters do this) is one
|
||||
sentence in the index, not two; the first position in document
|
||||
order is the one kept.
|
||||
"""
|
||||
seen: set[str] = set()
|
||||
unique: list[str] = []
|
||||
for item in items:
|
||||
key = item.casefold()
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
unique.append(item)
|
||||
return unique
|
||||
|
||||
|
||||
def _cap_text(text: str) -> str:
|
||||
"""Cap *text* at :data:`MAX_SVG_TEXT_CHARS` without cutting a word.
|
||||
|
||||
An over-cap text is cut at the LAST newline or period before the cap
|
||||
(whichever comes later) so the stored text ends on a boundary; only
|
||||
a window with no boundary at all (one colossal run of characters)
|
||||
is hard-cut.
|
||||
"""
|
||||
if len(text) <= MAX_SVG_TEXT_CHARS:
|
||||
return text
|
||||
window = text[:MAX_SVG_TEXT_CHARS]
|
||||
cut = max(window.rfind("\n"), window.rfind("."))
|
||||
if cut < 0:
|
||||
return window
|
||||
return text[: cut + 1].rstrip()
|
||||
|
||||
|
||||
def _structural_line(root: ElementTree.Element, stem: str) -> str:
|
||||
"""The LOCKED A2 fallback: one line of facts, nothing invented.
|
||||
|
||||
``Vector diagram "<stem>" — <N> shapes (<top-3 kinds>), viewBox
|
||||
<w>×<h>`` — each fact clause is omitted when the file does not
|
||||
state it, and the stem clause disappears entirely when empty (no
|
||||
empty quotes). With no facts and no stem the line is just
|
||||
``Vector diagram``.
|
||||
"""
|
||||
head = f'Vector diagram "{stem}"' if stem else "Vector diagram"
|
||||
facts = [c for c in (_shape_clause(root), _viewbox_clause(root)) if c]
|
||||
if not facts:
|
||||
return head
|
||||
return f"{head} — {', '.join(facts)}"
|
||||
|
||||
|
||||
def _shape_clause(root: ElementTree.Element) -> str:
|
||||
"""``<N> shapes (<kinds>)`` — the drawing's element census.
|
||||
|
||||
*N* counts every descendant element except ``svg``, ``title`` and
|
||||
``desc`` (structure, not drawing). The kinds are the
|
||||
:data:`_TOP_SHAPE_NAMES` most common local names, ordered by count
|
||||
(document order breaks ties) and annotated with their count — the
|
||||
count is left off a kind that appears exactly once, which is what
|
||||
the total already says. ``""`` when there are no shapes at all, so
|
||||
the clause is omitted rather than claiming "0 shapes".
|
||||
"""
|
||||
counts: dict[str, int] = {}
|
||||
first_seen: dict[str, int] = {}
|
||||
total = 0
|
||||
for index, el in enumerate(root.iter()):
|
||||
if el is root:
|
||||
continue
|
||||
name = local_name(el.tag)
|
||||
if not name or name in _NON_SHAPE_NAMES:
|
||||
continue
|
||||
counts[name] = counts.get(name, 0) + 1
|
||||
first_seen.setdefault(name, index)
|
||||
total += 1
|
||||
if not total:
|
||||
return ""
|
||||
top = sorted(counts, key=lambda name: (-counts[name], first_seen[name]))
|
||||
listing = ", ".join(
|
||||
f"{name}{_TIMES}{counts[name]}" if counts[name] > 1 else name
|
||||
for name in top[:_TOP_SHAPE_NAMES]
|
||||
)
|
||||
noun = "shape" if total == 1 else "shapes"
|
||||
return f"{total} {noun} ({listing})"
|
||||
|
||||
|
||||
def _viewbox_clause(root: ElementTree.Element) -> str:
|
||||
"""``viewBox <w>×<h>`` from the root's ``viewBox``, or ``""``.
|
||||
|
||||
The spec grammar is four whitespace-and/or-comma separated numbers;
|
||||
the last two are the width and height. Unparseable (wrong count,
|
||||
non-numeric, non-finite) → ``""`` — the clause is omitted rather
|
||||
than reporting a size the file does not state. Integral values
|
||||
print without a trailing ``.0``.
|
||||
"""
|
||||
raw = (root.get("viewBox") or "").strip()
|
||||
if not raw:
|
||||
return ""
|
||||
parts = [part for part in _VIEWBOX_SPLIT_RE.split(raw) if part]
|
||||
if len(parts) != 4:
|
||||
return ""
|
||||
try:
|
||||
_min_x, _min_y, width, height = (float(part) for part in parts)
|
||||
except ValueError:
|
||||
return ""
|
||||
if not (math.isfinite(width) and math.isfinite(height)):
|
||||
return ""
|
||||
return f"viewBox {_format_number(width)}{_TIMES}{_format_number(height)}"
|
||||
|
||||
|
||||
def _format_number(value: float) -> str:
|
||||
"""``8`` for 8.0, ``8.5`` for 8.5 — no trailing ``.0`` in a size."""
|
||||
return str(int(value)) if value.is_integer() else str(value)
|
||||
Reference in New Issue
Block a user