Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 181_svg_document_preview
Build and Push Containers / build-and-push-app (push) Successful in 1m57s
Details
Build and Push Containers / build-and-push-db (push) Successful in 15s
Details

Browse Source 
**Phase 181 — final verification pass (all tasks already complete; verified + docs pass)**

- Verified: `app/rag/svg_text.py` (100 % cov), vector branch in `_index_image_file`, `_row_kind_mismatch` text→image migration, `IMAGE_MIMES[".svg"]`, sandboxed CSP on `GET /api/documents/{id}/image`, `.doc-image-img--vector` in `document.js`/`styles.css`, figure/thumbnail deliberate non-changes.
- Tests/lint (exact commands): `uv run pytest --cov=app --cov-report=term-missing` → **4265 passed, TOTAL 99 %** (exit 0) · `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` → **5 passed in isolation** · `uv run ruff check .` → clean · `uv run pyright` → **0 errors**.
- Regression (each run in isolation, `--no-cov`, all rc=0): `test_image_documents`, `test_chat_image_questions`, `test_answer_images`, `test_chat_answer_clusters`, `test_image_anchor_context`, `test_image_upload_feedback`, `test_markdown_inline_links_images`, `test_prose_word_wrap`, `test_pdf_support`, `test_pictorial_pdf`, `test_garbage_pdf`, `test_pdf_image_extraction`, `test_pdf_image_context`, `test_pdf_pages_in_sources`, `test_pdf_strip_additive_selfheal`, `test_sync_image_progress`.
- Completion criteria: ①SVG→`is_image`+`image_path`, no vision call, own `<title>/<desc>/<text>` — **met**; ②`image/svg+xml` + `default-src 'none'; sandbox` + `private, max-age=3600`, raster headers byte-unchanged — **met**; ③chat figure → modal `<img>`, no XML — **met**; ④`/document.html` full-width vector in `.doc-image` + text below — **met**; ⑤`![alt](/api/documents/<id>/image)` paints — **met**; ⑥raster/PDF unchanged — **met**; ⑦all gates green — **met**; ⑧commit + phase-dir move — harness-owned.
- Documentation: found and fixed README inconsistencies — stale forward reference "left to phase 181 / is phase 181's job" → completed-tense sizing note; "Updating the documents" standalone-image list missing `svg`; new **Raster only** bullet for question uploads (`.svg` → 422) documenting the `chat_images` guard; missing `svg_indexed=N` / `import: svg indexed` log contract; env-table heading `122/123/124` → `122/123/124/181`; new troubleshooting entry for an `.svg` showing raw XML. Full audit: every `Settings` field and every `.env.example` var is documented; pages/commands/scripts all exist. AGENTS.md verified accurate (phase-181 §5 clause present) — no changes needed beyond what the phase wrote.
- Notable deviation (kept from task 02): `app/api/chat_images.py` subtracts `_VECTOR_IMAGE_EXTENSIONS` from the question-image accept set — a required consequence of LOCKED A7 (`svg` in the default image set) since `_CHAT_IMAGE_FILENAME_RE` refuses to *serve* `.svg`; the regex itself is untouched, as the phase specified.
- Note for the commit: changes include `README.md`, `AGENTS.md` and `.env.example` — the phase file's `git add app/ frontend/ tests/ .env.example .agents/phases/` line omits the two markdown files; stage them too.
- Next pending phase: `182_verbatim_show_text`.
This commit is contained in:
ducoterra
2026-10-14 22:54:43 -04:00
parent 6880bd0866
commit 814a400e6d
70 changed files with 4460 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/181_svg_document_preview/00_phase.md → .agents/phases/complete/181_svg_document_preview/00_phase.md
View File
.agents/phases/todo/181_svg_document_preview/01_svg_text_extraction.md → .agents/phases/complete/181_svg_document_preview/01_svg_text_extraction.md
View File
.agents/phases/todo/181_svg_document_preview/02_svg_index_as_image.md → .agents/phases/complete/181_svg_document_preview/02_svg_index_as_image.md
View File
.agents/phases/todo/181_svg_document_preview/03_svg_bytes_route_csp.md → .agents/phases/complete/181_svg_document_preview/03_svg_bytes_route_csp.md
View File
.agents/phases/todo/181_svg_document_preview/04_svg_preview_render.md → .agents/phases/complete/181_svg_document_preview/04_svg_preview_render.md
View File
.agents/phases/todo/181_svg_document_preview/05_unit_integration_e2e.md → .agents/phases/complete/181_svg_document_preview/05_unit_integration_e2e.md
View File
.agents/reports/181_svg_document_preview/181_svg_document_preview__00_phase.a1.err
View File
.agents/reports/181_svg_document_preview/181_svg_document_preview__00_phase.a1.md
+10
View File
@@ -0,0 +1,10 @@
**Phase 181 — final verification pass (all tasks already complete; verified + docs pass)**
- Verified: `app/rag/svg_text.py` (100 % cov), vector branch in `_index_image_file`, `_row_kind_mismatch` text→image migration, `IMAGE_MIMES[".svg"]`, sandboxed CSP on `GET /api/documents/{id}/image`, `.doc-image-img--vector` in `document.js`/`styles.css`, figure/thumbnail deliberate non-changes.
- Tests/lint (exact commands): `uv run pytest --cov=app --cov-report=term-missing` → **4265 passed, TOTAL 99 %** (exit 0) · `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` → **5 passed in isolation** · `uv run ruff check .` → clean · `uv run pyright` → **0 errors**.
- Regression (each run in isolation, `--no-cov`, all rc=0): `test_image_documents`, `test_chat_image_questions`, `test_answer_images`, `test_chat_answer_clusters`, `test_image_anchor_context`, `test_image_upload_feedback`, `test_markdown_inline_links_images`, `test_prose_word_wrap`, `test_pdf_support`, `test_pictorial_pdf`, `test_garbage_pdf`, `test_pdf_image_extraction`, `test_pdf_image_context`, `test_pdf_pages_in_sources`, `test_pdf_strip_additive_selfheal`, `test_sync_image_progress`.
- Completion criteria: ①SVG→`is_image`+`image_path`, no vision call, own `<title>/<desc>/<text>` — **met**; ②`image/svg+xml` + `default-src 'none'; sandbox` + `private, max-age=3600`, raster headers byte-unchanged — **met**; ③chat figure → modal `<img>`, no XML — **met**; ④`/document.html` full-width vector in `.doc-image` + text below — **met**; ⑤`![alt](/api/documents/<id>/image)` paints — **met**; ⑥raster/PDF unchanged — **met**; ⑦all gates green — **met**; ⑧commit + phase-dir move — harness-owned.
- Documentation: found and fixed README inconsistencies — stale forward reference "left to phase 181 / is phase 181's job" → completed-tense sizing note; "Updating the documents" standalone-image list missing `svg`; new **Raster only** bullet for question uploads (`.svg` → 422) documenting the `chat_images` guard; missing `svg_indexed=N` / `import: svg indexed` log contract; env-table heading `122/123/124` → `122/123/124/181`; new troubleshooting entry for an `.svg` showing raw XML. Full audit: every `Settings` field and every `.env.example` var is documented; pages/commands/scripts all exist. AGENTS.md verified accurate (phase-181 §5 clause present) — no changes needed beyond what the phase wrote.
- Notable deviation (kept from task 02): `app/api/chat_images.py` subtracts `_VECTOR_IMAGE_EXTENSIONS` from the question-image accept set — a required consequence of LOCKED A7 (`svg` in the default image set) since `_CHAT_IMAGE_FILENAME_RE` refuses to *serve* `.svg`; the regex itself is untouched, as the phase specified.
- Note for the commit: changes include `README.md`, `AGENTS.md` and `.env.example` — the phase file's `git add app/ frontend/ tests/ .env.example .agents/phases/` line omits the two markdown files; stage them too.
- Next pending phase: `182_verbatim_show_text`.
.agents/reports/181_svg_document_preview/181_svg_document_preview__00_phase.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/181_svg_document_preview/181_svg_document_preview__00_phase.a1.validate
+139
View File
@@ -0,0 +1,139 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 97%]
........................................................................ [ 99%]
................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 280 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 289 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 855 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/schemas.py 533 0 100%
--------------------------------------------------
TOTAL 6600 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/181_svg_document_preview/181_svg_document_preview__01_svg_text_extraction.a1.err
View File
.agents/reports/181_svg_document_preview/181_svg_document_preview__01_svg_text_extraction.a1.md
+18
View File
@@ -0,0 +1,18 @@
**Implemented** (task 01 — repo-side only, no live infra touched):
- `app/rag/svg_text.py` (new, pure, stdlib-only, `md_images.py` shape): `MAX_SVG_BYTES=2 MiB`, `MAX_SVG_TEXT_CHARS=4000`, frozen `SvgText(text, structural)`, `extract_svg_text(data, *, stem="")`, `local_name(tag)`.
- Pre-parse guards (LOCKED A3 sibling): empty/whitespace-only, `>MAX_SVG_BYTES`, case-insensitive `<!doctype`/`<!entity` refusal (stdlib otherwise *expands* internal entities — pinned by test), `ParseError` → `None` (never raises).
- Extraction (D2): `title`/`desc` (concat) + `text` (space-joined, covers `<tspan>`) in **document order** by local name (Clark-notation / declared-prefix agnostic), whitespace-normalised, case-insensitive dedupe keeping first, `\n`-joined, entity-decoded, capped at the last newline/period before 4000 (hard cut only if no boundary).
- LOCKED A2 fallback: `Vector diagram "<stem>" — <N> shapes (<top-3 kinds>), viewBox <w>×<h>`; every clause omitted when unstated, `1 shape`/`2 shapes` plural, top-3 by count with document-order tie-break, viewBox requires 4 finite numbers, integral values print without `.0`, empty stem → no empty quotes.
- `tests/unit/test_svg_text_extraction.py` — 45 pure-function tests (guards, order, tspans, namespaces, `xlink:` ignored, dedupe, entities, cap variants, all fallback clause/plural/omission shapes, comments, non-`svg` root).
**Gates (all 100% green):**
- `uv run pytest tests/unit/test_svg_text_extraction.py -q` → 45 passed; `--cov=app.rag.svg_text` → **100%**
- `uv run pytest --cov=app --cov-report=term-missing` → **4243 passed**, TOTAL **99%** (svg_text 111/111)
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors
- Completion-criteria one-liners verified: `SvgText(text='Top\na', structural=False)`, `Vector diagram "topo" — 1 shape (path)`, DOCTYPE case → `None`.
**Docs:** no doc impact — the module is not yet wired (no user-visible surface); README/AGENTS updates belong to tasks 02–04.
**Decisions:** (1) per-kind count in the fallback listing is shown only when it is >1 (`3 shapes (path×3)` vs `1 shape (path)`), so the task's pinned literal holds while D3's "with counts" still reads off multi-kind files; (2) added a non-finite guard (`NaN`/`inf` viewBox → clause omitted) so the line never states a size the file doesn't; (3) viewBox clause sits after the shape clause joined by `, `, and uses ` — ` as the single separator when shapes are absent (`Vector diagram "canvas" — viewBox 16×9`).
**Next pending task:** `.agents/phases/todo/181_svg_document_preview/02_svg_index_as_image.md`
.agents/reports/181_svg_document_preview/181_svg_document_preview__01_svg_text_extraction.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/181_svg_document_preview/181_svg_document_preview__01_svg_text_extraction.a1.validate
+138
View File
@@ -0,0 +1,138 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 44%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 96%]
........................................................................ [ 98%]
................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 280 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 234 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 289 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 841 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/schemas.py 533 0 100%
--------------------------------------------------
TOTAL 6579 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/181_svg_document_preview/181_svg_document_preview__02_svg_index_as_image.a1.err
View File
.agents/reports/181_svg_document_preview/181_svg_document_preview__02_svg_index_as_image.a1.md
+24
View File
@@ -0,0 +1,24 @@
All checks green. Report:
**Implemented** (task 02; work was largely staged by earlier context-killed attempts — I verified every step against the task file, ran the gates, and confirmed no gaps):
- `app/config.py`: `image_extensions` default → `png,jpg,jpeg,webp,gif,bmp,svg` + doc-comment on SVG as the one VECTOR member (A1) and `BOR_IMAGES` as the single knob (A7).
- `app/rag/importer.py`: vector branch in `_index_image_file` (`extract_svg_text` replaces `_describe_or_skip` for `.svg`, everything else shared), `svg_indexed` log-only counter + `import: svg indexed … structural=… chars=…` line (AGENTS rule 10), module/branch docstrings, and the D4 fast-path guard `_row_kind_mismatch()` (text-SVG rows convert on the next sync; toggle-off direction untouched per A7).
- Necessary companion: the image branch's own content-digest skip now also requires `doc.is_image` (an ASCII file's text digest equals its byte digest — without it the migration would report `unchanged` forever); documented + pinned by test.
- `app/api/chat_images.py`: question uploads = `image_extension_set − _VECTOR_IMAGE_EXTENSIONS` (`.svg` 422s — a vision endpoint rejects `image/svg+xml` and the serve route's regex would never serve it); `_CHAT_IMAGE_FILENAME_RE` left untouched per phase "NOT touched".
- Docs: `.env.example` (+ vector/BOR_IMAGES=0/paperclip notes), `README.md` formats paragraph, new "SVG is the one vector member" paragraph, settings-table row.
- Tests: new `tests/unit/test_svg_image_ingest.py` (10 tests: no-vision-call proof via sentinel seam, content/summary/chunks/copy/`svg_indexed`, structural-line textless SVG, over-cap + malformed fail-soft (no row, no copy), phase-125 `image_context` prepend, text→image conversion then fast path resumes, raster/vector fast-path regression, `_row_kind_mismatch` truth table); updated `test_image_documents.py`, `test_importer.py` (log line), `test_chat_image_questions.py`.
**Gates**
- `uv run pytest --cov=app --cov-report=term-missing` → **4253 passed**, TOTAL **99%** (`app/rag/svg_text.py` 100%, `app/rag/importer.py` 99% — only pre-existing misses).
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors/0 warnings.
- Targeted: `pytest tests/unit/test_svg_*.py tests/unit/test_image_documents.py tests/unit/test_chat_image_questions.py tests/integration/test_importer{,_dates}.py tests/integration/test_import_docs_git.py --no-cov` → 240 passed.
- E2E regression (image surfaces touched by the default change): `tests/e2e/test_image_documents.py` → 4 passed; `test_chat_image_questions.py` + `test_answer_images.py` → 10 passed.
**Docs**: `README.md` + `.env.example` updated as above; `AGENTS.md` no doc impact (no extension list there).
**Decisions / deviations**
1. Added `doc.is_image` to `_index_image_file`'s unchanged guard (beyond "keep everything after the seam as-is") — required for D4 to work; explained inline and pinned by `test_text_indexed_svg_row_is_converted_on_the_next_sync`.
2. Fail-soft log now reads `import: svg text extraction failed` for the vector case (raster wording byte-identical) for auditability; counter/behaviour unchanged.
3. **Owner action (I did not touch the user-owned live `.env`)**: `.env:63` still pins `BOR_IMAGE_EXTENSIONS=png,jpg,jpeg,webp,gif,bmp`, so the new default has no effect on the running app until `svg` is added (or the line removed) — LOCKED A7 / phase says a live `.env` keeps its value.
**Next pending task**: `.agents/phases/todo/181_svg_document_preview/03_svg_bytes_route_csp.md`
.agents/reports/181_svg_document_preview/181_svg_document_preview__02_svg_index_as_image.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/181_svg_document_preview/181_svg_document_preview__02_svg_index_as_image.a1.validate
+139
View File
@@ -0,0 +1,139 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 44%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
........................................................................ [ 99%]
..... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 280 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 234 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 289 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 855 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/schemas.py 533 0 100%
--------------------------------------------------
TOTAL 6595 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/181_svg_document_preview/181_svg_document_preview__02_svg_index_as_image.a2.err
View File
.agents/reports/181_svg_document_preview/181_svg_document_preview__02_svg_index_as_image.a2.md
+1
View File
@@ -0,0 +1 @@
(no final assistant message — see the .err log)
.agents/reports/181_svg_document_preview/181_svg_document_preview__02_svg_index_as_image.a2.md.status
+1
View File
@@ -0,0 +1 @@
error: assistant error: 400: {"message":"litellm.BadRequestError: OpenAIException - prompt exceeds the 131072-token context; increase --context or shorten the convers…
.agents/reports/181_svg_document_preview/181_svg_document_preview__02_svg_index_as_image.a2.validate
+139
View File
@@ -0,0 +1,139 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 44%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
........................................................................ [ 99%]
..... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 280 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 234 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 289 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 855 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/schemas.py 533 0 100%
--------------------------------------------------
TOTAL 6595 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/181_svg_document_preview/181_svg_document_preview__02_svg_index_as_image.a3.err
View File
.agents/reports/181_svg_document_preview/181_svg_document_preview__02_svg_index_as_image.a3.md
+1
View File
@@ -0,0 +1 @@
(no final assistant message — see the .err log)
.agents/reports/181_svg_document_preview/181_svg_document_preview__02_svg_index_as_image.a3.md.status
+1
View File
@@ -0,0 +1 @@
error: assistant error: 400: {"message":"litellm.BadRequestError: OpenAIException - prompt exceeds the 131072-token context; increase --context or shorten the convers…
.agents/reports/181_svg_document_preview/181_svg_document_preview__02_svg_index_as_image.a3.validate
+139
View File
@@ -0,0 +1,139 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 44%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
........................................................................ [ 99%]
..... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 280 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 234 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 289 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 855 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/schemas.py 533 0 100%
--------------------------------------------------
TOTAL 6595 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/181_svg_document_preview/181_svg_document_preview__03_svg_bytes_route_csp.a1.err
View File
.agents/reports/181_svg_document_preview/181_svg_document_preview__03_svg_bytes_route_csp.a1.md
+13
View File
@@ -0,0 +1,13 @@
**Implemented (task 03 — SVG bytes route + CSP)**
- `app/rag/summarizer.py`: `IMAGE_MIMES[".svg"] = "image/svg+xml"` + doc-comment (the one *vector* entry, bytes-route only — the vector import branch never reaches `image_data_url`).
- `app/api/docs.py`: new `SVG_MEDIA_TYPE` (derived from `IMAGE_MIMES[".svg"]`, so guard ≡ Content-Type) and `SVG_SANDBOX_CSP = "default-src 'none'; sandbox"`; `get_document_image` adds that one header **only** when the resolved media type is SVG. Lookup, public id-as-credential posture, four 404 shapes and `Cache-Control: private, max-age=3600` untouched; raster header set byte-identical. Route + module docstrings give the rationale (navigable URL → stored-XSS seam, `sandbox` = opaque origin, middleware preserves an inner CSP, global `nosniff` still applies, no `Content-Disposition`).
- New `tests/integration/test_svg_document_image.py` (6 tests: SVG mime+sandbox CSP+exact bytes incl. an unsanitized `<script>` sentinel; pinned header key-set, no `content-disposition`; guard keyed on the map not the client (`.SVG`, unknown-ext fallback → global CSP); PNG response unchanged (`CSP` constant, no `sandbox`); anonymous/public posture; all 404 shapes incl. non-uuid→404 and deleted copy).
- `tests/unit/test_image_documents.py`: `IMAGE_MIMES` identity pin extended to `.svg` (six raster entries unchanged); dropped the now-unused `_VECTOR_IMAGE_EXTENSIONS` import (still pinned in `test_chat_image_questions.py`).
**Gates** — `uv run pytest --cov=app --cov-report=term-missing`: **4259 passed**, TOTAL **99%** (`app/api/docs.py` 99%, `app/rag/summarizer.py` 100%); `uv run pytest tests/integration/test_svg_document_image.py tests/integration/test_docs_api.py -v` → **65 passed**; `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. Live check against a real uvicorn: SVG → `content-type: image/svg+xml` + `content-security-policy: default-src 'none'; sandbox`; PNG → `image/png` with no sandbox (seeded rows/copies removed afterwards).
**Docs** — README: SVG paragraph in *Image documents* gained the "served sandboxed" sentence; the middleware-stack area (where A20 headers are described) gained a **Security headers (A20)** note covering the preserved inner CSP. AGENTS.md: no impact (it documents no header behaviour).
**Decisions/deviations** — (1) Constants instead of inline literals (values verbatim per the task) so the guard can't drift from the map. (2) The task's "raster → NO `content-security-policy` key" is unachievable: `SecurityHeadersMiddleware` adds the global A20 CSP to every response, so the raster test pins `CSP == app-wide CSP constant`, no `sandbox`, and an identical header key set — same intent (nothing route-specific added). No live-infra changes.
**Next pending task:** `181_svg_document_preview/04_svg_preview_render.md`.
.agents/reports/181_svg_document_preview/181_svg_document_preview__03_svg_bytes_route_csp.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/181_svg_document_preview/181_svg_document_preview__03_svg_bytes_route_csp.a1.validate
+139
View File
@@ -0,0 +1,139 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 98%]
........................................................................ [ 99%]
........... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 280 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 289 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 855 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/schemas.py 533 0 100%
--------------------------------------------------
TOTAL 6600 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/181_svg_document_preview/181_svg_document_preview__04_svg_preview_render.a1.err
View File
.agents/reports/181_svg_document_preview/181_svg_document_preview__04_svg_preview_render.a1.md
+13
View File
@@ -0,0 +1,13 @@
All gates green. Final report:
**Implemented (task 04, phase 181):**
- `frontend/assets/document.js::docImageBlock` — adds the `doc-image-img--vector` modifier (base class kept) when `doc.format === "svg"`; full rationale comment (no-intrinsic-size → 300×150 default box, `doc.format` is the only type carrier per D7, LOCKED A3 never-inline, legacy-TEXT XML honesty).
- `frontend/assets/styles.css` — new `.doc-image-img--vector { width: 100%; height: auto; object-fit: contain; }` beside `.doc-image-img` with house comment; palette-neutral, base rule untouched.
- `frontend/assets/app.js::appendSourceImageFigure` — deliberate-non-change comment (96px cap scales size-less SVGs; modal is the preview; no `SourceRef.format` field). **Finding:** no code gap — figure keys off `image_url` only.
- `frontend/assets/sources.js` — verified-non-change comment on the `d.is_image` thumbnail branch. **Finding:** no gap found without needing an E2E: `.kb-doc-thumb-img` pins the img to the fixed 48px box (`width/height: 100%` + `object-fit: cover`), and the KB-tree row schema carries `image_url`, so an SVG thumb fills the box like a raster.
- `tests/unit/test_document_viewer.py` — docstring + 4 new tests: format-gate source pins, node behavioral run (svg → compound class, `src`=`image_url`, alt=summary/title-fallback incl. blank-summary; png → base class byte-identical), CSS pins (decls, zero colour literals, `object-fit: contain`, base rule intact), and D7 non-change pins for figure/thumbnail.
- Docs: README SVG section gains the viewer-sizing sentence + the stale "left to phase 181" forward reference in the phase-180 section updated; AGENTS.md §5 gains the phase-181 layout clause.
**Gates:** `uv run pytest` → 4263 passed; `uv run pytest --cov=app` → TOTAL 99% (>90%); `uv run ruff check .` → clean; `uv run pyright` → 0 errors; raster suites unedited & green: `tests/e2e/test_image_documents.py` (4 passed), `test_prose_word_wrap.py`+`test_pdf_support.py`+`test_pictorial_pdf.py` (14 passed), `test_chat_answer_clusters.py` (4 passed) — all `--no-cov`, in isolation. Phase E2E file itself is task 05's deliverable.
**Next pending task:** `181_svg_document_preview/05_unit_integration_e2e.md`.
.agents/reports/181_svg_document_preview/181_svg_document_preview__04_svg_preview_render.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/181_svg_document_preview/181_svg_document_preview__04_svg_preview_render.a1.validate
+139
View File
@@ -0,0 +1,139 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 97%]
........................................................................ [ 99%]
............... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 280 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 289 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 855 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/schemas.py 533 0 100%
--------------------------------------------------
TOTAL 6600 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/181_svg_document_preview/181_svg_document_preview__05_unit_integration_e2e.a1.err
View File
.agents/reports/181_svg_document_preview/181_svg_document_preview__05_unit_integration_e2e.a1.md
+23
View File
@@ -0,0 +1,23 @@
**Phase 181 task 05 complete — all gates green.**
**Implemented**
- `tests/e2e/test_svg_document_preview.py` (new, dedicated module app with `BOR_IMAGES=true` + suite-pinned `BOR_IMAGE_DIR`, phase-138 truncate-then-`import_sources` seeding, `E2E_REAL_LLM` module skip): `test_viewer_renders_svg_not_xml`, `test_chat_figure_and_modal_preview`, `test_blank_svg_is_indexed_and_searchable`, `test_markdown_image_ref_renders` (two-stage seeding), `test_raster_surfaces_unchanged`
- `tests/fixtures/svg_docs/` (new dedicated tree): `diagrams/topology.svg` (viewBox-only, `<title>`/`<desc>`/2 `<text>`, 17 shapes, 2.7 KB), `notes/infra.md` (text co-citation), `diagrams/blank.svg` (pure paths, LOCKED A2 subject)
- Extended `tests/integration/test_svg_document_image.py`: content-endpoint SVG contract (`is_image:true`, `image_url` = `/api/documents/<uuid>/image`, `format:"svg"`, `content` = extracted text, serve round trip) + the wire-contract pin that the literal `image_url` passes phase-180's `allowImageUrl` (regex extracted verbatim from `markdown.js`, with rejecting probes)
- Tasks 01–04's unit files (`test_svg_text_extraction.py`, `test_svg_image_ingest.py`) verified against the task-05 matrix — all items already pinned; no gaps found
**Gates (exact results)**
- `uv run pytest` → **4265 passed** (re-run after audit DB writes: still 4265 passed)
- `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%**, `app/rag/svg_text.py` **100%** (>90% / ≥95% met)
- `uv run pytest tests/e2e/test_svg_document_preview.py --no-cov` → **5 passed in isolation**
- `uv run ruff check .` → clean; `uv run pyright` → **0 errors**
- `uv run python -m scripts.api_payload_audit` (live seeded app) → rc 0, no LOOK rows; SVG route pinned: `:: api method=GET path=/api/documents/<id>/image status=200 bytes=2763 enc=gzip` — far under the 256 KB budget
**Docs**: no doc impact from this task (README/AGENTS/.env.example were updated by tasks 02/03 and re-verified accurate).
**Decisions/deviations**
- Chat-figure assertion scoped by the figure's `source/path` aria-label: the seed legitimately auto-cites blank.svg too (phase-124 auto-cite), so count-1 is asserted on the topology figure specifically.
- Fixture XML comment must avoid `--` (illegal in comments) — caught and fixed; keeps the fixture honestly parseable.
- No commit/move performed: per the harness's binding rule the pipeline makes the one atomic commit (`feat(kb): index SVGs as image documents and preview them instead of raw XML`, `app/ frontend/ tests/ .env.example .agents/`) and gates the phase-dir move; everything is left staged-ready in the working tree.
**Next pending task**: none in phase 181 (05 was the last); next todo phase is `182_verbatim_show_text`.
.agents/reports/181_svg_document_preview/181_svg_document_preview__05_unit_integration_e2e.a1.md.status
+1
View File
@@ -0,0 +1 @@
ok
.agents/reports/181_svg_document_preview/181_svg_document_preview__05_unit_integration_e2e.a1.validate
+139
View File
@@ -0,0 +1,139 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 97%]
........................................................................ [ 99%]
................. [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 280 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 289 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 855 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/schemas.py 533 0 100%
--------------------------------------------------
TOTAL 6600 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/181_svg_document_preview/181_svg_document_preview__05_unit_integration_e2e.a2.err
View File
.agents/reports/181_svg_document_preview/181_svg_document_preview__05_unit_integration_e2e.a2.md
+1
View File
@@ -0,0 +1 @@
(no final assistant message — see the .err log)
.agents/reports/181_svg_document_preview/181_svg_document_preview__05_unit_integration_e2e.a2.md.status
+1
View File
@@ -0,0 +1 @@
error: assistant error: 400: {"message":"litellm.BadRequestError: OpenAIException - prompt exceeds the 131072-token context; increase --context or shorten the convers…
.agents/reports/181_svg_document_preview/181_svg_document_preview__05_unit_integration_e2e.a2.validate
+139
View File
@@ -0,0 +1,139 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 97%]
........................................................................ [ 99%]
............... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 280 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 289 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 855 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/schemas.py 533 0 100%
--------------------------------------------------
TOTAL 6600 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/181_svg_document_preview/181_svg_document_preview__05_unit_integration_e2e.a3.err
View File
.agents/reports/181_svg_document_preview/181_svg_document_preview__05_unit_integration_e2e.a3.md
+1
View File
@@ -0,0 +1 @@
(no final assistant message — see the .err log)
.agents/reports/181_svg_document_preview/181_svg_document_preview__05_unit_integration_e2e.a3.md.status
+1
View File
@@ -0,0 +1 @@
error: assistant error: 400: {"message":"litellm.BadRequestError: OpenAIException - prompt exceeds the 131072-token context; increase --context or shorten the convers…
.agents/reports/181_svg_document_preview/181_svg_document_preview__05_unit_integration_e2e.a3.validate
+139
View File
@@ -0,0 +1,139 @@
........................................................................ [ 1%]
........................................................................ [ 3%]
........................................................................ [ 5%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 16%]
........................................................................ [ 18%]
........................................................................ [ 20%]
........................................................................ [ 21%]
........................................................................ [ 23%]
........................................................................ [ 25%]
........................................................................ [ 27%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 43%]
........................................................................ [ 45%]
........................................................................ [ 47%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 54%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 69%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 76%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 91%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 97%]
........................................................................ [ 99%]
............... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/admin_settings.py 114 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 280 1 99%
app/api/chat_images.py 52 0 100%
app/api/chats.py 111 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 239 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 204 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 289 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/compression.py 59 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 43 0 100%
app/core/errors.py 5 0 100%
app/core/http_cache.py 22 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/response_budget.py 63 0 100%
app/core/security_headers.py 20 0 100%
app/core/settings_catalog.py 18 0 100%
app/core/settings_migrate.py 32 0 100%
app/core/settings_overlay.py 111 0 100%
app/core/theming.py 40 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 105 0 100%
app/models.py 141 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 454 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 77 0 100%
app/rag/importer.py 855 4 99%
app/rag/llm.py 253 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 78 0 100%
app/rag/pdf.py 192 0 100%
app/rag/prompts.py 156 0 100%
app/rag/retriever.py 252 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 58 0 100%
app/rag/svg_text.py 111 0 100%
app/schemas.py 533 0 100%
--------------------------------------------------
TOTAL 6600 17 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.env.example
+9 -1
View File
@@ -164,7 +164,15 @@ BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py,container,network,volume,
# ignored by the import walks and a sync never prunes existing image
# documents.
# BOR_IMAGES=0 # 1 = index standalone images
# BOR_IMAGE_EXTENSIONS=png,jpg,jpeg,webp,gif,bmp # comma-separated, case-insensitive
# BOR_IMAGE_EXTENSIONS=png,jpg,jpeg,webp,gif,bmp,svg # comma-separated, case-insensitive
# # svg is the one VECTOR member: an .svg indexes from its own
# # <title>/<desc>/<text> (or a structural line when it carries no
# # words) and is NEVER sent to the vision model; it is served as
# # image/svg+xml so the chat figure, the modal and the document
# # viewer show the picture instead of the XML. With BOR_IMAGES=0 an
# # .svg stays a plain text document (raw XML in the viewer).
# # Question images (the paperclip) stay raster-only: an .svg upload
# # is a 422.
# BOR_IMAGE_DIR=~/bor-sources/images # persistent home for the served image bytes
# (uploads are replaced, checkouts re-cloned)
AGENTS.md
+8 -1
View File
@@ -30,7 +30,14 @@
`javascript:`/`data:` element), a container-capped
`img.md-img` (`max-width: 100%` — no horizontal overflow at 375px)
and an `h5`/`h6` sized into the existing type scale — never printed
reference text (phase 180); the document viewer's page strip is
reference text (phase 180); an SVG document paints as a
graphic — the viewer's image block adds
`.doc-image-img--vector` (`width: 100%`, `object-fit: contain`)
when `format` is `svg` so a `viewBox`-only drawing spans the
`.doc-image` card instead of the 300×150 default box, while the
chat figure and Sources thumbnail keep their fixed caps and the
bytes stay an `<img>` src (never inlined) (phase 181); the
document viewer's page strip is
CLASS-scoped —
the primary class shows the strip in place of the text, the
additive class never hides the text: Text/Pages toggle group
README.md
+70 -7
View File
@@ -141,8 +141,10 @@ string, so it carries no inline `onerror` handler — the JS fallbacks live
in `document.js`/`app.js`). `.md-img` carries the app's bordered-image card
treatment and is capped to its container — `max-width: 100%`, `height: auto`
and `object-fit: contain` — so a wide screenshot never produces a horizontal
scrollbar at 375px or at 1440px. Sizing an SVG that declares no `width` or
`height` is deliberately left to phase 181.
scrollbar at 375px or at 1440px. An SVG document that declares no `width`
or `height` is sized by phase 181's `.doc-image-img--vector` class in the
document viewer instead (see "Image documents"); an inline answer image
keeps the container cap above.
**The heading ladder reaches six levels** (phase 180, TODO.md L5): a
`#### x` line in an answer, the thinking block, a saved/shared chat, the
@@ -436,7 +438,9 @@ reflect the new files.
- Non-markdown files get format-aware chunking (YAML keys, JSON keys,
Python classes/defs).
- With `BOR_IMAGES=1`, standalone images (`png, jpg, jpeg, webp, gif,
bmp`) are indexed as documents too — see
bmp, svg`) are indexed as documents too — `svg` is the one **vector**
member, indexed from its own markup text with no vision call and
previewed as a graphic (phase 181) — see
[Image support](#image-support).
- A run that changed the KB also regenerates the **KB overview** and the
**folder summaries** (the drill-down catalog's stored per-folder
@@ -858,10 +862,46 @@ first-class document:
row, no copy), logs a warning, and counts in the import summary line
(`images_failed=N`) — the sync continues.
Accepted formats are `png, jpg, jpeg, webp, gif, bmp`
Accepted formats are `png, jpg, jpeg, webp, gif, bmp, svg`
(`BOR_IMAGE_EXTENSIONS`, case-insensitive) — a separate set from
`BOR_IMPORT_EXTENSIONS`; images are never added via that variable, the
toggle is the single knob.
toggle is the single knob. With `BOR_IMAGES=0` an `.svg` is therefore a
plain **text** document — raw XML in the viewer, no image copy, no
thumbnail — and the walks never admit it as an image (no special-casing
of the off state).
**SVG is the one vector member (phase 181).** An `.svg` takes the same
image path — its bytes are copied to `BOR_IMAGE_DIR` and served as
`image/svg+xml`, so the chat figure, the document modal and the
`/document.html` viewer show the *picture* instead of the XML — but it
is **never sent to the vision model** (vision endpoints reject
`image/svg+xml`). Its searchable text comes out of the file itself: the
`<title>`, `<desc>` and `<text>` words, or — for a drawing that carries
no words at all — one honest structural line built from the file (its
name, its shape census, its `viewBox`). A file the extractor refuses
(over 2 MiB, carrying a `<!DOCTYPE`/`<!ENTITY` declaration, or malformed
XML) fails soft like any other image: no row, no copy, `images_failed`.
A successful one logs one `import: svg indexed source=… path=…
structural=… chars=…` line and adds to the log-only `svg_indexed=N` on
the import summary line (`added`/`updated` count it like any other image
doc). The next sync also **converts** an `.svg` that an earlier run
indexed as plain text (its bytes hash is unchanged, so a hash-only fast
path would freeze it as XML forever) — existing libraries need no
re-import by hand. In the viewer the graphic **spans the image card**: a
`viewBox`-only SVG
carries no intrinsic size, so `doc.format === "svg"` adds
`.doc-image-img--vector` (`width: 100%` + `object-fit: contain`) instead
of letting the drawing sit in the browser's 300×150 default box; the
chat figure and the Sources thumbnail keep their fixed caps (96px /
48px) and the modal is the full-size preview. Question images (the
paperclip) stay raster-only: an `.svg` upload is a 422.
Those bytes are served **sandboxed**: because the
`/api/documents/<id>/image` URL is directly navigable (not just an
`<img>` source, which can never run SVG script), an SVG response carries
`Content-Security-Policy: default-src 'none'; sandbox` so a stored
drawing can never execute script or act with the app's origin — raster
image responses keep their headers exactly as they were.
### Question images
@@ -873,6 +913,12 @@ image:
to `POST /api/chat-images` (capped at `BOR_CHAT_IMAGE_MAX_MB`,
default 10 MiB — over is a 413) and stored under `BOR_CHAT_IMAGE_DIR`.
Saved and shared chats carry the returned *path* — never base64.
- **Raster only (phase 181).** The accepted upload extensions are the
**raster** members of `BOR_IMAGE_EXTENSIONS` — an `.svg` is a 422. The
vector family belongs to indexing, not to question attachments: a
vision endpoint rejects `image/svg+xml`, and the question-image serve
route deliberately never hands out a `.svg`, so accepting one would
store an unanswerable, unservable file.
- **Visible while uploading.** While the upload is in flight the
preview strip shows a spinner + *"Uploading image…"* (its remove ✕
disabled) and the Send button is disabled with the *"Uploading…"*
@@ -1068,6 +1114,16 @@ Outermost to innermost, the stack is security-headers → gzip (only when
security headers land on every response and `bytes=` in the budget line is the
route's own, uncompressed payload (see *Operational notes*):
> **Security headers (A20).** Every response carries the app-wide
> `Content-Security-Policy` (`default-src 'self'` plus the phase-123
> `img-src 'self' data:` carve-out), `X-Frame-Options: DENY` and
> `X-Content-Type-Options: nosniff` — and the middleware *preserves* a
> CSP an inner layer already set. One route uses that deliberately: SVG
> document bytes (phase 181) ship `default-src 'none'; sandbox` instead,
> because a directly-navigated stored `.svg` must never become a
> script-capable same-origin document. Raster/PDF image responses gain
> nothing — their header set is unchanged.
- **HTML pages** ship `Cache-Control: no-cache`, **no `etag`**, **no
`last-modified`** — each visit always gets a fresh 200 body.
- **Assets are versioned and cached for a year.** Pages reference their
@@ -1299,12 +1355,12 @@ startup (the DB rows shadow the env, `.env` is never rewritten).
| `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) |
| `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) |
### Image support (phases 122/123/124)
### Image support (phases 122/123/124/181)
| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_IMAGES` | `0` | Master switch for image support — **off by default; enable only when `BOR_LLM_CHAT_MODEL` supports vision**. Gates both image documents and question images |
| `BOR_IMAGE_EXTENSIONS` | `png,jpg,jpeg,webp,gif,bmp` | Extensions indexed as image documents (case-insensitive; separate from `BOR_IMPORT_EXTENSIONS`) |
| `BOR_IMAGE_EXTENSIONS` | `png,jpg,jpeg,webp,gif,bmp,svg` | Extensions indexed as image documents (case-insensitive; separate from `BOR_IMPORT_EXTENSIONS`). `svg` is the one **vector** member: indexed from its own `<title>`/`<desc>`/`<text>` with **no vision call**, served as `image/svg+xml` for a real preview |
| `BOR_IMAGE_DIR` | `~/bor-sources/images` | Persistent home for the served image bytes (uploads are replaced, checkouts re-cloned) |
| `BOR_CHAT_IMAGE_DIR` | `~/bor-sources/chat-images` | Where question images are stored (saved/shared chats carry the path, never base64) |
| `BOR_CHAT_IMAGE_MAX_MB` | `10` | Question-image upload cap (MiB; must be > 0) |
@@ -1539,6 +1595,13 @@ instead of reaching the app.
off (the default). Set `BOR_IMAGES=1` in `.env` and restart — only
when `BOR_LLM_CHAT_MODEL` supports vision. Turning the toggle off
never destroys existing image documents (syncs prune-guard them).
- **An `.svg` shows raw XML instead of the drawing** — image support is
off, so the file is a plain **text** document (phase 181). Set
`BOR_IMAGES=1` and re-sync: the next run converts the row in place
(bytes copied, `image_url` live, no re-import by hand). An `.svg` the
extractor refuses — over 2 MiB, carrying a `<!DOCTYPE`/`<!ENTITY`
declaration, or malformed XML — stays a text document and is counted
in `images_failed`.
- **Honest deflection** (the amber deflection bubble) — every question
passes the honesty gate: deflection happens
when the best cosine similarity is below `BOR_RELEVANCE_THRESHOLD`
app/api/chat_images.py
+26 -7
View File
@@ -22,7 +22,9 @@ unguessable ``uuid4().hex`` — no enumeration value).
The extension is the source of truth (the Content-Type header is a
hint — the archive-uploader precedent): it must be in the phase-122
image set (``settings.image_extension_set`` — the same frozenset the
image-document walk uses), lowercased; total bytes are streamed with a
image-document walk uses) minus the vector members
(:data:`_VECTOR_IMAGE_EXTENSIONS` — a question image is RASTER-only,
phase 181), lowercased; total bytes are streamed with a
``chat_image_max_mb`` cap (413, fixed detail naming the cap — never
echoing the filename).
"""
@@ -55,6 +57,20 @@ _STREAM_CHUNK = 1 << 20
#: upload endpoint's own naming).
_CHAT_IMAGE_FILENAME_RE = re.compile(r"^[0-9a-fA-F]{32}\.(png|jpe?g|webp|gif|bmp)$")
#: The VECTOR members of ``settings.image_extension_set`` — excluded
#: from the question-image accept set (phase 181: the vector family
#: belongs to INDEXING, not to question attachments). Two reasons, both
#: hard: an OpenAI-compatible vision endpoint rejects
#: ``image/svg+xml`` (the phase-181 LOCKED A1 reason — the vector file
#: never reaches the describer at all), and
#: :data:`_CHAT_IMAGE_FILENAME_RE` above deliberately refuses to SERVE a
#: ``.svg``, so an accepted upload would be stored, unanswerable and
#: unservable (a broken figure in the user's own bubble). Subtracting
#: instead of touching the serve regex keeps question attachments
#: raster-only by design while ``svg`` rides the image set for the
#: document walk.
_VECTOR_IMAGE_EXTENSIONS: frozenset[str] = frozenset({".svg"})
@router.post("/chat-images")
async def upload_chat_image(
@@ -66,10 +82,12 @@ async def upload_chat_image(
Gates, in order:
1. **extension** — the file name's extension (lowercased) must be in
the phase-122 image set (``settings.image_extension_set``); the
Content-Type header is a hint, never a source of truth (the
archive-uploader precedent). A missing/unknown extension is a
422 naming the accepted set — a fixed detail, no filename echo.
the phase-122 image set (``settings.image_extension_set``)
MINUS the vector members (:data:`_VECTOR_IMAGE_EXTENSIONS` —
question images are RASTER-only, phase 181); the Content-Type
header is a hint, never a source of truth (the archive-uploader
precedent). A missing/unknown/vector extension is a 422 naming
the accepted set — a fixed detail, no filename echo.
2. **size** — the bytes are streamed (1 MiB chunks) with the
``chat_image_max_mb`` cap; over-cap is a 413 naming the cap
(fixed detail — never echoing the filename), the temp file is
@@ -85,8 +103,9 @@ async def upload_chat_image(
settings = get_settings()
filename = file.filename or ""
ext = Path(filename).suffix.lower()
if ext not in settings.image_extension_set:
accepted = ", ".join(sorted(settings.image_extension_set))
accepted_extensions = settings.image_extension_set - _VECTOR_IMAGE_EXTENSIONS
if ext not in accepted_extensions:
accepted = ", ".join(sorted(accepted_extensions))
raise HTTPException(
status_code=422, detail=f"only {accepted} images are accepted"
)
app/api/docs.py
+59 -5
View File
@@ -52,10 +52,15 @@ BYTES route: the persistent copy behind an ``is_image`` document's
``image_path``, served with the extension's ``Content-Type`` (the
``app.rag.summarizer.IMAGE_MIMES`` map — one map, one truth) and a
``Cache-Control: private, max-age=3600`` header (the bytes are
content-hashed — long enough, bustable by re-upload). PUBLIC (phase
124 — the phase-123 chat-image precedent: the image is part of a
shared chat's rendered content, and the shared chat — the app's ONLY
anonymous surface — must render the answer figure faithfully for
content-hashed — long enough, bustable by re-upload), plus — for the
vector ``image/svg+xml`` entry only, phase 181 LOCKED A3 — a
``Content-Security-Policy: default-src 'none'; sandbox`` so a DIRECT
navigation to the URL cannot run script from stored SVG markup (the
``SecurityHeadersMiddleware`` preserves that inner CSP; raster
responses gain nothing). PUBLIC (phase 124 — the phase-123 chat-image
precedent: the image is part of a shared chat's rendered content, and
the shared chat — the app's ONLY anonymous surface — must render the
answer figure faithfully for
guests; the unguessable ``uuid4`` document id IS the credential, no
enumeration value): a missing doc, a non-image doc, a doc whose
``image_path`` is NULL, or a row whose copy was lost all map to 404
@@ -132,6 +137,23 @@ from app.schemas import (
router = APIRouter(tags=["kb"])
#: The vector media type (phase 181, task 03) — read out of
#: :data:`app.rag.summarizer.IMAGE_MIMES` rather than restated, so the
#: guard below can never drift from the ``Content-Type`` the route
#: sends for an ``.svg`` (one map, one truth).
SVG_MEDIA_TYPE = IMAGE_MIMES[".svg"]
#: The CSP an SVG bytes response carries (phase 181, task 03, LOCKED
#: A3). ``sandbox`` — no scripting, and the document gets an OPAQUE
#: origin (no same-origin fetch/cookie/form capability);
#: ``default-src 'none'`` — no subresource of any kind. Scoped to the
#: vector type ON PURPOSE: an ``<img>`` never runs SVG script, but this
#: URL is directly navigable, and the bytes come from an untrusted
#: source tree, so a direct visit must not become a script-capable
#: same-origin document. Raster types are untouched (their header set
#: stays byte-identical). See :func:`get_document_image`.
SVG_SANDBOX_CSP = "default-src 'none'; sandbox"
def doc_format(path: str, extensions: frozenset[str] = frozenset()) -> str:
"""Lowercased path suffix without its dot (``kubernetes.md`` → ``md``,
@@ -353,6 +375,33 @@ def get_document_image(
``application/octet-stream``) and ``Cache-Control: private,
max-age=3600`` (the bytes are content-hashed — long enough to be
useful, bustable by re-upload).
Phase 181 (task 03, LOCKED A3): an ``image/svg+xml`` response — the
vector entry of that same map, derived from the ROW's extension and
NEVER from a client value — carries ONE extra header,
:data:`SVG_SANDBOX_CSP` (``default-src 'none'; sandbox``). An
``<img>`` can never execute an SVG's script, which is why the
figure surfaces are safe as they stand; but this URL is also
DIRECTLY NAVIGABLE (a guest who pastes it into the address bar),
and without the policy a stored SVG would render as a
script-capable SAME-ORIGIN document — the ingestion path takes
untrusted files from a source tree, so that is a stored-XSS seam
scoped to the app's origin. ``sandbox`` blocks scripting AND makes
the response an opaque origin (no same-origin fetches, cookies or
form submission from the document); ``default-src 'none'`` blocks
every subresource load, so the SVG cannot pull a tracker or a
remote image either. Deliberately NOT applied to the raster types:
their header set stays byte-identical (a PNG needs no policy
change, and adding one would risk the phase-123/124 image
surfaces), and no ``Content-Disposition: attachment`` is set — the
graphic must render inline, and ``sandbox`` is the stronger, less
intrusive control. The route sets the value on the response
itself; :class:`app.core.security_headers.SecurityHeadersMiddleware`
PRESERVES a CSP an inner layer already set, so this is the policy
the browser sees (the app-wide ``CSP`` string would be wrong here —
``default-src 'self'`` would grant the SVG document the app's own
origins), while the middleware's global
``X-Content-Type-Options: nosniff`` still applies.
"""
try:
uid = uuid.UUID(doc_id)
@@ -368,10 +417,15 @@ def get_document_image(
# onerror fallback renders the "image unavailable" note.
raise HTTPException(status_code=404, detail="document not found")
media_type = IMAGE_MIMES.get(image_file.suffix.lower(), IMAGE_FALLBACK_MIME)
headers = {"Cache-Control": "private, max-age=3600"}
if media_type == SVG_MEDIA_TYPE:
# Phase 181 (LOCKED A3): vector bytes only — see the route
# docstring. Raster responses keep the header set they had.
headers["Content-Security-Policy"] = SVG_SANDBOX_CSP
return FileResponse(
image_file,
media_type=media_type,
headers={"Cache-Control": "private, max-age=3600"},
headers=headers,
)
app/config.py
+12 -1
View File
@@ -433,7 +433,18 @@ class Settings(BaseSettings):
#: knob). The validator rejects an empty list and malformed tokens,
#: exactly like ``import_extensions`` (a typo would otherwise index
#: zero images silently).
image_extensions: str = "png,jpg,jpeg,webp,gif,bmp"
#:
#: ``svg`` (phase 181, LOCKED A1) is the one VECTOR member of the
#: family: an ``.svg`` takes the same image branch (bytes copied to
#: ``image_dir``, ``is_image``/``image_path``, served as
#: ``image/svg+xml``) but NEVER the vision call — its searchable
#: text comes from the file itself
#: (:func:`app.rag.svg_text.extract_svg_text`, the ``<title>`` /
#: ``<desc>`` / ``<text>`` words or an honest structural line).
#: ``images`` remains the SINGLE knob (LOCKED A7): with it off an
#: ``.svg`` stays a plain text document (raw XML in the viewer) and
#: the walk never admits it as an image — no special-casing.
image_extensions: str = "png,jpg,jpeg,webp,gif,bmp,svg"
#: Where ingested image bytes are copied for serving (phase 122,
#: ``BOR_IMAGE_DIR``). Raw string — ``Path.expanduser()`` is applied
#: by the importer, not here (the ``sources_dir``/``upload_dir``
app/rag/importer.py
+125 -11
View File
@@ -111,6 +111,20 @@ summary (stored verbatim, no ``lite`` call, no pointer line), so the
``is_summary`` position −1 chunk mirrors ``Document.summary``, which
equals ``Document.content``.
The image branch is RASTER (vision-described, the paragraph above) OR
VECTOR (phase 181, LOCKED A1: an ``.svg``). A vector file takes the
SAME branch — bytes digest, ``image_dir`` copy, ``is_image``/``image_path``,
the ``summary = content`` convention, the phase-125 ``image_context``
enrichment — but NEVER the model call: its searchable text is extracted
from the file itself (:func:`app.rag.svg_text.extract_svg_text` — the
``<title>``/``<desc>``/``<text>`` words, or LOCKED A2's structural line
built from the stem, the shape census and the ``viewBox`` when the
drawing carries no words at all). A file the extractor refuses (over the
size cap, DOCTYPE/entity-bearing, malformed XML) fails soft exactly like
an undescribed raster: ``images_failed`` + a warning, no row, no copy.
Successful vector indexes count ``svg_indexed`` (log-only) and log one
``import: svg indexed`` line.
PDF files (phase 126, LOCKED A1/A2/A3/A4): a ``.pdf`` file (the
``pdf`` token of ``BOR_IMPORT_EXTENSIONS``) takes the PDF index path
(:func:`_index_pdf_file` — the branch sits in :func:`_index_file`
@@ -359,6 +373,7 @@ from app.rag.summarizer import (
describe_pdf_page,
generate_summary,
)
from app.rag.svg_text import SvgText, extract_svg_text
logger = logging.getLogger("app.importer")
@@ -433,6 +448,14 @@ class ImportSummary:
#: Fail-soft: the sync continues, this counter + the warning line
#: are the signal.
images_failed: int = 0
#: Vector image docs (phase 181, task 02 — LOCKED A1) indexed from
#: their OWN extracted text (:func:`app.rag.svg_text.extract_svg_text`)
#: with NO vision call. A subset of the image branch: it does NOT
#: move ``added``/``updated`` (those count as for any other image).
#: Log-only (the sync ``detail`` API shape is untouched — the stats
#: log-only convention, phase 133): the owner reads it in the sync
#: log beside the per-file ``import: svg indexed`` line.
svg_indexed: int = 0
#: PDFs (phase 133, task 03) whose text layer failed the garbage
#: gate (:func:`app.rag.pdf.is_usable_pdf_text` — empty layer or
#: XML/binary soup) and took the page-screenshot vision fallback
@@ -498,7 +521,7 @@ class ImportSummary:
logger.info(
"import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
"errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
"summary_backfilled=%d dates_updated=%d images_failed=%d "
"summary_backfilled=%d dates_updated=%d images_failed=%d svg_indexed=%d "
"image_context_backfilled=%d pdfs_screenshotted=%d "
"pdfs_pictorial=%d pdf_pictorial_pages=%d pdf_images_deduped=%d "
"pdf_images_blank_skipped=%d formats=%s",
@@ -515,6 +538,7 @@ class ImportSummary:
self.summary_backfilled,
self.dates_updated,
self.images_failed,
self.svg_indexed,
self.image_context_backfilled,
self.pdfs_screenshotted,
self.pdfs_pictorial,
@@ -1325,6 +1349,38 @@ def _regate_pdf_strip(
)
def _row_kind_mismatch(path: Path, doc: Document, settings: Settings) -> bool:
"""Phase 181 (task 02, design D4): does this row's KIND disagree with
how *path* would be indexed NOW — the ONE case that must fall
THROUGH the phase-147 unchanged fast path?
``True`` only while the ``images`` toggle is ON and the file matches
the image extension set while the row is still a TEXT row
(``is_image`` False). That is exactly the migration this phase needs:
an ``.svg`` imported before ``svg`` joined
:py:attr:`Settings.image_extension_set` is a text document whose
``content`` is the raw XML — and because the text digest of an ASCII
file equals the digest of its bytes, its ``file_hash`` (and even its
``content_hash``) matches perfectly, so a hash-only fast path would
freeze it as XML forever: no bytes route, no preview, no
``image_url`` citation figure — the exact complaint this phase
answers. Falling through runs the image branch, which rewrites the
row (``is_image``/``image_path``/extracted ``content`` + the
``image_dir`` copy) through the normal index write: no alembic
migration, no backfill script, and the next sync is fast again.
The REVERSE direction is deliberately NOT forced: with the toggle
OFF the walk never admits an image file (so an ``is_image`` row never
reaches :func:`_index_file` at all) and the phase-122 prune guard
keeps the row; requiring ``settings.images`` here keeps that
contract byte-identical (``svg`` in ``BOR_IMPORT_EXTENSIONS`` with
the toggle off still means "text document", LOCKED A7).
"""
if not settings.images or doc.is_image:
return False
return match_extension(path, settings.image_extension_set) is not None
async def _index_file(
session: Session,
*,
@@ -1400,10 +1456,17 @@ async def _index_file(
# fast path (and the branches' unchanged path) refresh the
# stored date from the same source.
raw_date = file_mtime_datetime(full_path)
if doc is not None and doc.file_hash == file_digest:
if (
doc is not None
and doc.file_hash == file_digest
and not _row_kind_mismatch(full_path, doc, settings)
):
# The fast path: the raw bytes are what was indexed last time
# — the bookkeeping (backfill / D1 / D4) runs in the shared
# helper, nothing else does.
# helper, nothing else does. Phase 181 (task 02, D4): "the same
# bytes" is only "the same document" when the row's KIND also
# matches how the file would be indexed NOW — see
# :func:`_row_kind_mismatch` (the SVG text→image migration).
await _unchanged_doc(
session, doc, source=source, rel=rel, llm=llm, summary=summary,
raw_date=raw_date, file_digest=file_digest, full_path=full_path,
@@ -3542,7 +3605,17 @@ async def _index_image_file(
on_chunk: Callable[[int, int], None] | None = None,
) -> None:
"""The phase-122 image branch of :func:`_index_file` — a standalone
image is indexed from its BYTES, never its text:
image is indexed from its BYTES, never its text.
Phase 181 (task 02, LOCKED A1) makes the branch RASTER **or**
VECTOR: an ``.svg`` takes this same path (digest, copy, ``is_image``,
``image_path``, chunking, ``summary = content``, the phase-125
``image_context`` enrichment) but its ``content`` comes from
:func:`app.rag.svg_text.extract_svg_text` instead of the vision
seam, and a success counts the log-only ``svg_indexed``. The bullets
and the fail-soft below are shared by both kinds — the only
difference is where the text comes from.
* the sha256 digest is over the raw bytes (the digest rule is
content identity — the same bytes are the same document);
@@ -3600,13 +3673,20 @@ async def _index_image_file(
digest = hashlib.sha256(data).hexdigest()
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
if doc is not None and doc.content_hash == digest:
if doc is not None and doc.is_image and doc.content_hash == digest:
# Phase 147: the shared unchanged path — a pre-phase NULL-hash
# row, or a bytes-changed/content-same image (the byte digest
# still matches). The helper stamps the new file hash (the
# next sync takes the fast path); for an image the summary is
# the stored description (``doc.content``), so the image-aware
# ``_store_summary`` backfill re-stores it verbatim.
# Phase 181 (task 02, D4): ``doc.is_image`` joins the guard — a
# NON-image row routed here by :func:`_row_kind_mismatch` is the
# migration case, not an unchanged one. An ASCII SVG indexed as
# TEXT digests to the SAME value here (the text path hashes
# ``content.encode()``, which for a byte-clean file equals the
# raw-bytes digest), so without it the text→image conversion
# would report ``unchanged`` forever and never produce the copy.
await _unchanged_doc(
session, doc, source=source, rel=rel, llm=llm, summary=summary,
raw_date=raw_date, file_digest=file_digest, full_path=full_path,
@@ -3629,16 +3709,50 @@ async def _index_image_file(
# it is generated BEFORE anything is written, so a failure skips the
# doc with no row and no copy (the copy is only made after a
# successful description — a failure never leaves an orphan).
content = await _describe_or_skip(
llm, data=data, source=source, rel=rel, full_path=full_path,
context=image_context,
)
#
# Phase 181 (task 02, LOCKED A1): the VECTOR branch. An ``.svg`` is
# described by NOBODY — OpenAI-compatible vision endpoints reject
# ``image/svg+xml`` and no rasteriser dependency is in scope — so
# its text comes out of the file itself (its ``<title>``/``<desc>``/
# ``<text>`` words, or LOCKED A2's structural line). Everything else
# in this function is shared: the byte digest, the ``image_dir``
# copy, ``is_image``/``image_path``, the title, the counters, the
# phase-125 ``image_context`` prepend below, the chunk pipeline and
# ``summary = content``. The seam below is not reached, so no vision
# call happens for a vector file (pinned by the unit tests).
is_vector = full_path.suffix.lower() == ".svg"
svg: SvgText | None = None
if is_vector:
svg = extract_svg_text(data, stem=full_path.stem)
content = None if svg is None else svg.text
else:
content = await _describe_or_skip(
llm, data=data, source=source, rel=rel, full_path=full_path,
context=image_context,
)
if content is None:
# LOCKED A3 fail-soft: an undescribed image is unsearchable
# noise — skip the doc entirely (no row, no copy).
# noise — skip the doc entirely (no row, no copy). Phase 181
# (LOCKED A2): the vector case skips ONLY when the extractor
# refused the file (over the size cap, DOCTYPE/entity-bearing,
# malformed XML) — a textless SVG is NOT a failure, it gets the
# structural line.
summary.images_failed += 1
logger.warning("import: image description failed source=%s path=%s", source, rel)
logger.warning(
"import: %s source=%s path=%s",
"svg text extraction failed" if is_vector else "image description failed",
source, rel,
)
return
if svg is not None:
# Phase 181 (task 02): the log-only vector counter + the
# auditability line (AGENTS rule 10) — ``structural=True`` marks
# the docs whose text is the LOCKED A2 fallback line.
summary.svg_indexed += 1
logger.info(
"import: svg indexed source=%s path=%s structural=%s chars=%d",
source, rel, svg.structural, len(svg.text),
)
if image_context is not None:
# Phase 125 (task 03): the enriched content — the reference
# context FIRST (for retrieval), then the description. The
app/rag/summarizer.py
+12
View File
@@ -149,6 +149,17 @@ PICTORIAL_PROMPT = f"{PICTORIAL_PAGE_MODE}: {PICTORIAL_INSTRUCTION}"
#: takes the :data:`IMAGE_FALLBACK_MIME` data-URL mime in the describe
#: call — the vision endpoint may reject it, and the fail-soft skip
#: (``images_failed``) is the honest outcome.
#:
#: Phase 181 adds ``.svg`` — the ONE **vector** entry in the map. It
#: exists for the BYTES route only: the vector import branch
#: (``app/rag/importer.py::_index_image_file``) takes
#: :func:`app.rag.svg_text.extract_svg_text` INSTEAD of the vision
#: describe call (LOCKED A1), so an ``.svg`` never reaches
#: :func:`image_data_url` and this value is never put in a multimodal
#: message. ``app/api/docs.py::get_document_image`` reads the same map
#: for the ``Content-Type`` (one map, one truth), and that route is
#: where the vector value earns its place — plus the sandboxed CSP the
#: vector bytes need and the rasters do not (LOCKED A3, phase 181).
IMAGE_MIMES: dict[str, str] = {
".png": "image/png",
".jpg": "image/jpeg",
@@ -156,6 +167,7 @@ IMAGE_MIMES: dict[str, str] = {
".webp": "image/webp",
".gif": "image/gif",
".bmp": "image/bmp",
".svg": "image/svg+xml",
}
#: The data-URL mime for an image extension :data:`IMAGE_MIMES` does not
app/rag/svg_text.py
+293
View File
@@ -0,0 +1,293 @@
"""SVG text extraction — the words a vector drawing carries (phase 181).
Pure functions, no I/O — fully unit-testable. Stdlib only (the
:mod:`app.rag.md_images` precedent, phase 125).
An ``.svg`` joins the image family (LOCKED A1, phase 181): its BYTES are
served to the browser so every citation surface paints the graphic
instead of XML. But OpenAI-compatible vision endpoints reject
``image/svg+xml``, and adding a rasteriser dependency is out of scope —
so the vector branch never calls the describer. The text that makes the
document **retrievable** comes from the file itself:
* ``<title>``, ``<desc>`` and ``<text>`` (LOCKED D2) — collected in
DOCUMENT order by LOCAL name (namespace-agnostic: ``<title>`` and
``{http://www.w3.org/2000/svg}title`` both count, declared ``svg:``
prefixes included), whitespace-normalised, de-duplicated
case-insensitively keeping the FIRST occurrence, joined with ``\\n``.
``<tspan>`` children are joined WITH a space so ``a<tspan>b</tspan>``
reads "a b", not "ab". XML entity references inside the text
(``&amp;``) are decoded by the parser — the correct direction, the
renderer re-escapes on display.
* a textless diagram (pure ``<path>``s — extremely common) still gets
honest, searchable text (LOCKED A2): one structural line built ONLY
from facts the file states, never invented prose.
Guards, applied BEFORE any parse (untrusted markup, the sibling of
LOCKED A3):
* empty / whitespace-only input → ``None``;
* more than :data:`MAX_SVG_BYTES` → ``None`` (memory guard);
* a ``<!DOCTYPE`` or ``<!ENTITY`` declaration anywhere in the bytes →
``None``. Refusing the doctype outright is what makes stdlib
:mod:`xml.etree.ElementTree` safe here: no external entities, no
entity expansion (the billion-laughs class);
* malformed XML (:class:`xml.etree.ElementTree.ParseError`) → ``None``
— never re-raised, the caller's fail-soft is a skip, not a crash.
What is deliberately NOT followed: ``<image>`` elements embedding a
raster inside the SVG (their ``href`` is not text), and
``aria-label``/``role`` attributes — ``<title>``/``<desc>`` are the
SVG-spec carriers of words and the fallback line covers the rest.
"""
from __future__ import annotations
import math
import re
from dataclasses import dataclass
from xml.etree import ElementTree
#: Hard input cap for one SVG (phase 181, LOCKED A3's sibling) — bytes
#: over this are refused without a parse attempt, so a hostile or
#: accidentally gigantic file can never be the memory event of a sync.
MAX_SVG_BYTES: int = 2 * 1024 * 1024
#: Cap on the EXTRACTED text (phase 181). It binds on the result, never
#: silently mid-word: an over-cap extraction is cut at the last newline
#: or period before the cap (a hard cut only when the window holds no
#: boundary at all).
MAX_SVG_TEXT_CHARS: int = 4000
#: A document type declaration in the raw bytes (lower-cased marker).
#: Its presence — anywhere — refuses the input (billion-laughs /
#: external-entity class).
_DOCTYPE_MARKER = b"<!doctype"
#: An entity declaration in the raw bytes (lower-cased marker) — same
#: refusal as :data:`_DOCTYPE_MARKER`, checked separately so a stray
#: ``<!ENTITY`` inside a stray doctype-less prolog is still caught.
_ENTITY_MARKER = b"<!entity"
#: Elements whose whole text block is one item, nested markup
#: concatenated (:meth:`ElementTree.Element.itertext`).
_BLOCK_TEXT_NAMES = frozenset({"title", "desc"})
#: Elements whose text is joined with a space so inline children
#: (``<tspan>``) do not weld their neighbours together.
_SPACED_TEXT_NAMES = frozenset({"text"})
#: Local names that are structure, not drawing: never counted as shapes
#: in the fallback line (:func:`_shape_clause`).
_NON_SHAPE_NAMES = frozenset({"svg", "title", "desc"})
#: How many element kinds the fallback line names (LOCKED D3: "the
#: three most common element local names with counts").
_TOP_SHAPE_NAMES = 3
#: ``viewBox`` components: four numbers separated by whitespace and/or
#: commas (the SVG-spec grammar, which allows either).
_VIEWBOX_SPLIT_RE = re.compile(r"[\s,]+")
#: The multiplication sign used between a shape name and its count, and
#: between the viewBox width and height — the app's own typography.
_TIMES = "×"
@dataclass(frozen=True)
class SvgText:
"""The searchable text of one SVG document (phase 181, LOCKED A1/A2).
* ``text`` — what gets stored as the document's ``content`` (and
``summary``, per the phase-122 image convention);
* ``structural`` — ``True`` when the document carried no words and
``text`` is the structural fallback line, so the importer can log
and count it separately from a genuinely worded diagram.
"""
text: str
structural: bool
def local_name(tag: object) -> str:
"""The namespace-agnostic name of an element tag.
``{http://www.w3.org/2000/svg}title`` and ``title`` both give
``"title"`` — ElementTree expands every declared prefix (including
``<svg:title>``) into Clark notation, so stripping the ``{uri}`` is
all "namespace-agnostic" needs. A non-string tag (comment /
processing-instruction marker) has no name → ``""``.
"""
if not isinstance(tag, str):
return ""
return tag.rsplit("}", 1)[-1]
def extract_svg_text(data: bytes, *, stem: str = "") -> SvgText | None:
"""Turn raw SVG bytes into the document's searchable text.
Returns ``None`` when the input is unusable or unsafe: empty /
whitespace-only, larger than :data:`MAX_SVG_BYTES`, carrying a
``<!DOCTYPE``/``<!ENTITY`` declaration, or not well-formed XML. The
caller's fail-soft (skip the document, count it) is the honest
outcome — this function never raises for bad input.
*stem* is the file name without extension: it names the document in
the structural fallback line and is never invented.
Otherwise a :class:`SvgText` whose ``text`` is the document-order
``<title>``/``<desc>``/``<text>`` words (``structural=False``), or —
when the file carries no words at all — the structural line built
from the stem, the shape census and the ``viewBox``
(``structural=True``, LOCKED A2).
"""
if not data or not data.strip():
return None
if len(data) > MAX_SVG_BYTES:
return None
lowered = data.lower()
if _DOCTYPE_MARKER in lowered or _ENTITY_MARKER in lowered:
return None
try:
root = ElementTree.fromstring(data)
except ElementTree.ParseError:
return None
items = _dedupe(_walk_text(root))
if items:
return SvgText(text=_cap_text("\n".join(items)), structural=False)
return SvgText(text=_structural_line(root, stem), structural=True)
def _walk_text(root: ElementTree.Element) -> list[str]:
"""Every worded ``title``/``desc``/``text`` block, in document order.
Items are whitespace-normalised; a block that normalises to nothing
(an empty ``<title></title>``) is dropped so the fallback can still
describe the file.
"""
items: list[str] = []
for el in root.iter():
name = local_name(el.tag)
if name in _BLOCK_TEXT_NAMES:
raw = "".join(el.itertext())
elif name in _SPACED_TEXT_NAMES:
raw = " ".join(el.itertext())
else:
continue
normalised = " ".join(raw.split())
if normalised:
items.append(normalised)
return items
def _dedupe(items: list[str]) -> list[str]:
"""Drop repeats case-insensitively, keeping the FIRST occurrence.
A ``<title>`` mirrored in a ``<desc>`` (exporters do this) is one
sentence in the index, not two; the first position in document
order is the one kept.
"""
seen: set[str] = set()
unique: list[str] = []
for item in items:
key = item.casefold()
if key in seen:
continue
seen.add(key)
unique.append(item)
return unique
def _cap_text(text: str) -> str:
"""Cap *text* at :data:`MAX_SVG_TEXT_CHARS` without cutting a word.
An over-cap text is cut at the LAST newline or period before the cap
(whichever comes later) so the stored text ends on a boundary; only
a window with no boundary at all (one colossal run of characters)
is hard-cut.
"""
if len(text) <= MAX_SVG_TEXT_CHARS:
return text
window = text[:MAX_SVG_TEXT_CHARS]
cut = max(window.rfind("\n"), window.rfind("."))
if cut < 0:
return window
return text[: cut + 1].rstrip()
def _structural_line(root: ElementTree.Element, stem: str) -> str:
"""The LOCKED A2 fallback: one line of facts, nothing invented.
``Vector diagram "<stem>" — <N> shapes (<top-3 kinds>), viewBox
<w>×<h>`` — each fact clause is omitted when the file does not
state it, and the stem clause disappears entirely when empty (no
empty quotes). With no facts and no stem the line is just
``Vector diagram``.
"""
head = f'Vector diagram "{stem}"' if stem else "Vector diagram"
facts = [c for c in (_shape_clause(root), _viewbox_clause(root)) if c]
if not facts:
return head
return f"{head} — {', '.join(facts)}"
def _shape_clause(root: ElementTree.Element) -> str:
"""``<N> shapes (<kinds>)`` — the drawing's element census.
*N* counts every descendant element except ``svg``, ``title`` and
``desc`` (structure, not drawing). The kinds are the
:data:`_TOP_SHAPE_NAMES` most common local names, ordered by count
(document order breaks ties) and annotated with their count — the
count is left off a kind that appears exactly once, which is what
the total already says. ``""`` when there are no shapes at all, so
the clause is omitted rather than claiming "0 shapes".
"""
counts: dict[str, int] = {}
first_seen: dict[str, int] = {}
total = 0
for index, el in enumerate(root.iter()):
if el is root:
continue
name = local_name(el.tag)
if not name or name in _NON_SHAPE_NAMES:
continue
counts[name] = counts.get(name, 0) + 1
first_seen.setdefault(name, index)
total += 1
if not total:
return ""
top = sorted(counts, key=lambda name: (-counts[name], first_seen[name]))
listing = ", ".join(
f"{name}{_TIMES}{counts[name]}" if counts[name] > 1 else name
for name in top[:_TOP_SHAPE_NAMES]
)
noun = "shape" if total == 1 else "shapes"
return f"{total} {noun} ({listing})"
def _viewbox_clause(root: ElementTree.Element) -> str:
"""``viewBox <w>×<h>`` from the root's ``viewBox``, or ``""``.
The spec grammar is four whitespace-and/or-comma separated numbers;
the last two are the width and height. Unparseable (wrong count,
non-numeric, non-finite) → ``""`` — the clause is omitted rather
than reporting a size the file does not state. Integral values
print without a trailing ``.0``.
"""
raw = (root.get("viewBox") or "").strip()
if not raw:
return ""
parts = [part for part in _VIEWBOX_SPLIT_RE.split(raw) if part]
if len(parts) != 4:
return ""
try:
_min_x, _min_y, width, height = (float(part) for part in parts)
except ValueError:
return ""
if not (math.isfinite(width) and math.isfinite(height)):
return ""
return f"viewBox {_format_number(width)}{_TIMES}{_format_number(height)}"
def _format_number(value: float) -> str:
"""``8`` for 8.0, ``8.5`` for 8.5 — no trailing ``.0`` in a size."""
return str(int(value)) if value.is_integer() else str(value)
frontend/assets/app.js
+11 -1
View File
@@ -2498,7 +2498,17 @@ function appendRelated(wrap, related) {
* replaced, in place, with a plain `.source-chip` anchor for the same
* document — the citation survives a lost byte copy, never a
* broken-image icon (the row's bytes route 404s when the copy was
* lost). */
* lost).
*
* Phase 181 (task 04, design D7): DELIBERATE NON-CHANGE for SVG. The
* figure keys off image_url alone, and a size-less (viewBox-only)
* vector drawing scales INSIDE the fixed cap of .source-image-img
* (max-height: 96px + max-width: 100% + object-fit: contain), so the
* compact chat citation needs no format-aware sizing; the full-size
* preview is the modal this figure/chip opens (doc.format is known
* there — see docImageBlock in document.js). NO format field is
* added to SourceRef — the wire contract stays byte-identical for
* every text doc (the phase-83/122 contract). */
function appendSourceImageFigure(meta, s) {
const label = `${s.source}/${s.path}`;
const fig = document.createElement("a");
frontend/assets/document.js
+20 -1
View File
@@ -573,7 +573,26 @@ function docImageBlock(doc) {
const wrap = document.createElement("div");
wrap.className = "doc-image";
const img = document.createElement("img");
img.className = "doc-image-img";
// Phase 181 (task 04, design D6/D7): SVG is a VECTOR image doc. A
// drawing carrying only a viewBox has NO intrinsic size, so the
// default replaced-element box (300×150) would sit small inside the
// full-width .doc-image card; the --vector modifier (the base class
// is KEPT so every current rule still applies) lets width: 100%
// span the card — the full-size preview. doc.format is the ONLY
// place in the app that knows the type: the bytes URL is
// extension-less and SourceRef carries no format field by design
// (D7). LOCKED A3: the SVG is NEVER inlined into the DOM —
// img.src = doc.image_url stays createElement + property
// assignment (this file's XSS contract, unchanged). Legacy
// honesty (D6): a row still indexed as TEXT (an .svg under
// BOR_IMAGES=0, or not yet re-synced) keeps rendering its XML in
// .doc-raw — the truth about that row (there are no bytes behind
// it); task 02's kind-mismatch guard converts it on the next sync.
// Nobody "fixes" that by sniffing the content string.
img.className =
doc.format === "svg"
? "doc-image-img doc-image-img--vector"
: "doc-image-img";
img.src = doc.image_url;
img.alt =
typeof doc.summary === "string" && doc.summary.trim() !== ""
frontend/assets/sources.js
+6 -1
View File
@@ -2324,7 +2324,12 @@ export async function mount(root) {
// Phase 122 (task 04): an image row gets the FIXED 48px thumbnail
// box before the path link (the .kb-doc-path flex wrapper — the
// link keeps its ellipsis). Text rows keep the bare-link cell,
// byte-identical to pre-phase (no box at all).
// byte-identical to pre-phase (no box at all). Phase 181 (task
// 04, design D7): verified NON-CHANGE for SVG rows — the thumb
// loads the same extension-less image_url and .kb-doc-thumb-img
// pins the img to the fixed 48px box (width/height 100% +
// object-fit: cover), so a size-less viewBox-only vector fills
// the box like a raster; no format sniffing needed here either.
if (d.is_image) {
const pathWrap = document.createElement("div");
pathWrap.className = "kb-doc-path";
frontend/assets/styles.css
+17
View File
@@ -5742,6 +5742,23 @@ a.answer-image:hover { text-decoration: underline; } /* flat — the caption und
border-radius: var(--radius-sm);
background: var(--bg);
}
/* The vector variant (phase 181, task 04, design D6 — applied in
document.js when doc.format === "svg", base class kept). An SVG
carrying only a viewBox has NO intrinsic size, so without an
explicit width the default replaced-element box (300×150) applies
and the drawing sits small inside the full-width .doc-image card;
width: 100% spans the card (the full-size preview the owner
asked for). object-fit: contain protects the drawing from being
squashed when the base rule's max-height: 70vh clamps a tall
vector — the box letterboxes, the graphic keeps its aspect ratio.
Static content — no animation (nothing for prefers-reduced-motion
to still). Palette-neutral: no colour here — the base rule's --bg
shows through (phase-08 tokens only, no new hue). */
.doc-image-img--vector {
width: 100%;
height: auto;
object-fit: contain;
}
/* The onerror fallback (the route's 404 — the row exists but the
copy was lost): a muted note in the card's place-of-image; the
description below still renders, so the doc stays readable. Phase
tests/e2e/test_svg_document_preview.py
+607
View File
@@ -0,0 +1,607 @@
"""Phase 181 E2E (Playwright): SVG documents preview as graphics, not XML.
Source: ``TODO.md`` L3 — "Need SVG render support on the front end when
citing and svg source. Also would be nice to click an svg and see a
preview in the RAG tab rather than the xml".
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov
The suite's OWN module app (the phase-122 ``test_image_documents.py``
pattern) runs with ``BOR_IMAGES=true`` and every suite-writable home
(``BOR_IMAGE_DIR`` included) pinned into the suite's tmp dir — process
env ranks above an operator's local gitignored ``.env``, so the toggle
cannot leak in either direction. Seeding is the phase-138 pattern
(``test_markdown_blockquotes`` / ``test_markdown_inline_links_images``):
truncate, then run the REAL importer in-process against a COPY of the
phase's dedicated fixture tree ``tests/fixtures/svg_docs/`` (the shared
fixture set's counts are pinned by other suites and must not move) with
the deterministic mock embeddings — the vector branch of task 02
indexes both SVGs with NO vision call, the PNG half of the last test is
described by the mock's ``IMAGE_DESCRIPTION`` branch.
Test → story mapping (Playwright Mapping Rule):
1. ``test_viewer_renders_svg_not_xml`` — the Sources row for
``diagrams/topology.svg`` opens the modal showing
``img.doc-image-img--vector`` (``naturalWidth > 0``, non-empty alt),
NO ``pre.doc-raw`` and no ``<svg`` XML text anywhere in the modal
body, the extracted ``<title>`` text below the graphic; the "Full
page" escape hatch renders the same graphic at ≥ 600px width at a
1280px viewport (the task-04 vector sizing contract — a
``viewBox``-only SVG would otherwise sit in the 300×150 default box).
2. ``test_chat_figure_and_modal_preview`` — the grounded scripted-read
question cites the SVG doc as the ``.source-image`` figure (the
``<img>`` loads, the caption is the extracted text), and clicking
the figure opens the modal with the full graphic (the "preview in
the RAG tab" contract); Escape closes it.
3. ``test_blank_svg_is_indexed_and_searchable`` — ``diagrams/blank.svg``
(pure paths) exists as an image document whose content is the
structural LOCKED A2 line ``Vector diagram "blank" — …``, is listed
on the Sources page with its thumbnail, and opens in the modal with
that line as its description.
4. ``test_markdown_image_ref_renders`` — the two-stage seeding: import
the tree, read the SVG doc's id from the DB, write
``notes/inline.md`` containing ``![topology](/api/documents/<id>/image)``
and re-import; the viewer for that document shows ``img.md-img``
pointing at the bytes route and LOADING (``naturalWidth > 0``) — the
phase-180 renderer + this route end to end.
5. ``test_raster_surfaces_unchanged`` — the smoke guard: a PNG doc in
the same tree still renders through ``.doc-image-img`` (no
``--vector`` class) and its bytes route carries NO sandbox CSP (the
app-wide policy only) — phase 122/124 behaviour byte-unchanged.
"""
from __future__ import annotations
import asyncio
import base64
import json
import os
import shutil
import subprocess
import sys
from collections.abc import Iterator
from pathlib import Path
from threading import Thread
from typing import Any
import httpx
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import select, text
from app.config import Settings
from app.db import SessionLocal
from app.models import Chunk, Document
from app.rag.agent import IMAGE_DOC_MARKER
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from app.rag.svg_text import extract_svg_text
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
if USE_REAL_LLM:
pytest.skip(
"phase 181 pins the deterministic mock's grounded turn and seeded "
"figures — the real model answers differently",
allow_module_level=True,
)
REPO = Path(__file__).resolve().parents[2]
FIXTURES = REPO / "tests" / "fixtures" / "svg_docs"
#: The source name = the imported tree root's basename (the importer's
#: ``archive_source_name``/directory rule — the phase-180 pattern).
SOURCE = "svg_docs"
TOPO_PATH = "diagrams/topology.svg"
BLANK_PATH = "diagrams/blank.svg"
#: The fixture diagram's own words — what the vector branch indexes
#: (LOCKED A1) and what every text assertion below pins. Pinned as a
#: literal here (not re-derived from the extractor) so a change to the
#: extraction contract breaks this suite LOUDLY, next to the product.
TOPO_TEXT = (
"Homelab network topology\n"
"Wiring diagram of the homelab: the edge router uplinks two managed "
"switches, the hypervisor rack and the nas storage shelf.\n"
"rack-1 switch\n"
"nas storage"
)
#: The grounded scripted-read question (phase-118 A1: a zero-read turn
#: chips nothing — the mock emits the scripted ``read`` then echoes the
#: result, the phase-122 figure precedent). Token-overlap cosines
#: against the mock embeddings: topology.svg ≈0.70, the infra.md text
#: co-citation ≈0.72 (both over the mock-calibrated 0.30 gate); the
#: blank.svg line sits at ≈0.04 and never pollutes the citation set.
QUESTION = (
"Read the suggested document: read svg_docs/diagrams/topology.svg — "
"what does the homelab network topology diagram show about the edge "
"router, rack-1 switch and the nas storage?"
)
#: A real 1×1 PNG (the phase-122/124 suites' bytes) — the raster smoke
#: guard of the last test, written into the suite's tree copy only there.
PNG_1X1 = base64.b64decode(
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
)
# ---------------------------------------------------------------------------
# Module app: BOR_IMAGES pinned true, the image dir in the suite's scratch
# (the in-process importer and the app under test SHARE that directory —
# the copies the seed writes are the bytes the route serves).
# ---------------------------------------------------------------------------
def _app_env(mock_llm: int, port: int, scratch: Path) -> dict[str, str]:
"""The conftest app env (leak guards included) with the phase-181
knobs: ``BOR_IMAGES=true`` EXPLICITLY (an operator's local ``.env``
cannot leak the toggle off) and ``BOR_IMAGE_DIR`` pinned into the
suite's scratch (the app must never read or write the owner's real
image home)."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = f"http://127.0.0.1:{mock_llm}/v1"
# Mock-calibrated gate (conftest pattern) — QUESTION grounds on these.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
env["BOR_LLM_RETRY_DELAY"] = "0"
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# Leak guards (conftest pattern).
env["BOR_GIT_SOURCES"] = ""
env["BOR_DOCS_REPO"] = ""
env["BOR_SUGGESTIONS"] = json.dumps(
Settings.model_fields["suggestions"].default
)
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
# Phase 181: the toggle under test + the suite-private homes.
env["BOR_IMAGES"] = "true"
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
env["BOR_IMAGE_DIR"] = str(_image_dir(scratch))
return env
def _image_dir(scratch: Path) -> Path:
return scratch / "images"
@pytest.fixture(scope="module")
def scratch(tmp_path_factory: pytest.TempPathFactory) -> Path:
return tmp_path_factory.mktemp("bor_svg_preview")
@pytest.fixture(scope="module")
def app_server(
mock_llm: int, scratch: Path
) -> Iterator[str]:
"""The suite's own app — ``BOR_IMAGES=true`` (the conftest session app
is never started in this isolated run, so no port clash)."""
port = int(os.environ.get("E2E_APP_PORT_SVG_PREVIEW", "8196"))
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
cwd=REPO,
env=_app_env(mock_llm, port, scratch),
)
try:
_wait_http(f"http://127.0.0.1:{port}/api/health")
yield f"http://127.0.0.1:{port}"
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
# ---------------------------------------------------------------------------
# Seeding: the phase-138 truncate-then-import pattern against a per-test
# COPY of the dedicated fixture tree (the two-stage test appends a file to
# the copy, never to the tracked fixture).
# ---------------------------------------------------------------------------
def _fresh_tree(scratch: Path) -> Path:
"""A pristine copy of the fixture tree at ``<scratch>/src/svg_docs``
(the basename IS the source name). Rmtree-first: the two-stage test
leaves an ``inline.md`` behind, and every other test must not see it."""
tree = scratch / "src" / SOURCE
if tree.exists():
shutil.rmtree(tree)
tree.parent.mkdir(parents=True, exist_ok=True)
shutil.copytree(FIXTURES, tree)
return tree
async def _import(tree: Path, mock_llm: int, scratch: Path) -> ImportSummary:
"""The REAL importer over the fixture copy (task 02's vector branch
runs in-process against the mock — the same LLM the app under test
talks to). A coroutine: the caller runs it through
:func:`_run_in_thread` (Playwright owns the test thread's loop)."""
settings = Settings(
# the owner's .env cannot leak the vector knobs (_env_file is a
# runtime-only pydantic-settings kwarg — pyright cannot see it).
_env_file=None, # pyright: ignore[reportCallIssue]
llm_base_url=f"http://127.0.0.1:{mock_llm}/v1",
images=True, # the toggle under test (LOCKED A7's on state)
image_dir=str(_image_dir(scratch)),
image_extensions=Settings.model_fields["image_extensions"].default,
)
return await import_sources([tree], LLMClient(settings))
def _run_in_thread(coro: Any) -> Any:
"""Run a coroutine on a worker thread — Playwright's sync API keeps an
asyncio loop on the test thread (the house helper, phase 180)."""
box: dict[str, Any] = {}
def runner() -> None:
try:
box["value"] = asyncio.run(coro)
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
box["error"] = e
t = Thread(target=runner)
t.start()
t.join()
if "error" in box:
raise box["error"]
return box["value"]
def _truncate() -> None:
"""Fresh KB per test (the suites share one Postgres). ``sources_meta``
resets with the KB — the re-import's bump UPSERTS the row back
(the phase-122 truncate list, ``bump_sources_version``)."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, folder_summaries, sources_meta"
)
)
db.commit()
def _seed(mock_llm: int, scratch: Path) -> ImportSummary:
"""Truncate + import the fresh fixture copy. Both SVGs arrive through
the VECTOR branch (``svg_indexed == 2`` — no vision call happened),
infra.md as the text co-citation: three docs total."""
_truncate()
summary = _run_in_thread(_import(_fresh_tree(scratch), mock_llm, scratch))
assert (summary.files, summary.added, summary.images_failed) == (3, 3, 0)
assert summary.svg_indexed == 2, summary
return summary
def _doc(source: str, path: str) -> Document:
with SessionLocal() as db:
doc = db.scalar(
select(Document).where(Document.source == source, Document.path == path)
)
assert doc is not None, f"no documents row for ({source!r}, {path!r})"
return doc
def _health_db(app_url: str) -> bool:
try:
return httpx.get(f"{app_url}/api/health", timeout=5).json()["db"] == "up"
except Exception: # noqa: BLE001 — unreachable is the skip case
return False
def _skip_without_db(app_url: str) -> None:
if not _health_db(app_url):
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
def _drill(page: Page, name: str) -> None:
"""One level of the KB tree (the phase-122/180 click, made settle-
safe): the drill keeps rendering the CURRENT level while the node
fetch runs (phase 167), so the level is settled exactly when the
clicked link is gone from the replaced table (intermediate levels
with no direct files must not wait on an empty ``#docs-tbody``)."""
link = page.locator(f'#folders-tbody a.folder-link:text-is("{name}")')
expect(link).to_be_visible(timeout=30_000)
link.click()
expect(link).to_have_count(0)
def _open_doc_modal(page: Page, app_url: str, folder: str, doc_path: str) -> Any:
"""Sources → source → folder → the document's row → the same-page
modal. Returns the modal locator (loaded)."""
login(page, app_url) # lands on /sources.html (admin)
_drill(page, SOURCE)
_drill(page, folder)
row = page.locator("#docs-tbody tr", has_text=doc_path)
expect(row).to_have_count(1)
row.locator("td:nth-child(2) a.doc-link").click()
modal = page.locator("#doc-modal")
expect(modal).to_be_visible(timeout=30_000)
return modal
def _seed_page(page: Page, app_url: str, mock_llm: int, scratch: Path) -> None:
_skip_without_db(app_url)
page.set_default_timeout(30_000)
_seed(mock_llm, scratch)
# ---------------------------------------------------------------------------
# 1. The viewer: the SVG paints as a graphic — the XML is nowhere
# ---------------------------------------------------------------------------
def test_viewer_renders_svg_not_xml(
page: Page, app_url: str, mock_llm: int, scratch: Path
) -> None:
"""TODO.md L3's viewer half: the modal shows the RENDERED vector
graphic (the ``--vector`` sizing class, task 04) from the bytes
route, alt = the extracted text; NO raw-XML surface exists — no
``pre.doc-raw``, no ``<svg`` markup text anywhere in the modal body
— and the extracted ``<title>`` sentence reads as the description
below the graphic. The "Full page" escape hatch (the
``/document.html`` viewer) spans the drawing: ≥ 600px wide at a
1280px viewport, where the 300×150 replaced-element default box
would have sat small (the phase's sizing contract)."""
_seed_page(page, app_url, mock_llm, scratch)
doc = _doc(SOURCE, TOPO_PATH)
assert doc.is_image is True # the premise: a vector IMAGE row
modal = _open_doc_modal(page, app_url, "diagrams", TOPO_PATH)
expect(page.locator("#doc-modal-title")).to_have_text("topology")
img = modal.locator("img.doc-image-img--vector")
expect(img).to_have_count(1)
expect(img).to_be_visible(timeout=15_000)
assert img.evaluate("el => el.naturalWidth") > 0, "the SVG bytes did not paint"
assert img.evaluate("el => el.naturalHeight") > 0
alt = img.get_attribute("alt")
assert alt is not None and "Homelab network topology" in alt
assert img.get_attribute("src") == f"/api/documents/{doc.id}/image"
expect(modal.locator(".doc-image-unavailable")).to_have_count(0)
# The honest negative (the owner's complaint, verbatim): NO XML text
# and no raw-code surface for an SVG row.
expect(modal.locator("pre.doc-raw")).to_have_count(0)
body_text = modal.inner_text()
assert "<svg" not in body_text, "raw XML text painted into the modal"
assert "</title>" not in body_text
# The extracted words ARE the document's readable content, below
# the graphic (the phase-165 prose arm — summary === content).
prose = modal.locator(".doc-prose")
expect(prose).to_have_count(1)
expect(prose).to_contain_text("Homelab network topology")
expect(prose).to_contain_text("rack-1 switch")
# The Full page escape hatch: the same graphic, full column width.
href = page.locator("#doc-modal-open").get_attribute("href")
assert href is not None and "/document.html" in href
page.goto(app_url + href)
viewer_img = page.locator(".doc-image-img--vector")
expect(viewer_img).to_be_visible(timeout=15_000)
box = viewer_img.bounding_box()
assert box is not None and box["width"] >= 600, (
f"the viewBox-only SVG did not span the viewer: {box!r}"
)
# ---------------------------------------------------------------------------
# 2. The chat figure + the click-through preview in the RAG tab
# ---------------------------------------------------------------------------
def test_chat_figure_and_modal_preview(
page: Page, app_url: str, mock_llm: int, scratch: Path
) -> None:
"""TODO.md L3's citation half: the grounded question whose answer
cites the SVG doc (the scripted read echoes the read result — the
``IMAGE_DOC_MARKER`` line + the extracted words) shows the compact
``.source-image`` figure whose ``<img>`` LOADS from the bytes route
with the caption settling to the extracted text; clicking the
figure opens the SAME-page modal showing the full graphic (the
"click an svg … see a preview in the RAG tab rather than the xml"
contract) and Escape closes it."""
_seed_page(page, app_url, mock_llm, scratch)
doc = _doc(SOURCE, TOPO_PATH)
login(page, app_url, next="/")
expect(page.locator("#kb-banner")).to_be_hidden()
page.fill("#message-input", QUESTION)
page.click("#send-btn")
bubble = page.locator(".msg.brain .bubble").last
expect(bubble).to_contain_text(IMAGE_DOC_MARKER, timeout=30_000)
expect(bubble).to_contain_text("Homelab network topology", timeout=30_000)
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
# The citation figure (phase 130: the figure IS the citation).
# Scoped by the figure's aria-label (the source/path label) — the
# seed also carries blank.svg, and an image doc the seed surfaces
# is auto-cited too (phase 124); the ASSERTED figure is the
# topology doc's.
row = page.locator(".msg.brain .msg-meta.msg-images")
expect(row).to_have_count(1)
fig = row.locator(f'.source-image[aria-label="{SOURCE}/{TOPO_PATH}"]')
expect(fig).to_have_count(1)
fig_img = fig.locator(".source-image-img")
expect(fig_img).to_be_visible(timeout=15_000)
assert fig_img.evaluate("el => el.naturalWidth") > 0, (
"the chat figure did not load the SVG bytes"
)
assert fig_img.get_attribute("src") == f"/api/documents/{doc.id}/image"
caption = fig.locator(".source-image-caption")
expect(caption).to_contain_text("Homelab network topology", timeout=15_000)
# The click → modal preview: the full graphic, no XML, and Escape.
fig.click()
modal = page.locator("#doc-modal")
expect(modal).to_be_visible(timeout=30_000)
modal_img = modal.locator("img.doc-image-img--vector")
expect(modal_img).to_have_count(1)
expect(modal_img).to_be_visible(timeout=15_000)
assert modal_img.evaluate("el => el.naturalWidth") > 0
assert "<svg" not in modal.inner_text()
page.keyboard.press("Escape")
expect(modal).to_be_hidden()
# ---------------------------------------------------------------------------
# 3. The textless drawing: indexed with the structural line, previewable
# ---------------------------------------------------------------------------
def test_blank_svg_is_indexed_and_searchable(
page: Page, app_url: str, mock_llm: int, scratch: Path
) -> None:
"""LOCKED A2: ``diagrams/blank.svg`` carries NO words — pure paths —
yet it is NOT skipped: it exists as an image document whose indexed
text is the structural line built only from facts in the file
(``Vector diagram "blank" — <N> shapes (…), viewBox 640×480``), the
line is embedded (searchable), and the Sources page lists the doc
with its rendered thumbnail; the modal shows the graphic with that
line as the description below it."""
_seed_page(page, app_url, mock_llm, scratch)
doc = _doc(SOURCE, BLANK_PATH)
assert doc.is_image is True and doc.image_path is not None
assert doc.content.startswith('Vector diagram "blank" —')
assert "viewBox 640×480" in doc.content
assert doc.summary == doc.content
# Searchable: the structural line is an EMBEDDED chunk (both the
# content chunk and the is_summary mirror).
with SessionLocal() as db:
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc.id)).all()
assert [c.content for c in chunks] and all(
c.content == doc.content for c in chunks
)
assert all(c.embedding is not None and len(c.embedding) == 768 for c in chunks)
login(page, app_url)
_drill(page, SOURCE)
_drill(page, "diagrams")
row = page.locator("#docs-tbody tr", has_text=BLANK_PATH)
expect(row).to_have_count(1)
thumb = row.locator(".kb-doc-thumb-img")
expect(thumb).to_have_count(1)
expect(thumb).to_be_visible(timeout=15_000)
assert thumb.evaluate("el => el.naturalWidth") > 0, "the thumbnail is not painted"
# The modal: the graphic + the honest structural description (the
# file's only possible words).
row.locator("td:nth-child(2) a.doc-link").click()
modal = page.locator("#doc-modal")
expect(modal).to_be_visible(timeout=30_000)
expect(modal.locator("img.doc-image-img--vector")).to_have_count(1)
expect(modal.locator(".doc-prose")).to_contain_text('Vector diagram "blank"')
expect(modal.locator("pre.doc-raw")).to_have_count(0)
# ---------------------------------------------------------------------------
# 4. The phase-180 synergy: ![alt](/api/documents/<id>/image) paints
# ---------------------------------------------------------------------------
def test_markdown_image_ref_renders(
page: Page, app_url: str, mock_llm: int, scratch: Path
) -> None:
"""The two-stage seeding the phases' hand-off needs: import the tree,
read the SVG doc's id from the DB, THEN write ``notes/inline.md``
containing ``![topology](/api/documents/<id>/image)`` and re-import
(one doc added, the SVG row untouched — the fast path). The viewer
for that document renders ``img.md-img`` pointing at the bytes route
and LOADING it (``naturalWidth > 0``) — phase 180's allowlist
accepts the route shape, phase 181's route answers it, and the
reference never prints as text."""
_seed_page(page, app_url, mock_llm, scratch)
topo = _doc(SOURCE, TOPO_PATH)
image_url = f"/api/documents/{topo.id}/image"
assert extract_svg_text # the extractor produced the row's text (premise)
tree = scratch / "src" / SOURCE
(tree / "notes" / "inline.md").write_text(
"# Inline reference notes\n\n"
"The topology drawing, inline:\n\n"
f"![topology]({image_url})\n\n"
"Same route, inside a link: [the diagram](/sources.html).\n",
encoding="utf-8",
)
second = _run_in_thread(_import(tree, mock_llm, scratch))
assert second.added == 1, second
assert second.svg_indexed == 0, "the unchanged SVG must take the fast path"
modal = _open_doc_modal(page, app_url, "notes", "notes/inline.md")
md_img = modal.locator("img.md-img")
expect(md_img).to_have_count(1)
assert md_img.get_attribute("src") == image_url
assert md_img.get_attribute("alt") == "topology"
expect(md_img).to_be_visible(timeout=15_000)
assert md_img.evaluate("el => el.naturalWidth") > 0, (
"the bytes route did not answer the markdown reference"
)
# The reference renders — it never prints.
assert "![topology](" not in modal.inner_text()
# ---------------------------------------------------------------------------
# 5. The smoke guard: raster image docs are byte-unchanged
# ---------------------------------------------------------------------------
def test_raster_surfaces_unchanged(
page: Page, app_url: str, mock_llm: int, scratch: Path
) -> None:
"""Phase 122/124 behaviour survives the vector branch: a PNG doc in
the SAME tree is described through the vision seam (the mock's
fixed answer), renders through the plain ``.doc-image-img`` with NO
``--vector`` class, and its bytes route carries NO sandbox CSP — the
app-wide middleware policy only, the pre-phase header set."""
_seed_page(page, app_url, mock_llm, scratch)
tree = scratch / "src" / SOURCE
(tree / "diagrams" / "raster.png").write_bytes(PNG_1X1)
summary = _run_in_thread(_import(tree, mock_llm, scratch))
assert (summary.files, summary.added) == (4, 1)
assert summary.svg_indexed == 0
doc = _doc(SOURCE, "diagrams/raster.png")
assert doc.is_image is True
image_url = f"/api/documents/{doc.id}/image"
login(page, app_url)
_drill(page, SOURCE)
_drill(page, "diagrams")
row = page.locator("#docs-tbody tr", has_text="diagrams/raster.png")
expect(row).to_have_count(1)
row.locator("td:nth-child(2) a.doc-link").click()
modal = page.locator("#doc-modal")
expect(modal).to_be_visible(timeout=30_000)
img = modal.locator(".doc-image-img")
expect(img).to_have_count(1)
expect(img).to_be_visible(timeout=15_000)
# The raster class string is UNTOUCHED by the vector modifier.
assert (img.get_attribute("class") or "") == "doc-image-img"
expect(modal.locator("img.doc-image-img--vector")).to_have_count(0)
resp = page.request.get(app_url + image_url)
assert resp.status == 200
assert resp.headers["content-type"] == "image/png"
csp = resp.headers.get("content-security-policy", "")
assert "sandbox" not in csp, "the SVG sandbox policy leaked onto a raster route"
tests/fixtures/svg_docs/diagrams/blank.svg
Vendored
+18
View File
@@ -0,0 +1,18 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Phase 181 fixture: the TEXTLESS drawing (the LOCKED A2 class) — pure
paths and rects, no <title>, no <desc>, no <text>. The extractor's
structural fallback line is the only text this document can ever
carry, built from facts in the file: the stem, the shape census and
the viewBox. The phase's E2E pins the indexed content to
Vector diagram "blank" — … and the Sources thumbnail to the graphic.
-->
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 480">
<rect x="20" y="20" width="600" height="440" fill="none" stroke="#455a64" stroke-width="3"/>
<path d="M80 120 L240 120 L240 240 L80 240 Z" fill="#cfd8dc" stroke="#455a64" stroke-width="3"/>
<path d="M400 120 L560 120 L560 240 L400 240 Z" fill="#cfd8dc" stroke="#455a64" stroke-width="3"/>
<path d="M240 180 H400" stroke="#0288d1" stroke-width="4"/>
<path d="M320 240 V360" stroke="#0288d1" stroke-width="4"/>
<path d="M200 360 H440" stroke="#0288d1" stroke-width="4"/>
<rect x="280" y="340" width="80" height="60" fill="#eceff1" stroke="#455a64" stroke-width="3"/>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.1 KiB

tests/fixtures/svg_docs/diagrams/topology.svg
Vendored
+37
View File
@@ -0,0 +1,37 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Phase 181 fixture: a hand-written homelab wiring diagram.
Deliberate properties (the phase's E2E contract, tests/e2e/
test_svg_document_preview.py):
* a viewBox and NO width/height attributes — a vector drawing with no
intrinsic size, the exact class the phase-181 vector sizing class
(`doc-image-img` plus its `vector` modifier) exists for — the
300x150 default box would sit small inside the full-width card;
* its words live in <title>, <desc> and two <text> labels — what
app.rag.svg_text.extract_svg_text indexes (LOCKED A1, no vision call);
* a dozen plain <path>/<rect> shapes give the drawing real geometry
while the title/desc/text carry the searchable words.
-->
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 800">
<title>Homelab network topology</title>
<desc>Wiring diagram of the homelab: the edge router uplinks two managed switches, the hypervisor rack and the nas storage shelf.</desc>
<rect x="40" y="40" width="1120" height="720" fill="none" stroke="#37474f" stroke-width="4"/>
<rect x="120" y="96" width="240" height="120" rx="12" fill="#eceff1" stroke="#37474f" stroke-width="3"/>
<path d="M160 156 h160 M240 126 v60" stroke="#546e7a" stroke-width="5"/>
<rect x="480" y="96" width="240" height="120" rx="12" fill="#eceff1" stroke="#37474f" stroke-width="3"/>
<path d="M520 136 h160 M520 166 h160 M520 196 h160" stroke="#546e7a" stroke-width="4"/>
<rect x="840" y="96" width="240" height="120" rx="12" fill="#eceff1" stroke="#37474f" stroke-width="3"/>
<path d="M880 136 h160 M880 166 h160 M880 196 h160" stroke="#546e7a" stroke-width="4"/>
<path d="M360 156 C 420 156 420 156 480 156" fill="none" stroke="#0288d1" stroke-width="5"/>
<path d="M720 156 C 780 156 780 156 840 156" fill="none" stroke="#0288d1" stroke-width="5"/>
<path d="M600 216 C 600 300 600 340 600 400" fill="none" stroke="#0288d1" stroke-width="5"/>
<rect x="360" y="400" width="240" height="140" rx="12" fill="#e8f5e9" stroke="#2e7d32" stroke-width="3"/>
<path d="M400 440 h160 M400 470 h160 M400 500 h160" stroke="#2e7d32" stroke-width="4"/>
<rect x="680" y="400" width="240" height="140" rx="12" fill="#fff3e0" stroke="#ef6c00" stroke-width="3"/>
<path d="M720 440 h160 M720 470 h160 M720 500 h160" stroke="#ef6c00" stroke-width="4"/>
<path d="M480 540 C 480 620 600 620 600 620 M720 540 C 720 620 600 620 600 620" fill="none" stroke="#78909c" stroke-width="4"/>
<path d="M600 620 v100" fill="none" stroke="#78909c" stroke-width="4"/>
<text x="180" y="250" font-family="sans-serif" font-size="34" fill="#37474f">rack-1 switch</text>
<text x="700" y="590" font-family="sans-serif" font-size="34" fill="#ef6c00">nas storage</text>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 2.7 KiB

tests/fixtures/svg_docs/notes/infra.md
Vendored
+14
View File
@@ -0,0 +1,14 @@
# Homelab infrastructure notes
The wiring diagram `diagrams/topology.svg` — titled "Homelab network
topology" — is the current-state reference for the rack layout. When a
question is about the homelab network topology, that diagram is the
picture to open: it shows the edge router uplinking two managed
switches, the hypervisor rack and the nas storage shelf.
Uplink rules, restated in prose so the words are searchable next to the
diagram: the edge router hands one 10Gb link to rack-1 switch and one
to the second switch; the hypervisor VLANs live on rack-1 switch, and
nas storage sits behind the second switch on its own segment. The
diagram carries no other labels — anything else about the network
belongs in this note, not in the drawing.
tests/integration/test_svg_document_image.py
+443
View File
@@ -0,0 +1,443 @@
"""Integration tests: phase 181 (task 03) — the SVG half of the image
BYTES route ``GET /api/documents/{doc_id}/image``.
The contract this module pins (LOCKED A3):
* an ``is_image`` row whose stored copy is an ``.svg`` serves
``Content-Type: image/svg+xml`` (the ``app.rag.summarizer.IMAGE_MIMES``
map — one map, one truth with the describe call's data-URL mime) plus
ONE extra response header, ``Content-Security-Policy: default-src
'none'; sandbox``. An ``<img>`` can never run an SVG's script, but the
URL is DIRECTLY navigable, and the bytes come from an untrusted source
tree — so a direct visit must not hand the browser a script-capable
SAME-ORIGIN document. The route sets the policy itself and
``SecurityHeadersMiddleware`` PRESERVES a CSP an inner layer already
set, so this is what the browser sees; the middleware's global
``X-Content-Type-Options: nosniff`` and ``X-Frame-Options: DENY`` still
land on the response;
* the SVG bytes are served VERBATIM (no rewriting, no
``Content-Disposition: attachment`` — the graphic must render inline
and the policy is the control, not a sanitizer);
* every RASTER response is byte-identical to its pre-phase shape — the
same header KEY set, the app-wide CSP from the middleware (no
``sandbox``, no route-specific policy), the same ``Cache-Control`` and
the same exact bytes;
* the phase-122 posture is unchanged: PUBLIC (phase 124 LOCKED A4 — the
unguessable ``uuid4`` id IS the credential, the shared chat shows the
figure to a guest) with the four 404 ``document not found`` shapes
(unknown id, unparseable id — 404 and never 422, a text doc, a row
whose copy is gone from disk).
Task 05 of this phase adds the INGESTION-TO-SERVE round trip: the
task-03 content contract on ``GET /api/documents/content`` for an SVG
row (``is_image: true``, an ``image_url`` of the form
``/api/documents/<uuid>/image``, ``format: "svg"`` and the extracted
``<title>``/``<desc>``/``<text>`` words as ``content``) and the
WIRE-CONTRACT PIN that ties the phases together: that literal
``image_url`` is accepted by phase 180's ``allowImageUrl`` renderer
allowlist — the ``![alt](<image_url>)`` synergy (an answer or a
markdown document referencing the bytes route paints the graphic).
Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient.
"""
from __future__ import annotations
import hashlib
import re
import uuid
from collections.abc import Iterator
from datetime import UTC, datetime
from pathlib import Path
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import text
import app.api.docs as docs_api
from app.core.security_headers import CSP
from app.main import app as fastapi_app
from app.models import Chunk, Document
from app.rag.summarizer import IMAGE_MIMES
from app.rag.svg_text import extract_svg_text
#: The phase-180 shared renderer — the file whose ``allowImageUrl``
#: allowlist decides whether a ``![alt](src)`` reference paints an
#: ``img.md-img`` or stays inert text (the wire-contract pin below).
MARKDOWN_JS = Path(__file__).resolve().parents[2] / "frontend" / "assets" / "markdown.js"
#: The 1×1 PNG the raster control row stores (the ``test_docs_api.py``
#: fixture's bytes — a real PNG header so a Content-Type mix-up is
#: visible in the value, not just in the length).
PNG_1X1 = bytes.fromhex(
"89504e470d0a1a0a0000000d49484452000000010000000108060000001f15c489"
"0000000d4944415478da63fcffff3f0300050201000d5ca5e90000000049454e44ae426082"
)
#: One SVG carrying BOTH its own words and a ``<script>`` sentinel. The
#: script is what makes the pin meaningful: the route does NOT sanitize
#: (it streams the stored bytes untouched — the viewer needs the real
#: graphic), so the ONLY thing standing between a stored SVG and a
#: script-capable document is the policy this module pins.
SVG_WITH_SCRIPT = (
b'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 120 80">'
b"<title>Topology</title><desc>Two nodes, one link</desc>"
b'<text x="8" y="12">edge router</text>'
b"<path d='M10 60 L110 20'/>"
b"<script>alert('probe')</script></svg>"
)
#: The response-header KEY set both the raster and the vector serve share
#: (FileResponse's own stat headers + the route's Cache-Control + the
#: three the SecurityHeadersMiddleware adds). Pinned so a future change
#: cannot silently drop or add a header on either branch — the vector
#: branch differs from the raster branch in the VALUE of one key, never
#: in the key set.
SERVE_HEADER_KEYS = {
"accept-ranges",
"cache-control",
"content-length",
"content-security-policy",
"content-type",
"etag",
"last-modified",
"x-content-type-options",
"x-frame-options",
}
def _cleanup_kb(db) -> None:
"""Truncate the KB tables (commit first — SQLAlchemy does NOT
autoflush pending ORM objects before a raw ``text()`` statement, and
a pending chunk INSERT flushed after the TRUNCATE would FK-violate)."""
db.commit()
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
@pytest.fixture()
def kb(db) -> Iterator:
"""The real Postgres with the KB tables emptied before and after the
test (the module's own rows are the only ones present)."""
_cleanup_kb(db)
yield db
_cleanup_kb(db)
def _seed_image_row(
db,
tmp_path: Path,
*,
source: str = "SvgSrc",
path: str = "diagram.svg",
data: bytes = SVG_WITH_SCRIPT,
content: str = "Topology — Two nodes, one link — edge router",
image_path: str | None = "auto",
) -> Document:
"""One ``is_image`` document row plus its persistent copy (the
importer's ``image_dir`` layout — ``<image_dir>/<doc-id><ext>``)
under *tmp_path*. ``image_path="auto"`` writes the copy at the
importer's layout; ``None`` leaves the row without a copy path; any
other string pins the column verbatim (the lost-copy corner writes
the copy and unlinks it in the test itself)."""
doc = Document(
id=uuid.uuid4(),
source=source,
path=path,
full_path=f"/tmp/{path}",
title=Path(path).stem,
content=content,
content_hash=hashlib.sha256(data).hexdigest(),
indexed_at=datetime.now(UTC),
created_at=datetime(2026, 1, 1, tzinfo=UTC),
summary=content,
is_image=True,
)
if image_path == "auto":
copy = tmp_path / f"{doc.id}{Path(path).suffix}"
copy.write_bytes(data)
doc.image_path = str(copy)
elif image_path is not None:
doc.image_path = image_path
db.add(doc)
db.flush()
db.add_all(
[
Chunk(
document_id=doc.id, position=0, content=content, embedding=[0.01] * 768
),
Chunk(
document_id=doc.id,
position=-1,
content=content,
embedding=[0.01] * 768,
is_summary=True,
),
]
)
db.commit()
return doc
def _seed_text_row(db, *, source: str = "TextSrc", path: str = "note.md") -> uuid.UUID:
"""One plain TEXT document (never ``is_image``) — a 404 shape."""
doc = Document(
id=uuid.uuid4(),
source=source,
path=path,
full_path=f"/tmp/{path}",
title=Path(path).stem,
content="# Note\n\nBody.",
content_hash=hashlib.sha256(path.encode("utf-8")).hexdigest(),
indexed_at=datetime.now(UTC),
created_at=datetime(2026, 1, 1, tzinfo=UTC),
)
db.add(doc)
db.commit()
return doc.id
# --- the vector serve contract ------------------------------------------
def test_svg_row_serves_the_vector_mime_with_the_sandboxed_csp(
kb, tmp_path: Path, admin_client: TestClient
) -> None:
"""An SVG row → 200 ``image/svg+xml`` + the sandboxed CSP + the
unchanged Cache-Control, and the body is byte-equal to the stored
copy (the script sentinel included — the route is a transport, not a
sanitizer; ``sandbox`` blocks scripting and makes the document an
opaque origin, ``default-src 'none'`` blocks every subresource)."""
doc = _seed_image_row(kb, tmp_path)
assert doc.image_path is not None
r = admin_client.get(f"/api/documents/{doc.id}/image")
assert r.status_code == 200
assert r.headers["content-type"] == "image/svg+xml"
# LOCKED A3, verbatim.
assert r.headers["content-security-policy"] == "default-src 'none'; sandbox"
assert r.headers["cache-control"] == "private, max-age=3600"
assert r.headers["x-content-type-options"] == "nosniff"
assert r.headers["x-frame-options"] == "DENY"
assert r.content == SVG_WITH_SCRIPT # the EXACT stored bytes, nothing stripped
assert Path(doc.image_path).read_bytes() == r.content
def test_svg_route_header_key_set_is_the_serve_surface(
kb, tmp_path: Path, admin_client: TestClient
) -> None:
"""The vector response's header KEY set equals the shared serve
surface — one extra VALUE (the CSP), no extra header, and no
``content-disposition`` (the graphic must render inline)."""
doc = _seed_image_row(kb, tmp_path)
r = admin_client.get(f"/api/documents/{doc.id}/image")
assert set(r.headers) == SERVE_HEADER_KEYS
assert "content-disposition" not in r.headers
def test_svg_mime_guard_is_derived_from_the_map_never_from_the_client(
kb, tmp_path: Path, admin_client: TestClient
) -> None:
"""The guard keys on the resolved ``media_type`` — the extension map
(lower-cased, from the ROW's stored path), never a client value: an
upper-case ``.SVG`` copy is still vector, and an extension the map
does not name still takes the generic fallback with NO sandbox."""
assert docs_api.SVG_MEDIA_TYPE == IMAGE_MIMES[".svg"] == "image/svg+xml"
assert docs_api.SVG_SANDBOX_CSP == "default-src 'none'; sandbox"
upper = _seed_image_row(kb, tmp_path, path="diagram.SVG", source="UpperSrc")
assert upper.image_path is not None
upper_copy = Path(upper.image_path)
upper_copy.unlink()
upper_copy = upper_copy.with_name(f"{upper.id}.SVG")
upper_copy.write_bytes(SVG_WITH_SCRIPT)
upper.image_path = str(upper_copy)
kb.commit()
r = admin_client.get(f"/api/documents/{upper.id}/image")
assert r.status_code == 200
assert r.headers["content-type"] == "image/svg+xml"
assert r.headers["content-security-policy"] == docs_api.SVG_SANDBOX_CSP
unknown = _seed_image_row(
kb, tmp_path, path="mystery.bin", source="UnknownSrc", data=PNG_1X1
)
r = admin_client.get(f"/api/documents/{unknown.id}/image")
assert r.status_code == 200
assert r.headers["content-type"] == "application/octet-stream"
# No sandbox — the policy is scoped to the vector type only.
assert r.headers["content-security-policy"] == CSP
# --- the raster branch is untouched -------------------------------------
def test_raster_row_response_headers_are_unchanged_by_the_svg_rule(
kb, tmp_path: Path, admin_client: TestClient
) -> None:
"""A PNG row is served EXACTLY as before phase 181: ``image/png``,
the same Cache-Control, the same header KEY set, and a CSP equal to
the app-wide A20 string from ``SecurityHeadersMiddleware`` — no
route-specific policy, no ``sandbox`` anywhere in it. The byte
equality of the body is the raster half of "nothing changed"."""
doc = _seed_image_row(
kb,
tmp_path,
source="PngSrc",
path="pic.png",
data=PNG_1X1,
content="A red square on a white background.",
)
r = admin_client.get(f"/api/documents/{doc.id}/image")
assert r.status_code == 200
assert r.headers["content-type"] == "image/png"
assert r.headers["cache-control"] == "private, max-age=3600"
assert r.headers["content-security-policy"] == CSP
assert "sandbox" not in r.headers["content-security-policy"]
assert r.headers["x-content-type-options"] == "nosniff"
assert set(r.headers) == SERVE_HEADER_KEYS
assert r.content == PNG_1X1
# --- the public posture and the 404 shapes (unchanged) ------------------
def test_svg_bytes_are_public_with_the_sandbox(
kb, tmp_path: Path, admin_client: TestClient
) -> None:
"""Phase 124 LOCKED A4 stays intact for the vector type: a FRESH
client (no session — the shared chat's guest) gets the same bytes
and the same sandboxed CSP as a signed-in one. The policy is not an
access control (the unguessable ``uuid4`` id is), so making the
response safe does not make it private."""
doc = _seed_image_row(kb, tmp_path)
anonymous = TestClient(fastapi_app)
r = anonymous.get(f"/api/documents/{doc.id}/image")
assert r.status_code == 200
assert r.headers["content-type"] == "image/svg+xml"
assert r.headers["content-security-policy"] == "default-src 'none'; sandbox"
assert r.content == SVG_WITH_SCRIPT
r_admin = admin_client.get(f"/api/documents/{doc.id}/image")
assert r_admin.status_code == 200
assert r_admin.content == r.content
def test_svg_route_404_shapes(kb, tmp_path: Path, admin_client: TestClient) -> None:
"""Every non-servable case is still 404 ``document not found`` —
including the SVG rows: an unknown uuid, a NON-uuid string (404, not
422 — a guessed id is an unknown document), a TEXT document (no
bytes behind it), an image row whose ``image_path`` is NULL, and an
SVG row whose copy was deleted from disk."""
text_doc_id = _seed_text_row(kb)
no_copy = _seed_image_row(kb, tmp_path, path="nocopy.svg", image_path=None)
lost = _seed_image_row(kb, tmp_path, path="lost.svg", source="LostSrc")
assert lost.image_path is not None
Path(lost.image_path).unlink()
for doc_id in (
str(uuid.uuid4()), # unknown id
"not-a-uuid", # unparseable id → 404, never 422
str(text_doc_id), # text document
str(no_copy.id), # image row, image_path NULL
str(lost.id), # image row, copy gone from disk
):
r = admin_client.get(f"/api/documents/{doc_id}/image")
assert r.status_code == 404, doc_id
assert r.json() == {"detail": "document not found"}, doc_id
# --- the content endpoint on an SVG row (task 05, the ingestion shape) ---
def test_content_endpoint_serves_the_svg_row_as_a_vector_image(
kb, tmp_path: Path, admin_client: TestClient
) -> None:
"""``GET /api/documents/content`` for an SVG row carries the four
keys the viewer's vector branch needs: ``is_image: true``, an
``image_url`` of the exact ``/api/documents/<uuid>/image`` form
(the bytes route above), ``format: "svg"`` — the ONLY place in the
app that knows the type (task 04's ``doc-image-img--vector`` keys
off it) — and ``content`` equal to the SVG's OWN extracted
``<title>``/``<desc>``/``<text>`` words (LOCKED A1: the row's text
is what the importer stored, never raw XML re-derived here). The
``<script>`` sentinel in the stored bytes never reaches this
payload — the DB text IS the extraction."""
extracted = extract_svg_text(SVG_WITH_SCRIPT, stem="diagram")
assert extracted is not None and extracted.structural is False
doc = _seed_image_row(
kb, tmp_path, content=extracted.text, source="VecSrc", path="vector.svg"
)
r = admin_client.get(
"/api/documents/content", params={"source": "VecSrc", "path": "vector.svg"}
)
assert r.status_code == 200
body = r.json()
assert body["is_image"] is True
assert body["image_url"] == f"/api/documents/{doc.id}/image"
assert body["format"] == "svg"
assert body["content"] == extracted.text
assert body["content"] == "Topology\nTwo nodes, one link\nedge router"
assert body["summary"] == extracted.text
# The serve round trip: the image_url the payload advertises answers.
img = admin_client.get(body["image_url"])
assert img.status_code == 200
assert img.headers["content-type"] == "image/svg+xml"
assert img.content == SVG_WITH_SCRIPT
# --- the wire-contract pin: allowImageUrl accepts the route (task 05) ---
def _allow_image_url() -> re.Pattern[str]:
"""The phase-180 ``allowImageUrl`` regex, extracted verbatim from
``frontend/assets/markdown.js`` (the JS literal is valid Python
syntax — one source, no hand-mirrored copy). A failed extraction
fails loudly: the pin only means something if it is THE shipped
allowlist."""
js = MARKDOWN_JS.read_text(encoding="utf-8")
match = re.search(
r"function allowImageUrl\(url\) \{\s*return (/.*?/[a-z]*)\.test\(url\)",
js,
re.S,
)
assert match is not None, "allowImageUrl no longer has the shape this pin reads"
literal = match.group(1)
pattern, _, flags = literal[1:].rpartition("/")
assert set(flags) <= {"i"}, f"unexpected regex flags {flags!r}"
return re.compile(pattern, re.IGNORECASE if "i" in flags else 0)
def test_image_url_shape_passes_the_phase_180_image_allowlist(
kb, tmp_path: Path, admin_client: TestClient
) -> None:
"""The phases-tie-together pin: the literal ``image_url`` an SVG row
advertises passes the renderer's ``allowImageUrl`` — so an answer
(or a markdown document) that carries ``![alt](<image_url>)`` paints
``img.md-img`` from THIS route instead of printing the reference
(phase 180 + phase 181 end to end; the E2E's
``test_markdown_image_ref_renders`` is the browser half). The
sanity probe proves the extracted regex actually discriminates:
the same allowlist rejects a ``javascript:`` src (LOCKED A5)."""
doc = _seed_image_row(kb, tmp_path, source="PinSrc", path="wire.svg")
r = admin_client.get(
"/api/documents/content", params={"source": "PinSrc", "path": "wire.svg"}
)
image_url = r.json()["image_url"]
assert image_url == f"/api/documents/{doc.id}/image"
# The literal path the renderer receives (no query, no fragment).
assert re.fullmatch(r"/api/documents/[0-9a-f-]{36}/image", image_url)
allow = _allow_image_url()
assert allow.search(image_url), f"allowImageUrl rejected its own route: {image_url}"
# The discriminating half: the allowlist is real, not "accept all".
assert allow.search("javascript:alert(1)") is None
assert allow.search("data:text/html,<script>") is None
tests/unit/test_chat_image_questions.py
+39 -7
View File
@@ -68,6 +68,7 @@ from pydantic import ValidationError
from app.api import chat as chat_api
from app.api import chat_images
from app.api.chat_images import _VECTOR_IMAGE_EXTENSIONS
from app.config import Settings
from app.main import app as fastapi_app
from app.models import Document
@@ -354,6 +355,32 @@ def test_upload_rejects_a_non_image_extension_with_the_fixed_detail(
assert not image_dir.exists() # the dir is created on the accepted path only
def test_upload_rejects_a_vector_svg_even_though_it_indexes(
client: TestClient, image_dir: Path
) -> None:
"""Phase 181: ``svg`` joins the DEFAULT image set for the DOCUMENT
walk, and question attachments stay RASTER-only — the upload 422s
with the six-extension detail and writes nothing.
Without the subtraction this would be a silent trap: the stored
``<uuid>.svg`` is refused by the serve route's filename regex (a
broken figure in the user's own bubble) and an OpenAI-compatible
vision endpoint rejects ``image/svg+xml``, so the turn could never
be answered. The vector family belongs to INDEXING, not to
attachments."""
assert ".svg" in Settings(_env_file=None).image_extension_set # pyright: ignore[reportCallIssue]
assert frozenset({".svg"}) == _VECTOR_IMAGE_EXTENSIONS
r = _upload(
client, "diagram.svg", b'<svg xmlns="http://www.w3.org/2000/svg"></svg>',
mime="image/svg+xml",
)
assert r.status_code == 422
assert r.json()["detail"] == (
"only .bmp, .gif, .jpeg, .jpg, .png, .webp images are accepted"
)
assert not image_dir.exists()
def test_upload_rejects_a_name_without_an_extension(
client: TestClient, image_dir: Path
) -> None:
@@ -916,7 +943,10 @@ def test_the_attach_reveal_is_gated_on_the_config_flag() -> None:
def test_the_client_precheck_matches_the_servers_six_extensions() -> None:
"""ONE list of the six, on both sides: the client pre-check (the
file NAME's extension — the accept attribute is advisory) and the
server's phase-122 frozenset (the upload endpoint's authority).
server's phase-122 frozenset MINUS the vector members (the upload
endpoint's authority — phase 181: question images stay RASTER-only,
a vision endpoint rejects ``image/svg+xml`` and the serve route's
filename regex would never serve one).
The client list must stay the code default's six — a drift here
would either block a valid pick or waste a round-trip on a 422."""
js = _app_js()
@@ -931,12 +961,14 @@ def test_the_client_precheck_matches_the_servers_six_extensions() -> None:
"png",
"webp",
}
# and it equals the server's default set (dots stripped — the
# client list is dotless, the server's is dotted).
server_set = {
e.lstrip(".")
for e in Settings(_env_file=None).image_extension_set # pyright: ignore[reportCallIssue]
}
# and it equals the server's default raster set (dots stripped — the
# client list is dotless, the server's is dotted). ``svg`` is in the
# DEFAULT image set for the document walk (phase 181) and
# deliberately NOT attachable here (see
# ``chat_images._VECTOR_IMAGE_EXTENSIONS``).
raw_default = Settings(_env_file=None).image_extension_set # pyright: ignore[reportCallIssue]
assert ".svg" in raw_default # the vector member (phase 181)
server_set = {e.lstrip(".") for e in raw_default - _VECTOR_IMAGE_EXTENSIONS}
assert client_set == server_set
# the file input's accept attribute names the same six (mime form).
m = re.search(r'<input type="file" id="attach-file"\s+accept="([^"]+)"', _index_html())
tests/unit/test_document_viewer.py
+191
View File
@@ -94,6 +94,20 @@ format table as node behavioral runs over the real module, and the
``.doc-prose`` CSS (the 72rem reading column + ``pre-wrap`` +
``overflow-wrap: anywhere``, tokens only) beside the ``.doc-raw``
contract that stays locked (phase 116 A1 / phase 165 A3).
Phase 181 (task 04) — the SVG vector preview: ``docImageBlock`` adds
the ``doc-image-img--vector`` modifier ONLY when
``doc.format === "svg"`` (the base class is always kept) — a
``viewBox``-only SVG has NO intrinsic size, so the default 300×150
replaced-element box would sit small inside the full-width
``.doc-image`` card — pinned as a node behavioral run of the real
``docImageBlock`` (svg → the compound class, ``src`` from
``image_url`` never inlined, ``alt`` = summary with the title
fallback; png → the class string byte-identical to the base class)
plus the ``.doc-image-img--vector`` CSS pins (``width: 100%`` +
``height: auto`` + ``object-fit: contain`` under the base rule's
70vh clamp, zero colour literals, the base ``.doc-image-img`` rule
untouched).
"""
from __future__ import annotations
@@ -2102,3 +2116,180 @@ def test_doc_prose_css_wraps_in_the_reading_column() -> None:
raw_rule = css[raw_i : css.find("}", raw_i)]
assert "white-space: pre" in raw_rule
assert "overflow-x: auto" in raw_rule
# ---------------------------------------------------------------------------
# Phase 181 (task 04) — the SVG vector preview: docImageBlock's format
# gate (base class + the doc-image-img--vector modifier, LOCKED A3:
# bytes only ever through <img>) and the .doc-image-img--vector CSS
# (a viewBox-only SVG has no intrinsic size — width: 100% spans the
# .doc-image card, object-fit: contain survives the 70vh clamp,
# palette-neutral). The chat figure (app.js) and the Sources
# thumbnail (sources.js) are VERIFIED non-changes (design D7): both
# key off image_url alone and their boxes pin the img
# (.source-image-img 96px cap, .kb-doc-thumb-img fixed 48px box).
# ---------------------------------------------------------------------------
def test_doc_image_block_gates_the_vector_modifier_on_format() -> None:
"""Source pins (node-free): docImageBlock selects the compound
class on ``doc.format === "svg"`` and the bare base class
otherwise (every current .doc-image-img rule keeps applying);
the bytes stay an <img> src property assignment — never inlined
(LOCKED A3), never innerHTML. The comment records the rationale
(no-intrinsic-size / 300×150) and the legacy-TEXT honesty
(XML-in-.doc-raw is the truth; no content sniffing)."""
js = _read(DOCUMENT_JS)
fn = _fn(js, "docImageBlock")
assert 'doc.format === "svg"' in fn
assert '"doc-image-img doc-image-img--vector"' in fn
assert ': "doc-image-img";' in fn, "the raster path keeps the bare base class"
assert "img.src = doc.image_url;" in fn, "the bytes URL stays a property assignment"
assert "innerHTML" not in fn
# The rationale + honesty comments (house style: why, not what).
for citation in ("Phase 181", "viewBox", "300\u00d7150", "LOCKED A3", ".doc-raw"):
assert citation in fn, f"the comment must cite {citation!r}"
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
def test_doc_image_block_svg_yields_the_vector_class_png_unchanged() -> None:
"""Behavioral (node, the house fake DOM): the REAL docImageBlock —
an SVG doc renders class="doc-image-img doc-image-img--vector"
with src = image_url and alt = the summary (title fallback when
the summary is null/blank); a PNG doc's class string is the base
class ONLY (byte-identical to pre-phase — the phase-122/133
raster surfaces cannot drift); the wrapper stays .doc-image."""
js = _read(DOCUMENT_JS)
# The docs ride in via json.dumps (the summary carries quotes and
# an em-dash — the alt contract must survive verbatim).
cases = {
"svg": {
"format": "svg",
"image_url": "/api/documents/7/image",
"summary": 'Vector diagram "topo" \u2014 3 shapes',
"title": "topo",
},
"svgNoSummary": {
"format": "svg",
"image_url": "/api/documents/9/image",
"summary": None,
"title": "rack",
},
"svgBlankSummary": {
"format": "svg",
"image_url": "/api/documents/10/image",
"summary": " ",
"title": "blank",
},
"png": {
"format": "png",
"image_url": "/api/documents/8/image",
"summary": "A photo",
"title": "t",
},
}
script = (
_CONTENT_BRANCH_FAKE_DOM
+ _fn(js, "docImageBlock")
+ "\nconst cases = "
+ json.dumps(cases)
+ """
;
function imageFor(doc) {
const wrap = docImageBlock(doc);
const img = wrap.children[0];
return {
wrapClass: wrap.className, imgClass: img.className, src: img.src, alt: img.alt,
};
}
const out = {};
for (const [k, doc] of Object.entries(cases)) out[k] = imageFor(doc);
console.log(JSON.stringify(out));
"""
)
out = json.loads(_run_node(script))
assert out["svg"] == {
"wrapClass": "doc-image",
"imgClass": "doc-image-img doc-image-img--vector",
"src": "/api/documents/7/image",
"alt": 'Vector diagram "topo" \u2014 3 shapes',
}
# alt falls back to the title (the WCAG alt contract, unchanged).
assert out["svgNoSummary"]["alt"] == "rack"
assert out["svgBlankSummary"]["alt"] == "blank"
assert out["svgBlankSummary"]["imgClass"] == "doc-image-img doc-image-img--vector"
# The raster class string is byte-identical to pre-phase (base only).
assert out["png"]["imgClass"] == "doc-image-img"
assert out["png"]["src"] == "/api/documents/8/image"
def test_doc_image_vector_css_spans_the_card_without_squashing() -> None:
"""styles.css: ``.doc-image-img--vector`` (immediately after
``.doc-image-img``) is ``width: 100%`` + ``height: auto`` +
``object-fit: contain`` — a viewBox-only SVG (no intrinsic size:
the default replaced-element box is 300\u00d7150) spans the
full-width .doc-image card, and contain letterboxes instead of
squashing when the base rule's max-height: 70vh clamps a tall
drawing. Zero colour literals (palette-neutral — the monochrome
invariant); the base .doc-image-img rule is UNTOUCHED (the
phase-122 raster contract)."""
css = _read(STYLES_CSS)
base_i = css.find(".doc-image-img {")
vec_i = css.find(".doc-image-img--vector {")
assert -1 < base_i < vec_i, ".doc-image-img--vector must sit next to the base rule"
# Provenance comment directly above the rule (house style).
comment_start = css.rfind("/*", 0, vec_i)
provenance = css[comment_start:vec_i]
assert -1 < comment_start < vec_i
for citation in ("phase 181", "viewBox", "300\u00d7150", "object-fit", "doc.format"):
assert citation in provenance, f"the comment must cite {citation!r}"
assert "no animation" in provenance
rule = css[vec_i : css.find("}", vec_i)]
for decl in ("width: 100%", "height: auto", "object-fit: contain"):
assert decl in rule, f".doc-image-img--vector must carry {decl!r}"
# Palette-neutral — zero colour literals (no new hue).
assert not re.search(r"#[0-9a-fA-F]{3,8}\b", rule)
assert "rgb(" not in rule and "hsl(" not in rule and "color-mix(" not in rule
# The untouched base rule (the raster + onerror contract).
base_rule = css[base_i : css.find("}", base_i)]
for decl in ("max-width: 100%", "max-height: 70vh", "height: auto"):
assert decl in base_rule, f"the base rule must keep {decl!r}"
def test_figure_and_thumbnail_key_off_image_url_alone() -> None:
"""Design D7's verified non-changes: the chat figure
(appendSourceImageFigure) and the Sources KB-tree thumbnail
(docThumb) resolve their bytes from ``image_url`` alone — no
format field on the wire, none added — and their CSS boxes pin
the img (.source-image-img: 96px cap + object-fit: contain; the
.kb-doc-thumb-img: the fixed 48px box), so a size-less SVG
scales inside each cap exactly like a raster. Pins: neither
function sniffs a format at all (the deliberate non-change,
design D7), the phase-181 comments sit where they were made,
and the two rules keep their box declarations."""
app_js = _read(APP_JS)
assert app_js.count("function appendSourceImageFigure(") == 1
fig = _fn(app_js, "appendSourceImageFigure")
assert "img.src = s.image_url;" in fig
assert "format" not in fig, "no format sniffing on the chat figure (D7)"
fn_start = app_js.index("function appendSourceImageFigure(")
assert "phase 181" in app_js[max(0, fn_start - 2400) : fn_start].lower(), (
"the deliberate-non-change comment sits above the figure"
)
sources_js = _read(SOURCES_JS)
thumb_start = sources_js.index("function docThumb(d)")
thumb_end = sources_js.index("\n }", thumb_start)
thumb = sources_js[thumb_start:thumb_end]
assert "img.src = d.image_url;" in thumb
assert "format" not in thumb, "no format sniffing on the thumbnail (D7)"
branch_i = sources_js.index("if (d.is_image) {")
assert "phase 181" in sources_js[branch_i - 900 : branch_i].lower(), (
"the verified-non-change comment sits above the is_image branch"
)
css = _read(STYLES_CSS)
cap_i = css.find(".source-image-img {")
cap = css[cap_i : css.find("}", cap_i)]
assert "max-height: 96px" in cap and "object-fit: contain" in cap
box_i = css.find(".kb-doc-thumb-img {")
box = css[box_i : css.find("}", box_i)]
assert "width: 100%" in box and "height: 100%" in box
tests/unit/test_image_documents.py
+29 -8
View File
@@ -106,19 +106,25 @@ def _settings(**kwargs: Any) -> Settings:
def test_image_defaults_match_locked_a3() -> None:
"""LOCKED A3: image support is OFF by default (enable only when the
chat model supports vision), the built-in extension set is the six
common formats, and the image bytes' persistent home is the
"""LOCKED A3 + phase 181 LOCKED A7: image support is OFF by default
(enable only when the chat model supports vision), the built-in
extension set is the six common raster formats PLUS ``svg`` (the one
VECTOR member — A1: it indexes from its own extracted text, never a
vision call), and the image bytes' persistent home is the
``~/bor-sources/images`` raw string (``expanduser`` is the
importer's job — the ``sources_dir``/``upload_dir`` convention)."""
importer's job — the ``sources_dir``/``upload_dir`` convention).
``images`` stays the SINGLE knob (A7): with it off an ``.svg`` is a
plain text document, no special-casing."""
s = _settings()
assert s.images is False
# Dotted, lowercased form — the shape the walk's matcher consumes
# (the ``import_extension_set`` precedent).
assert s.image_extension_set == {".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp"}
assert s.image_extension_set == {
".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp", ".svg",
}
assert s.image_dir == "~/bor-sources/images"
# The raw string is stored untouched (no expanduser here).
assert s.image_extensions == "png,jpg,jpeg,webp,gif,bmp"
assert s.image_extensions == "png,jpg,jpeg,webp,gif,bmp,svg"
def test_image_set_is_separate_from_the_import_set() -> None:
@@ -127,7 +133,9 @@ def test_image_set_is_separate_from_the_import_set() -> None:
the import set leaves the image set untouched, and vice versa."""
s = _settings(import_extensions="md")
assert s.import_extension_set == {".md"}
assert s.image_extension_set == {".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp"}
assert s.image_extension_set == {
".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp", ".svg",
}
s2 = _settings(image_extensions="md")
assert s2.image_extension_set == {".md"}
assert ".md" in s2.import_extension_set # the default import set is intact
@@ -762,7 +770,16 @@ def test_image_mimes_cover_the_default_family() -> None:
default image family — dotted lowercase keys, the
``image_extension_set`` shape — and the unlisted-token fallback is
the generic mime (a custom ``BOR_IMAGE_EXTENSIONS`` format fails
soft at the vision endpoint, never a guessed type)."""
soft at the vision endpoint, never a guessed type).
The map is pinned by IDENTITY (not membership), so a silent removal
or re-pointing of any entry fails here. Phase 181 (task 03) adds
``.svg`` as the ONE vector entry; the six raster entries are
untouched by it, and the vector import branch never reaches
:func:`image_data_url` (LOCKED A1: no vision call for an SVG), so
the addition cannot change a describe call — the value exists for
the BYTES route (``app.api.docs.get_document_image`` reads the same
map for its ``Content-Type`` and its sandboxed-CSP guard)."""
assert IMAGE_MIMES == {
".png": "image/png",
".jpg": "image/jpeg",
@@ -770,8 +787,12 @@ def test_image_mimes_cover_the_default_family() -> None:
".webp": "image/webp",
".gif": "image/gif",
".bmp": "image/bmp",
".svg": "image/svg+xml",
}
assert IMAGE_FALLBACK_MIME == "application/octet-stream"
# Every member of the default image set now has a mime — raster and
# vector alike (the vector one is serve-side only, per above).
assert ".svg" in _settings().image_extension_set
assert all(ext in IMAGE_MIMES for ext in _settings().image_extension_set)
tests/unit/test_importer.py
+2 -1
View File
@@ -1008,6 +1008,7 @@ def test_import_summary_log_line_includes_summary_counters(
s.summary_backfilled = 1
s.dates_updated = 0
s.images_failed = 0
s.svg_indexed = 4
s.image_context_backfilled = 2
s.pdfs_screenshotted = 1
s.pdfs_pictorial = 2
@@ -1021,7 +1022,7 @@ def test_import_summary_log_line_includes_summary_counters(
assert line == (
"import: summary files=3 added=3 updated=0 unchanged=0 pruned=0 errors=0 "
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
"dates_updated=0 images_failed=0 image_context_backfilled=2 "
"dates_updated=0 images_failed=0 svg_indexed=4 image_context_backfilled=2 "
"pdfs_screenshotted=1 pdfs_pictorial=2 pdf_pictorial_pages=5 "
"pdf_images_deduped=3 pdf_images_blank_skipped=1 "
"formats=yaml:2,md:1"
tests/unit/test_svg_image_ingest.py
+482
View File
@@ -0,0 +1,482 @@
"""Unit: phase 181 task 02 — an ``.svg`` indexes as a VECTOR image document.
The pure extractor (:mod:`app.rag.svg_text`) is pinned in
``tests/unit/test_svg_text_extraction.py``; this module pins what the
IMPORTER does with it (LOCKED A1/A2/A7):
* the vector branch of :func:`app.rag.importer._index_image_file` — an
``.svg`` lands ``is_image=True`` with ``image_path`` under
``image_dir``, its ``content`` is the file's OWN extracted text, its
``summary`` mirrors it (the phase-122 convention), and the vision seam
:func:`app.rag.importer._describe_or_skip` is NEVER called (the seam is
patched with a recording stand-in that would return a sentinel — the
empty recording list + the content assertion are the proof, and a
call-counting seam cannot silently succeed);
* the log-only ``svg_indexed`` counter and the ``import: svg indexed``
auditability line (AGENTS rule 10), including ``structural=True`` for
a drawing whose only words are the LOCKED A2 fallback line;
* the fail-soft (LOCKED A2's boundary): a file the extractor REFUSES
(over the byte cap, malformed XML) skips exactly like an undescribed
raster — ``images_failed``, a warning, no row, no orphan copy — while
a merely textless SVG does NOT skip;
* the phase-125 ``image_context`` enrichment prepends for an SVG exactly
as for a raster;
* the phase-181 fast-path kind guard (:func:`_row_kind_mismatch`): a row
imported as TEXT before ``svg`` joined ``BOR_IMAGE_EXTENSIONS`` is
converted on the NEXT sync (same bytes, same ``file_hash`` — and for an
ASCII file the text digest equals the byte digest too, which is why the
image branch's own content-digest skip now requires ``is_image``), and
a row already of the right kind — raster or vector — keeps the
phase-147 fast path byte-for-byte (no second seam call, no second
``svg_indexed``).
"""
from __future__ import annotations
import asyncio
import hashlib
import logging
from pathlib import Path
from typing import Any
import pytest
from sqlalchemy import select
import app.rag.importer as importer
from app.config import Settings
from app.models import Chunk, Document
from app.rag.importer import import_sources
from app.rag.svg_text import MAX_SVG_BYTES
from tests.fakes import FakeEmbedder
#: A diagram that carries words: ``<title>`` + ``<desc>`` + a ``<text>``
#: inside a group (document order is the extractor's contract).
SVG_TOPO = (
b'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 120 80">'
b"<title>Server rack topology</title>"
b"<desc>Two switches uplink to the router.</desc>"
b'<g><path d="M0 0h10"/><text>rack-1</text></g>'
b"</svg>"
)
#: The text :func:`app.rag.svg_text.extract_svg_text` yields for
#: :data:`SVG_TOPO` (pinned literally here so a change to the extraction
#: contract breaks BOTH modules, not silently).
SVG_TOPO_TEXT = "Server rack topology\nTwo switches uplink to the router.\nrack-1"
#: A pure-shape drawing: no words at all (the LOCKED A2 class).
SVG_SHAPES_ONLY = (
b'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64">'
b'<path d="M0 0h10"/><path d="M0 20h10"/><circle cx="1" cy="1" r="2"/>'
b"</svg>"
)
#: The LOCKED A2 structural line for :data:`SVG_SHAPES_ONLY` — built only
#: from facts the file states (stem, shape census, viewBox).
SVG_SHAPES_ONLY_TEXT = (
'Vector diagram "gizmo" — 3 shapes (path×2, circle), viewBox 64×64'
)
#: Over :data:`MAX_SVG_BYTES`: the size guard refuses it BEFORE any parse.
SVG_OVER_CAP = b"<svg>" + b"\n" * (MAX_SVG_BYTES + 1) + b"</svg>"
#: Well-formed bytes, unparseable markup (the second refusal class).
SVG_MALFORMED = b'<svg xmlns="http://www.w3.org/2000/svg"><title>oops'
#: A real 1×1 PNG for the raster fast-path regression pins.
PNG_1X1 = b"\x89PNG\r\n\x1a\n" + b"\x00" * 24
#: What a describe seam would hand back if the vector branch ever reached
#: it — never a plausible diagram text, so it cannot pass unnoticed.
VISION_SENTINEL = "SHOULD NEVER BE DESCRIBED: a vision call ran for an SVG."
def _settings(**kwargs: Any) -> Settings:
"""Build Settings without reading a .env file (deterministic tests)."""
kwargs.setdefault("_env_file", None)
return Settings(**kwargs) # pyright: ignore[reportCallIssue] (kwarg exists at runtime)
def _image_llm(tmp_path: Path, **kwargs: Any) -> FakeEmbedder:
"""A fake LLM whose settings index images into *tmp_path*."""
kwargs.setdefault("_env_file", None)
kwargs.setdefault("images", True)
kwargs.setdefault("image_dir", str(tmp_path / "images"))
llm = FakeEmbedder()
llm.settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
return llm
def _record_description(monkeypatch: pytest.MonkeyPatch) -> list[str]:
"""Patch the vision seam with a recorder (never a real call): the
paths it is asked to describe, always answered with
:data:`VISION_SENTINEL`. An empty list + a non-sentinel ``content``
is the proof that the vector branch bypassed vision entirely."""
handed: list[str] = []
async def _fake(
llm: importer.Embedder,
*,
data: bytes,
source: str,
rel: str,
full_path: Path,
context: str | None = None,
) -> str | None:
handed.append(rel)
return VISION_SENTINEL
monkeypatch.setattr(importer, "_describe_or_skip", _fake)
return handed
def _doc(db, source: str, rel: str) -> Document:
doc = db.scalar(
select(Document).where(Document.source == source, Document.path == rel)
)
assert doc is not None, f"no documents row for ({source!r}, {rel!r})"
return doc
def _cleanup_source(db, source: str) -> None:
for doc in db.scalars(select(Document).where(Document.source == source)).all():
db.delete(doc)
db.commit()
# --- the vector branch ------------------------------------------------------
def test_svg_indexes_as_an_image_document_with_no_vision_call(
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
) -> None:
"""LOCKED A1: the ``.svg`` takes the image branch (``is_image``,
``image_path``, the ``<doc-id>.svg`` copy, the ``summary = content``
convention, one embedded content chunk + the ``is_summary`` chunk)
and its searchable text is the file's own ``<title>``/``<desc>``/
``<text>`` — with the vision seam never invoked. ``svg_indexed`` and
the per-file ``import: svg indexed`` line make the run auditable."""
root = tmp_path / "svgsrc"
root.mkdir()
(root / "topo.svg").write_bytes(SVG_TOPO)
llm = _image_llm(tmp_path)
handed = _record_description(monkeypatch)
try:
with caplog.at_level(logging.INFO, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
assert summary.errors == 0
assert summary.formats == {"svg": 1} # counted under its image token
assert summary.svg_indexed == 1
assert handed == [], "an SVG must never reach the vision seam"
doc = _doc(db, root.name, "topo.svg")
assert doc.is_image is True
assert doc.title == "topo" # the non-markdown stem rule (shared with raster)
assert doc.content == SVG_TOPO_TEXT
assert doc.summary == doc.content # the phase-122 image convention
assert doc.image_context is None # standalone: the pre-phase row shape
assert doc.image_path is not None
copy = Path(doc.image_path)
assert copy.parent == Path(llm.settings.image_dir).expanduser()
assert copy.name == f"{doc.id}.svg" # row and copy agree on the id
assert copy.read_bytes() == SVG_TOPO # the exact bytes, unserved here
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc.id)).all()
content_chunks = [c for c in chunks if not c.is_summary]
assert [c.content for c in content_chunks] == [doc.content]
summary_chunks = [c for c in chunks if c.is_summary]
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
assert summary_chunks[0].content == doc.content
for c in chunks:
assert c.embedding is not None and len(c.embedding) == 768
indexed = [
r.getMessage() for r in caplog.records if "import: svg indexed" in r.getMessage()
]
assert indexed == [
f"import: svg indexed source={root.name} path=topo.svg "
f"structural=False chars={len(SVG_TOPO_TEXT)}"
]
line = next(
r.getMessage()
for r in caplog.records
if r.getMessage().startswith("import: summary files=")
)
assert "svg_indexed=1" in line and "images_failed=0" in line
finally:
_cleanup_source(db, root.name)
def test_textless_svg_indexes_with_the_structural_line(
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
) -> None:
"""LOCKED A2: a diagram of pure ``<path>``s is NOT skipped — it is
indexed with the structural line built from the file (stem, shape
census, viewBox), counted like any other vector doc and logged with
``structural=True`` so the owner can tell it from a worded diagram."""
root = tmp_path / "shapeonly"
root.mkdir()
(root / "gizmo.svg").write_bytes(SVG_SHAPES_ONLY)
llm = _image_llm(tmp_path)
handed = _record_description(monkeypatch)
try:
with caplog.at_level(logging.INFO, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
assert summary.svg_indexed == 1
assert handed == []
doc = _doc(db, root.name, "gizmo.svg")
assert doc.is_image is True
assert doc.content == SVG_SHAPES_ONLY_TEXT
assert doc.summary == doc.content
line = next(
r.getMessage() for r in caplog.records if "import: svg indexed" in r.getMessage()
)
assert "structural=True" in line
finally:
_cleanup_source(db, root.name)
@pytest.mark.parametrize(
("name", "payload"),
[
pytest.param("huge.svg", SVG_OVER_CAP, id="over-the-byte-cap"),
pytest.param("broken.svg", SVG_MALFORMED, id="malformed-xml"),
],
)
def test_unusable_svg_skips_the_doc_counts_and_logs(
db,
tmp_path: Path,
monkeypatch: pytest.MonkeyPatch,
caplog: pytest.LogCaptureFixture,
name: str,
payload: bytes,
) -> None:
"""The fail-soft stays LOCKED A3-shaped for the vector case: a file
the extractor refuses skips the doc ENTIRELY — no row, no copy (not
even the image dir), ``images_failed`` + a warning naming the vector
extraction, the sync continues. This is the ONLY skip class the phase
keeps for an SVG (:func:`app.rag.svg_text.extract_svg_text`)."""
root = tmp_path / "badsvg"
root.mkdir()
(root / name).write_bytes(payload)
llm = _image_llm(tmp_path)
handed = _record_description(monkeypatch)
try:
with caplog.at_level(logging.INFO, logger="app.importer"):
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.images_failed) == (1, 0, 1)
assert summary.svg_indexed == 0
assert handed == []
assert db.scalar(select(Document).where(Document.source == root.name)) is None
assert not Path(llm.settings.image_dir).expanduser().exists(), (
"no copy — and not even the image dir — may be left behind"
)
warnings = [
r
for r in caplog.records
if "svg text extraction failed" in r.getMessage()
]
assert len(warnings) == 1 and warnings[0].levelno == logging.WARNING
assert "image description failed" not in caplog.text
line = next(
r.getMessage()
for r in caplog.records
if r.getMessage().startswith("import: summary files=")
)
assert "images_failed=1" in line and "svg_indexed=0" in line
finally:
_cleanup_source(db, root.name)
def test_markdown_reference_context_prepends_for_an_svg(
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Phase 125's enrichment is kind-agnostic: a ``![alt](topo.svg)``
reference in the same source still builds ``image_context`` (from the
file on disk, no vision call) and the doc's content becomes
``<context>\\n\\n<extracted text>`` — context first for retrieval, the
vector file's own words second."""
root = tmp_path / "svganchor"
root.mkdir()
(root / "guide.md").write_text(
"Why this diagram exists.\n\n![The rack plan](topo.svg)\n\nWhat follows it.\n",
encoding="utf-8",
)
(root / "topo.svg").write_bytes(SVG_TOPO)
llm = _image_llm(tmp_path)
handed = _record_description(monkeypatch)
try:
summary = asyncio.run(import_sources([root], llm, session=db))
assert (summary.files, summary.added, summary.images_failed) == (2, 2, 0)
assert summary.svg_indexed == 1
assert handed == []
doc = _doc(db, root.name, "topo.svg")
assert doc.is_image is True
assert doc.image_context is not None
assert doc.image_context.startswith(f"Referenced in {root.name}/guide.md:")
assert 'Alt text: "The rack plan"' in doc.image_context
assert doc.content == f"{doc.image_context}\n\n{SVG_TOPO_TEXT}"
assert doc.summary == doc.content
finally:
_cleanup_source(db, root.name)
# --- the fast-path kind guard (the text → image migration) ------------------
def test_text_indexed_svg_row_is_converted_on_the_next_sync(
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Design D4: the row an earlier run indexed as TEXT (``svg`` not yet
in the image set) is converted by the NEXT sync — same bytes, same
``file_hash`` — and the sync AFTER that is fast again.
The conversion needs BOTH halves of the guard: the fast path in
:func:`_index_file` must fall through, and the image branch's own
content-digest skip must require ``is_image`` — an ASCII file's text
digest equals its byte digest, so without it the row would report
``unchanged`` forever and never gain a copy (asserted below, because
it is the whole reason the guard exists)."""
root = tmp_path / "migrate"
root.mkdir()
(root / "topo.svg").write_bytes(SVG_TOPO)
handed = _record_description(monkeypatch)
# Stage A — the pre-phase state: images OFF, ``svg`` imported as text.
text_llm = _image_llm(
tmp_path,
images=False,
import_extensions=f"{_settings().import_extensions},svg",
)
try:
summary_a = asyncio.run(import_sources([root], text_llm, session=db))
assert (summary_a.files, summary_a.added, summary_a.svg_indexed) == (1, 1, 0)
before = _doc(db, root.name, "topo.svg")
assert before.is_image is False
assert before.image_path is None
assert before.content == SVG_TOPO.decode("utf-8") # the raw XML the owner saw
byte_digest = hashlib.sha256(SVG_TOPO).hexdigest()
assert before.file_hash == byte_digest
# The trap D4 closes: the text digest of a byte-clean file IS the
# byte digest, so no digest comparison can spot the mismatch.
assert before.content_hash == byte_digest
doc_id = before.id
# Stage B — the toggle on: the guard fires, the row converts.
image_llm = _image_llm(tmp_path)
summary_b = asyncio.run(import_sources([root], image_llm, session=db))
assert (summary_b.added, summary_b.updated, summary_b.unchanged) == (0, 1, 0)
assert (summary_b.images_failed, summary_b.svg_indexed) == (0, 1)
assert handed == [], "the conversion must not call vision either"
after = _doc(db, root.name, "topo.svg")
assert after.id == doc_id # converted in place — the id (and copy name) is stable
assert after.is_image is True
assert after.content == SVG_TOPO_TEXT
assert after.summary == after.content
assert after.file_hash == byte_digest # the bytes never changed
# The image branch stamps the BYTES digest as ``content_hash`` —
# the SAME value the text path had already stored, which is why
# that skip needed its own ``is_image`` guard.
assert after.content_hash == byte_digest
copy = Path(str(after.image_path))
assert copy == Path(image_llm.settings.image_dir).expanduser() / f"{doc_id}.svg"
assert copy.read_bytes() == SVG_TOPO
# The text chunks are replaced, not appended: one content chunk
# (the extracted text) + the is_summary mirror.
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc_id)).all()
assert [c.content for c in chunks if not c.is_summary] == [SVG_TOPO_TEXT]
assert [c.content for c in chunks if c.is_summary] == [SVG_TOPO_TEXT]
# Stage C — the guard fired once: the fast path resumes.
summary_c = asyncio.run(import_sources([root], image_llm, session=db))
assert (summary_c.added, summary_c.updated, summary_c.unchanged) == (0, 0, 1)
assert (summary_c.svg_indexed, summary_c.images_failed) == (0, 0)
assert handed == []
assert _doc(db, root.name, "topo.svg").is_image is True
finally:
_cleanup_source(db, root.name)
def test_correctly_classified_raster_row_still_takes_the_fast_path(
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The phase-147 fast path survives the guard for every row whose
KIND already agrees: a raster image with a matching ``file_hash`` is
UNCHANGED on the next sync — the seam is not called a second time and
no vector counter moves."""
root = tmp_path / "rasterfast"
root.mkdir()
(root / "pic.png").write_bytes(PNG_1X1)
llm = _image_llm(tmp_path)
handed = _record_description(monkeypatch)
try:
first = asyncio.run(import_sources([root], llm, session=db))
assert (first.added, first.unchanged, first.svg_indexed) == (1, 0, 0)
assert handed == ["pic.png"] # the raster branch DID use the seam
doc = _doc(db, root.name, "pic.png")
assert doc.is_image is True
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
assert second.svg_indexed == 0
assert handed == ["pic.png"], "the fast path must not re-describe the image"
assert _doc(db, root.name, "pic.png").content == VISION_SENTINEL
finally:
_cleanup_source(db, root.name)
def test_correctly_classified_vector_row_still_takes_the_fast_path(
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Same pin for the vector row: once converted (or freshly indexed)
it is UNCHANGED — no second extraction, no second ``svg_indexed``,
no second copy write."""
root = tmp_path / "vectorfast"
root.mkdir()
(root / "topo.svg").write_bytes(SVG_TOPO)
llm = _image_llm(tmp_path)
handed = _record_description(monkeypatch)
try:
first = asyncio.run(import_sources([root], llm, session=db))
assert (first.added, first.svg_indexed) == (1, 1)
copy = Path(str(_doc(db, root.name, "topo.svg").image_path))
copy.unlink() # the fast path must not re-create it either
second = asyncio.run(import_sources([root], llm, session=db))
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
assert (second.svg_indexed, second.images_failed) == (0, 0)
assert handed == []
assert not copy.exists()
finally:
_cleanup_source(db, root.name)
def test_row_kind_mismatch_table() -> None:
"""The guard's whole truth table (pure — no DB, no filesystem):
it fires ONLY for a TEXT row whose file the CURRENT settings would
index as an image. The reverse direction (an ``is_image`` row while
the toggle is off, or a plain text file) stays on the fast path —
the phase-122 prune guard owns the off state (LOCKED A7)."""
svg = Path("/src/diagrams/topo.svg")
md = Path("/src/notes.md")
text_row = Document(source="s", path="topo.svg", is_image=False)
image_row = Document(source="s", path="topo.svg", is_image=True)
def _settings_(*, images: bool) -> Settings:
return _settings(images=images)
# The migration: the toggle is on, the file is an image, the row is not.
assert importer._row_kind_mismatch(svg, text_row, _settings_(images=True)) is True
# Already an image doc → nothing to migrate.
assert importer._row_kind_mismatch(svg, image_row, _settings_(images=True)) is False
# Toggle off → the walk never admits the file, the prune guard protects
# the row: never force the image path (LOCKED A7).
assert importer._row_kind_mismatch(svg, text_row, _settings_(images=False)) is False
assert importer._row_kind_mismatch(svg, image_row, _settings_(images=False)) is False
# A file outside the image set is a text file either way.
assert importer._row_kind_mismatch(md, text_row, _settings_(images=True)) is False
# An owner-narrowed image set (no ``svg``) is not a mismatch either.
narrow = _settings(images=True, image_extensions="png,jpg")
assert importer._row_kind_mismatch(svg, text_row, narrow) is False
tests/unit/test_svg_text_extraction.py
+426
View File
@@ -0,0 +1,426 @@
"""Unit tests: SVG text extraction (phase 181, task 01).
:func:`app.rag.svg_text.extract_svg_text` is a PURE function — the text
that makes an SVG document retrievable WITHOUT a vision call (LOCKED
A1). No DB, no filesystem, no LLM: every case is a literal SVG string,
so the whole contract is pinned here before the importer branch (task
02) relies on it.
The matrix:
* the guards that run BEFORE any parse — empty / whitespace-only input,
the :data:`~app.rag.svg_text.MAX_SVG_BYTES` cap, the
``<!DOCTYPE``/``<!ENTITY`` refusal (case-insensitive, both a doctype
wrapper and a bare entity declaration), malformed XML → ``None``
(never a raise);
* extraction — ``<title>``/``<desc>``/``<text>`` in DOCUMENT order,
``<tspan>`` children joined with a space, namespace-prefixed and
namespaced tags treated alike, ``xlink:`` attributes ignored,
whitespace normalised, repeats dropped case-insensitively keeping the
first, ``&amp;``/``&#8212;`` decoded by the parser;
* the :data:`~app.rag.svg_text.MAX_SVG_TEXT_CHARS` cap — cut at the
last newline/period before the cap, hard-cut only when the window
holds no boundary at all;
* the LOCKED A2 structural fallback — stem, shape census (counts,
plural, top-3 selection with a document-order tie-break) and the
``viewBox`` clause, each omitted when the file does not state it.
Importer-level tests (the vector branch, the no-vision-call seam) live
in the phase's ingest module (tasks 02/05).
"""
from __future__ import annotations
from app.rag.svg_text import (
MAX_SVG_BYTES,
MAX_SVG_TEXT_CHARS,
SvgText,
extract_svg_text,
local_name,
)
SVG_NS = "http://www.w3.org/2000/svg"
XLINK_NS = "http://www.w3.org/1999/xlink"
# --- constants -------------------------------------------------------------
def test_module_constants_are_pinned() -> None:
"""The caps are part of the contract the importer's tests assume."""
assert MAX_SVG_BYTES == 2 * 1024 * 1024
assert MAX_SVG_TEXT_CHARS == 4000
# --- the local_name helper -------------------------------------------------
def test_local_name_strips_the_clark_notation_namespace() -> None:
assert local_name(f"{{{SVG_NS}}}title") == "title"
assert local_name("title") == "title"
def test_local_name_of_a_non_string_tag_is_empty() -> None:
"""Comment / PI markers are callables — they carry no name."""
assert local_name(object()) == ""
# --- guards (no parse attempted) -------------------------------------------
def test_empty_input_is_a_refusal_not_a_crash() -> None:
assert extract_svg_text(b"") is None
assert extract_svg_text(b"", stem="diagram") is None
def test_whitespace_only_input_is_a_refusal() -> None:
assert extract_svg_text(b" \n\t \r\n") is None
def test_bytes_over_the_size_cap_are_refused() -> None:
"""The memory guard: an over-cap file never reaches the parser."""
padded = b"<svg><title>" + b"x" * (MAX_SVG_BYTES + 1) + b"</title></svg>"
assert len(padded) > MAX_SVG_BYTES
assert extract_svg_text(padded, stem="huge") is None
def test_bytes_exactly_at_the_size_cap_are_parsed() -> None:
"""The cap is ``>``-based: exactly ``MAX_SVG_BYTES`` is in range."""
opener, closer = b"<svg><title>", b"</title></svg>"
filler = b"x" * (MAX_SVG_BYTES - len(opener) - len(closer))
result = extract_svg_text(opener + filler + closer)
assert result is not None
assert result.structural is False
def test_doctype_with_an_internal_entity_is_refused() -> None:
"""Stdlib ElementTree happily EXPANDS an internal entity — proving
why the doctype-bearing input is refused outright (billion-laughs
class): the reference must never be resolved."""
svg = b'<!DOCTYPE svg [<!ENTITY x "y">]><svg><title>&x;</title></svg>'
assert extract_svg_text(svg, stem="bomb") is None
def test_bare_entity_declaration_is_refused() -> None:
assert extract_svg_text(b'<!ENTITY x "y"><svg><title>x</title></svg>') is None
def test_entity_declaration_hidden_inside_content_is_refused() -> None:
"""The marker scan is over the raw bytes — a declaration anywhere
(even in a position the parser would reject) refuses the file."""
assert extract_svg_text(b"<svg>\n<!entity x 'y'>\n</svg>") is None
def test_guards_are_case_insensitive() -> None:
assert extract_svg_text(b'<!doctype svg SYSTEM "http://x/y.dtd"><svg/>') is None
assert extract_svg_text(b'<!entity x "y"><svg/>') is None
def test_malformed_xml_is_a_refusal_not_an_exception() -> None:
"""The caller's fail-soft is a skip (``images_failed``), so a parse
error must never escape."""
assert extract_svg_text(b"<svg><title>unclosed") is None
assert extract_svg_text(b"<svg><rect></svg>") is None
assert extract_svg_text(b"\x00\x01\x02") is None
assert extract_svg_text(b"not xml at all") is None
# --- extraction: title / desc / text ---------------------------------------
def test_title_desc_and_text_are_collected_in_document_order() -> None:
svg = (
b"<svg>"
b"<title>Top Title</title>"
b"<desc>What the diagram shows</desc>"
b"<text>first label</text>"
b"<text>second label</text>"
b"</svg>"
)
assert extract_svg_text(svg, stem="topo") == SvgText(
text="Top Title\nWhat the diagram shows\nfirst label\nsecond label",
structural=False,
)
def test_a_title_after_its_shape_still_follows_document_order() -> None:
svg = b"<svg><text>later</text><g><title>group title</title></g><desc>d</desc></svg>"
assert extract_svg_text(svg) == SvgText(
text="later\ngroup title\nd", structural=False
)
def test_title_and_desc_concatenate_nested_markup() -> None:
svg = b"<title>Deploy <tspan>flow</tspan> map</title>"
full = b"<svg>" + svg + b"</svg>"
assert extract_svg_text(full) == SvgText(text="Deploy flow map", structural=False)
def test_text_children_are_joined_with_a_space() -> None:
"""``<text>CPU<tspan>usage</tspan></text>`` reads "CPU usage" —
without the space the labels weld into one unsearchable token."""
svg = b'<svg><text>CPU<tspan>usage</tspan>7d</text></svg>'
assert extract_svg_text(svg) == SvgText(text="CPU usage 7d", structural=False)
def test_whitespace_is_normalised_in_every_item() -> None:
svg = b"<svg><title> rack\n\t layout map </title><text> a\n b </text></svg>"
assert extract_svg_text(svg) == SvgText(
text="rack layout map\na b", structural=False
)
def test_blank_title_is_dropped_and_never_stops_the_walk() -> None:
svg = b"<svg><title> </title><desc> </desc><text>real label</text></svg>"
assert extract_svg_text(svg) == SvgText(text="real label", structural=False)
def test_only_blank_text_falls_back_to_the_structural_line() -> None:
"""A blank ``<title>``/``<text>`` contributes no words, so the file
is still described by what it is (the empty ``<text>`` counts as a
drawn element)."""
svg = b'<svg viewBox="0 0 4 4"><title></title><text> </text></svg>'
assert extract_svg_text(svg, stem="blank") == SvgText(
text='Vector diagram "blank" — 1 shape (text), viewBox 4×4', structural=True
)
def test_repeats_are_dropped_case_insensitively_keeping_the_first() -> None:
svg = (
b"<svg>"
b"<desc>Power Draw</desc>"
b"<title>power draw</title>"
b"<text>Other</text>"
b"</svg>"
)
assert extract_svg_text(svg) == SvgText(text="Power Draw\nOther", structural=False)
def test_parser_decodes_entity_references_in_the_text() -> None:
"""``&amp;`` comes out decoded — the renderer re-escapes on display
(phase 180), so storing the decoded value is the correct direction."""
svg = b"<svg><title>Red &amp; Blue &#8212; map</title></svg>"
assert extract_svg_text(svg) == SvgText(text="Red & Blue — map", structural=False)
def test_attributes_are_never_text() -> None:
svg = (
b'<svg xmlns:xlink="'
+ XLINK_NS.encode()
+ b'"><title lang="en">Visible</title>'
b'<image xlink:href="secret-internal-name.png"/></svg>'
)
assert extract_svg_text(svg, stem="diagram") == SvgText(
text="Visible", structural=False
)
def test_an_image_only_svg_yields_no_words_from_its_href() -> None:
"""ASSUMPTION (task 01): an embedded ``<image>`` is not followed —
its ``href`` is not text, so the structural line describes the file."""
svg = (
b'<svg xmlns:xlink="'
+ XLINK_NS.encode()
+ b'" viewBox="0 0 10 10">'
b'<image xlink:href="inner.png"/>'
b"</svg>"
)
result = extract_svg_text(svg, stem="poster")
assert result is not None
assert result.structural is True
assert result.text == 'Vector diagram "poster" — 1 shape (image), viewBox 10×10'
# --- extraction: namespaces ------------------------------------------------
def test_the_default_svg_namespace_is_ignored() -> None:
svg = (
b'<svg xmlns="'
+ SVG_NS.encode()
+ b'"><title>Namespace title</title><desc>ns desc</desc>'
b'<text>ns label</text></svg>'
)
assert extract_svg_text(svg) == SvgText(
text="Namespace title\nns desc\nns label", structural=False
)
def test_a_declared_prefix_form_counts_like_the_plain_tag() -> None:
"""``ElementTree`` expands ``<svg:title>`` to Clark notation, so the
local-name match covers it for free."""
svg = (
b'<svg xmlns:svg="'
+ SVG_NS.encode()
+ b'"><svg:title>Prefixed</svg:title>'
b"<svg:desc>Prefixed desc</svg:desc></svg>"
)
assert extract_svg_text(svg) == SvgText(
text="Prefixed\nPrefixed desc", structural=False
)
def test_a_foreign_namespace_element_is_not_worded_text() -> None:
"""A ``<title>`` in a NON-SVG vocabulary is still matched by local
name (the walk is namespace-agnostic) while the foreign element
counts as a shape when it carries no words."""
svg = (
b'<svg xmlns:h="http://www.w3.org/1999/xhtml">'
b'<foreignObject><h:p>paragraph</h:p></foreignObject>'
b"</svg>"
)
result = extract_svg_text(svg, stem="mixed")
assert result is not None
assert result.structural is True
assert result.text == 'Vector diagram "mixed" — 2 shapes (foreignObject, p)'
def test_the_xml_declaration_encoding_is_honoured() -> None:
svg = b"<?xml version='1.0' encoding='UTF-8'?><svg><title>Caf\xc3\xa9</title></svg>"
assert extract_svg_text(svg) == SvgText(text="Café", structural=False)
def test_comments_are_not_text_and_not_shapes() -> None:
svg = b'<svg viewBox="0 0 2 2"><!-- a comment --><path d="M0 0"/></svg>'
result = extract_svg_text(svg, stem="clean")
assert result is not None
assert result.text == 'Vector diagram "clean" — 1 shape (path), viewBox 2×2'
# --- the MAX_SVG_TEXT_CHARS cap --------------------------------------------
def test_extraction_under_the_cap_is_untouched() -> None:
svg = b"<svg><title>" + b"x" * (MAX_SVG_TEXT_CHARS - 20) + b"</title></svg>"
result = extract_svg_text(svg)
assert result is not None
assert len(result.text) == MAX_SVG_TEXT_CHARS - 20
def test_over_cap_text_ends_on_a_sentence_boundary() -> None:
svg = b"<svg><title>" + b"x. " * 2000 + b"</title></svg>"
result = extract_svg_text(svg)
assert result is not None
assert result.structural is False
assert len(result.text) <= MAX_SVG_TEXT_CHARS
assert len(result.text) > MAX_SVG_TEXT_CHARS - 10
assert result.text.endswith(".")
assert result.text.count("x.") < 2000 # the cut is mid-item
def test_over_cap_text_prefers_the_last_newline_boundary() -> None:
"""Two items joined with ``\\n``: the newline is the LAST boundary
inside the window, so the second item is dropped whole (never a
partial word) and the trailing newline is stripped."""
first = b"a" * (MAX_SVG_TEXT_CHARS - 10)
svg = b"<svg><title>" + first + b"</title><text>" + b"b" * 200 + b"</text></svg>"
result = extract_svg_text(svg)
assert result is not None
assert result.text == first.decode()
def test_over_cap_text_without_any_boundary_is_hard_cut() -> None:
svg = b"<svg><title>" + b"x" * (MAX_SVG_TEXT_CHARS + 1000) + b"</title></svg>"
result = extract_svg_text(svg)
assert result is not None
assert result.text == "x" * MAX_SVG_TEXT_CHARS
# --- the structural fallback (LOCKED A2) -----------------------------------
def test_single_shape_line_matches_the_pinned_wording() -> None:
svg = b'<svg><path d="M0 0 L1 1"/></svg>'
assert extract_svg_text(svg, stem="topo") == SvgText(
text='Vector diagram "topo" — 1 shape (path)', structural=True
)
def test_shape_noun_pluralises_and_counts_are_listed() -> None:
svg = b"<svg><path/><path/><path/></svg>"
assert extract_svg_text(svg, stem="topo") == SvgText(
text='Vector diagram "topo" — 3 shapes (path×3)', structural=True
)
def test_top_three_kinds_with_document_order_tie_break() -> None:
svg = b"<svg><g/><g/><g/><path/><path/><circle/><rect/></svg>"
result = extract_svg_text(svg, stem="census")
assert result is not None
assert result.text == 'Vector diagram "census" — 7 shapes (g×3, path×2, circle)'
assert result.structural is True
def test_structure_elements_are_never_counted_as_shapes() -> None:
svg = b"<svg><title> </title><desc/><g><rect/></g></svg>"
result = extract_svg_text(svg, stem="nested")
assert result is not None
assert result.text == 'Vector diagram "nested" — 2 shapes (g, rect)'
def test_a_nested_svg_element_is_structure_too() -> None:
svg = b"<svg><g><svg><rect/></svg></g></svg>"
result = extract_svg_text(svg, stem="inset")
assert result is not None
assert result.text == 'Vector diagram "inset" — 2 shapes (g, rect)'
def test_viewbox_reports_width_and_height() -> None:
svg = b'<svg viewBox="0 0 128 64"><path/></svg>'
result = extract_svg_text(svg, stem="wide")
assert result is not None
assert result.text == 'Vector diagram "wide" — 1 shape (path), viewBox 128×64'
def test_viewbox_accepts_commas_and_integral_floats() -> None:
svg = b'<svg viewBox=" 0 , 0 , 8.0 , 6.0 "><path/></svg>'
result = extract_svg_text(svg, stem="comma")
assert result is not None
assert result.text == 'Vector diagram "comma" — 1 shape (path), viewBox 8×6'
def test_viewbox_keeps_fractional_sizes_without_padding() -> None:
svg = b'<svg viewBox="0 0 8.5 6.25"><g><circle/></g></svg>'
result = extract_svg_text(svg)
assert result is not None
assert result.text == "Vector diagram — 2 shapes (g, circle), viewBox 8.5×6.25"
def test_unparseable_viewbox_omits_the_clause() -> None:
for raw in (b"", b"0 0 8", b"0 0 8 6 4", b"0 0 wide tall", b"0 0 NaN NaN"):
svg = b'<svg viewBox="' + raw + b'"><path/></svg>'
result = extract_svg_text(svg, stem="odd")
assert result is not None, raw
assert result.text == 'Vector diagram "odd" — 1 shape (path)', raw
def test_missing_viewbox_omits_the_clause() -> None:
result = extract_svg_text(b"<svg><path/></svg>", stem="nobox")
assert result is not None
assert "viewBox" not in result.text
def test_viewbox_without_shapes_is_the_only_fact() -> None:
svg = b'<svg viewBox="0 0 16 9"></svg>'
assert extract_svg_text(svg, stem="canvas") == SvgText(
text='Vector diagram "canvas" — viewBox 16×9', structural=True
)
def test_empty_stem_never_renders_empty_quotes() -> None:
assert extract_svg_text(b"<svg></svg>") == SvgText(
text="Vector diagram", structural=True
)
assert extract_svg_text(b'<svg viewBox="0 0 1 1"><path/></svg>') == SvgText(
text="Vector diagram — 1 shape (path), viewBox 1×1", structural=True
)
def test_a_root_that_is_not_svg_still_yields_honest_text() -> None:
"""The walk is tag-based — a mis-named root is not rejected, and the
root is never counted among the shapes."""
result = extract_svg_text(b"<html><title>Legacy</title></html>", stem="legacy")
assert result == SvgText(text="Legacy", structural=False)
structural = extract_svg_text(b"<html><body/></html>", stem="legacy")
assert structural == SvgText(
text='Vector diagram "legacy" — 1 shape (body)', structural=True
)